|
|
     |
||
 |
     |
 |
|
8.1 RelevanceThe evaluation evidence indicates that, for the most part, the Canada/Prince Edward Island Labour Market Development Agreement (LMDA) and Employment Benefits and Support Measures (EBSMs) are relevant to the needs of the targeted industry sectors, employers, communities and Islanders. The LMDA focuses on resource and seasonal industries (e.g., agriculture, fishing and tourism), the staples of the Island, as well as on targeted high-value industries such as information technology and aerospace, which have the potential to create long-term jobs for the province. Still, there were some opinions expressed that the LMDA needs to be better focused on the unique needs of PEI (e.g., programs should be better adapted to the seasonal economy, high level of unemployment, and low annual earnings of Islanders). Moreover, there was a pervasive concern that the relevance of the EBSMs is limited by the restrictive program eligibility criteria imposed by the Employment Insurance (EI) Act. This concern appears to stem from a widely held misperception that the mandate of the LMDA encompasses the entire PEI labour force, when in fact the Agreement is designed to serve EI clients only. It was believed that the focus of EBSM assistance on EI eligible clients meant that some individuals in need of assistance were not being served. In particular, many respondents felt that small business development, skills upgrading for currently employed or underemployed Islanders (in part to supply the skills required in new targeted industries), and the needs of youth and persons with a weak attachment to the labour market are not being adequately addressed by the LMDA or any other initiatives. There was some conjecture among key informants on the issue of focusing on seasonal industry. On the one hand, taking measures to extend the duration of the work season in seasonal industries was seen as acceptable given that these industries will be a reality on the Island for the foreseeable future. Still, respondents holding this view also noted that seasonal jobs should be seen only as stepping stones to longer term employment, that seasonal workers should be equipped with multiple skills to make them portable, and that there is a need to promote industries complementary to seasonal industries. On the other hand, the majority of key informants suggested that the emphasis for the LMDA should be on developing new year-round industries so that the Island can end its dependence on seasonal industry. There were also conflicting views with respect to the degree of overlap between the EBSMs and other employment and training assistance measures. Senior managers and stakeholders felt that the EBSMs are well harmonized with other federal and provincial programs mainly because of the wide range of representation on LMDA committees and working groups. On the other hand, managers below the senior management level and provincial government representatives perceived that there is a fair amount of duplication and a lack of coordination among programs. 8.2 Design and DeliveryConsidering the complexity of the task, the implementation of the Canada/PEI LMDA has gone reasonably well to date, though further development and improvements will be needed. On the positive side, the high degree of cooperation and collaboration among all LMDA players, the delivery of successful initiatives such as adult basic education/literacy and aerospace, and Human Resource Centre of Canada (HRCC) staff's understanding of their role in EBSM delivery were all noted by interview and focus group respondents as successful aspects of LMDA implementation. On the other hand, a lack of reliable, timely labour market information and client tracking information for LMDA planning and management, excessive administrative requirements and the associated delays in project approvals, and somewhat of a HRCC staff shortage were regarded as key weaknesses that will require attention. Also, both federal and provincial managers identified the lack of technical resources to support needed LMDA information systems as a problem. Although productive partnerships with community organizations have been developed and despite the fact that local-level consultations with stakeholder organizations had been conducted during the development of the 1998-1999 LMDA business plan, stakeholders as well as front-line HRCC staff felt that consultations and partnerships with "grass roots" community organizations still need to be improved. Moreover, there is a need to increase awareness of the LMDA and EBSMs at the community level. The EBSMs were generally regarded as flexible and responsive to local and client needs. Interview and focus group respondents had some reservations, however. In particular, respondents identified a need to further refine and adapt the programs to the unique needs of PEI (e.g., many seasonal industries, relatively high unemployment, and low annual earnings), to harmonize LMDA and Social Assistance programming, and to better serve and inform clients about available programs. Also, as already noted, the EI eligibility criteria of the Employment Insurance Act and delays in project approvals were thought to limit the responsiveness of EBSMs. The perceived strengths of Human Resources Development Canada's approach to delivering the EBSMs include a cooperative and positive effort on the part of highly experienced HRCC staff, flexible and decentralized program delivery, and a reasonable amount of paper work for clients/funding recipients. On the other hand, some clients and stakeholders perceived that increased flexibility at some HRCCs and not others meant that service delivery could be inconsistent from one HRCC to another. In the survey, clients indicated being most satisfied with the quality of education or training they received and with the knowledge of employment counsellors, but comparatively less satisfied with the quality of referral services and with the information available to help them choose suitable programs. In addition, LMDA programs and services are being successfully delivered in both official languages as intended. 8.3 Federal-Provincial PartnershipThe federal-provincial partnership has been working reasonably well, though some "growing pains" are still being experienced. Both federal and provincial managers identified several strengths of the partnership and the co-management approach. In particular, these respondents perceived a high degree of cooperation between partners and noted that co-management facilitates collaborative decision-making, mutual understanding and coordination of federal and provincial initiatives. They acknowledged that the partnership does take a lot of effort and compromise, however. For example, front-line HRCC staff observed that the partnership has been difficult and frustrating at times, and that the degree of cooperation between the two levels of government could have been better in the pilot of Skills Development. Moreover, as already noted, most respondents observed that co-management adds another layer of bureaucracy and complexity to the LMDA, resulting in delays in project approvals. Many Human Resources Development Canada (HRDC) staff have been reluctant to embrace the new co-management approach due to their concern that program delivery may eventually be fully devolved to the Province, which causes them anxiety over their job security. Although most senior managers felt that the LMDA is generally compatible with broader government objectives, provincial key informants asserted that the EBSMs need to be further adapted to better match the objectives of the provincial government. 8.4 Success to DateThe results of the accountability target attainment computations indicated that the EBSMs exceeded both the 1998/99 return-to-work and unpaid EI benefits targets. Despite exceeding the return to work target, questions were raised about whether or not all returns to work were being captured by the information systems. Qualitative evidence of impacts on participants indicated favourable employment outcomes, if not immediately then expected in the long-term because of positive skill and psychological (e.g., self-confidence, self-esteem) impacts. Still, less than one half of survey respondents said their EBSM intervention was very important in attaining their current job although this reflected a more positive perceived impact than the comparison group. Employers consulted in focus groups agreed that skill and psychological impacts are occurring, although a number said they were often unable to retain their wage-subsidy workers because of financial difficulties. Quantitative survey evidence indicated that at this stage of the evaluation, only Self-Employment (SE) led to consistently positive employment and income-support outcomes. Targeted Wage Subsidies (TWS) and Training/Feepayers (TFP) increased the likelihood of being employed for 12 consecutive weeks. Employment Assistance Services (EAS), TWS and Job Creation Partnerships (JCP) produced negative earnings outcomes. More conclusive evidence of EBSM impact on employment and income support will be available at the summative stage of evaluation. Results were mixed for perceived employer impacts. On the one hand, employers in the focus groups said that TWS relieved some employers of cash-flow pressures in hiring and that training unskilled workers enabled some businesses to be sustainable and even competitive. On the other hand, some employers were unable to retain workers because of cash flow problems and others said the EBSMs would be unable to channel workers into sectors and occupations where they were in demand. In addition, some employers in certain areas (e.g., seasonal, information technology, new/small businesses) did not think the LMDA as implemented could meet their specific needs because of perceived shortcomings in the rules. Some employers believed the EBSMs would be insufficient to effect a change in attitudes with respect to valuing training and dependence on seasonal industries and income support. For impacts on the community, some key informants and focus group participants felt it may be too early to address this issue. Among those who were able to, there were mixed results. Some respondents felt the LMDA may have helped in addressing short-term needs of communities and Islanders, but there was concern that the exclusion of non-EI eligible persons may limit the LMDA's ability to address long-term needs. Others mentioned that, to truly benefit the province, there needs to be greater emphasis on interventions providing labour market development than on those providing job creation and wage subsidies. On the other hand, some respondents spoke of the benefits of the LMDA for the community, including the extension of the "shoulder" season, community learning centres, lifelong learning and adult literacy training measures, the technology mentoring program, lower Social Assistance (SA) caseload, and partnerships between communities and the government. 8.4.1 Rural-Urban DifferencesA review of the qualitative and quantitative evidence indicates that there were few clear rural-urban differences in terms of perspectives on the LMDA and in its impacts. With respect to the latter, multivariate analysis indicated that the EBSMs had positive impacts on rural residents in terms of the likelihood of seasonal employment, the percentage of weeks employed and three consecutive months of employment, but were beneficial for urban residents in terms of full-time employment. Also, they produced negative effects for rural residents in terms of earnings. We also attempted to contrast the views of focus group participants located in urban, rural and isolated rural centres, but were again unable to observe any clear pattern. Differences observed between rural and urban clients include the following:
In sum, no strong pattern of differences in impacts was observed between urban and rural participants, although the available evidence suggests that rural clients are more likely to be affected by the EBSMs than urban clients. Multivariate analyses of program impacts show negative effects for rural participants with respect to earnings outcomes, which may be explained by the fact that opportunities for entering better-paying jobs are fewer in rural areas. However, in light of the higher unemployment in rural areas, as indicated by the higher rates of Employment Insurance dependency37 in these regions, it is perhaps surprising that employment outcomes were not more consistently positive for urban residents. Also, in the few areas where differences in satisfaction between urban and rural residents were observed, rural residents report higher satisfaction. This latter finding may imply that urban residents have higher expectations with respect to EBSM services. 8.5 Lessons LearnedThe key lessons learned in the formative evaluation are presented in this section. 8.5.1 Information Systems and Results MeasurementLesson 1 There are serious information system problems that have been perceived to exist which hamper the ability to track both clients' progress and results in a timely fashion and to provide accurate and usable labour market information. A large number of problem areas, also noted in the pan-Canadian study, that contributed to the information system difficulties underline the severity of the problem. They are as follows:
The information difficulties identified had implications for the current evaluation. The administrative data problems, in particular the lack of accurate return-to-work indicators, compromises the ability to accurately assess the results targets. Further, the absence or inaccuracy of identifiers in the administrative dataset compromised our ability to accurately profile participants, to include all members of the population in the sample frame and therefore develop a sample that was truly representative, and to contact all members of the sample for the survey. Attempts to acquire the necessary information required a review of several different databases from several different sources. This process was far more complex than it should have been and still failed to yield complete data. If the data issues go unresolved, they will pose problems for the summative evaluation as well. Lesson 2 In assessing the returns-to-work and unpaid EI benefits targets set for the Canada/PEI Labour Market Development Agreement (LMDA) EBSMs, EKOS' computation found that both targets were exceeded in 1998/99, particularly the former. The fact that the return-to-work target was exceeded by almost 50 percent, despite the fact that the target had been increased from the year before, would suggest increased EBSM effectiveness due to growing familiarity with processes. However, it could also reflect targets that were set too low and/or an economy that was generating increased job opportunities. Moreover, despite the fact that the EBSM generated returns-to-work exceeding targets, EKOS' analysis indicated a discrepancy between the survey and administrative datasets in terms of capturing returns-to-work. Moreover, EKOS raised concerns about how unpaid EI benefits were computed. This was a problem identified nationally in the pan-Canadian EBSM formative evaluation, as well as in evaluations of LMDAs in other parts of the country. Lesson 3 Analysis of EBSM impacts indicated that SE was particularly effective in integrating participants into the labour market compared to non-participants. This was found even after controlling for differences between the characteristics of participants and non-participants (e.g., differences in education level) that could have accounted for the employment advantage in favour of participants over and above the intervention itself. The fact that other interventions were found, for the most part, not to play a significant role in participants' entering employment, despite positive effects initially found in the purely descriptive analysis, suggests that in measuring impacts the characteristics of participants must be taken into consideration. Also, because not all interventions are expected to have immediate employment outcomes like SE, the analysis also suggests that it may be too early to measure the true employment-generating properties of all EBSMs, i.e., within the timeframe of this formative evaluation. With respect to other outcomes, SE was found to lower EI use compared to non-participants, but no evidence was found for the reduction of dependence on income support, to date. No increase in earnings was detected for this or any other EBSM intervention. 8.5.2 Service DeliveryLesson 4 Although most clients appear to be quite satisfied with the programs and services that they have received, some are nonetheless unhappy about the way they are treated while accessing services. They are dissatisfied with the layout of the HRCCs and with not being adequately informed about programs and services. Lesson 5 The EBSMs are largely responsive to local needs and in part this can be attributed to HRCC managers' spending authority of up to $75,000. There is, however, a perceived lack of consistency among the different HRCCs in the way programs are delivered, the amount of financial assistance granted to clients, etc. There are, of course, trade-offs involved here the more local flexibility and responsiveness, the less consistency in program delivery from one local area to another. While local flexibility is a desired attribute, clients perceive that there are too many inconsistencies across HRCCs and they feel that this is unfair treatment. 8.5.3 Program Relevance and DesignLesson 6 For the most part, the LMDA is considered relevant to the needs of the targeted industry sectors, employers, communities and Islanders. Respondents also feel that the LMDA is relevant in the sense that the EBSMs can be used to extend the shoulder season of seasonal industries and to help diversify the Island economy into non-seasonal industries. It was also noted, however, that this diversification may require a change in the way some Islanders view their working lives and their economy in terms of dependence on seasonal work. Although some senior federal managers feel that the EBSMs are consistent with government priorities, objectives and HRDC's mandate, provincial officials suggest that the former HRDC programs need to be further adapted to meet the specific needs of the Island. Lesson 7 There are mixed views and some confusion regarding the extent to which the EBSMs are harmonized and complementary with other provincial and federal initiatives. While senior managers tend not to see any major problems with overlap, middle management and front-line personnel perceive that there is some duplication and a lack of coordination among federal and provincial programs (e.g., wage subsidies, self-employment and youth programs). Moreover, in the 1998 national EBSM formative evaluation, the potential for duplication in Atlantic Canada was noted. Lesson 8 Many respondents believe that the imposition of EI eligibility as a criterion for LMDA program participation creates a serious gap in programming because many Islanders (e.g., currently employed, self-employed, youth, chronically unemployed, persons with disabilities) are unable to access EBSMs as a result. Further, many respondents point to a perceived lack of services for currently employed individuals (i.e., for skills upgrading, particularly in priority sectors) as an example of a major gap in labour market programming. This evaluation focussed only on EBSMs and their associated client base and did not look at the specific needs of these other groups. Meeting their needs would require a comprehensive labour market development strategy for the whole province which goes far beyond the issues addressed in this evaluation. 8.5.4 Community PartnershipsLesson 9 Although productive partnerships have been developed with community organizations, stakeholders and front-line HRCC staff feel that consultations and partnerships with "grass roots" community organizations need to be improved. Despite the fact that they were consulted for the 1998/1999 LMDA business plan, community groups would like more input into LMDA planning and implementation. Findings from the 1998 national EBSM evaluation also indicate a need for improved community consultations and partnerships. 8.5.5 Federal-Provincial Partnership and Co-ManagementLesson 10 At senior management levels, co-management has had a number of positive impacts, such as better communication, mutual understanding and learning, collaborative decision-making and better coordination of joint initiatives. However, there are perceptions that management does not always communicate well with front-line staff; thus there has been some confusion about programs. Also, HRDC staff perceive that the lack of information sharing between federal and provincial partners on EI clients and Social Assistance clients limits the opportunities to achieve productive, long-term solutions for both types of clients. Further, there is much anxiety among staff about what will happen in the long term in labour market development (i.e., whether programs will be fully devolved to the Province and whether some federal staff will lose their jobs as a result). Lesson 11 The co-management model for the LMDA has been quite helpful to both levels of government. When the Agreement was first signed in 1997, the Province, with the exception of the Health and Social Services Department, had little experience in delivering employment programs and services department, while HRDC benefited from the economic development expertise of the Province. The added layer of bureaucracy in the administration of the programming has, however, caused delays in reviewing and approving project applications. 8.5.6 Pan-Canadian FindingsLesson 12 Many of the findings from this evaluation echo the pan-Canadian 1998 Formative Evaluation of the Employment Benefits and Support Measures. They follow:
8.6 Management ResponseThe Management Committee of the Canada-PEI Labour Market Development Agreement (LMDA) has reviewed the Formative Evaluation of the Canada-Prince Edward Island Labour Market Development Agreement and is very pleased to have received this feedback. The Committee will use the evaluation results to improve its decision-making processes and to enhance labour market programs and services available to Islanders. The Formative Evaluation focuses on the partnership model initiated to design and manage the LMDA and on the Employment Benefits and Support Measures (EBSMs) available to EI eligible clients. A greater portion of the data collected concentrated on the five employment benefits, which are those programs designed for short-term outcomes. Qualitative information was also gathered on the support measures, such as the Local Labour Market Partnerships (LLMP). The evaluation provides a preliminary review of LMDA implementation based on information collected from December 1998 until October 1999. Further patterns and program outcomes will be captured in the summative evaluation to follow. Generally, the Formative Evaluation concludes that LMDA programs are relevant to the labour needs of Islanders, communities, employers and growth sectors of industry. EBSMs can be used to extend the shoulder season of seasonal industries and to help diversify the Island economy into non-seasonal industries. As well, strengths of the partnership approach are confirmed, but some areas for improvement are also highlighted. One identified strength of the co-managed model is the collective knowledge and perspectives the labour market partners bring to planning and decision making. The combination of HRDC's extensive experience in labour market programming and service delivery with the Province's expertise in economic development has proven effective for labour market development on PEI. For example, the focus on industry sectors with potential for growth has successfully created employment for many Islanders. By considering labour market development within the PEI economic context, labour market funding can be invested strategically. A principle established in the LMDA to guide the labour market partners in the implementation of their partnership arrangement is to harmonize employment initiatives "...to ensure that there is no unnecessary overlap and duplication. The evaluation uncovered that having representatives from both governments on the Management Committee has served to decrease duplication of services. However, management continues to place programs strategically to minimize overlap and to address gaps in service delivery. While cooperation between the labour market partners is significant, some difficulties with the management structure were identified in the evaluation. Concern was expressed that the co-managed model causes delays in some approval processes. As well, the on-going operation of this strengthened model was found to be resource intensive. The Management Committee acknowledges that working together has been a learning experience, but that improvements have been made over time. Initial delays have been addressed and the Committee continues to strive towards timely and efficient processes. Evaluation of the EBSMs demonstrates that targets are being consistently met. Multivariate analysis of EBSM participation results showed a difference in outcomes between rural and urban participants — rural clients achieved longer periods of employment while urban clients were more likely to move into full-time employment. In addition, positive qualitative employment outcomes, including self-confidence and esteem building, were attributed to EBSM participation. Employers viewed the EBSMs as making their business more sustainable and consistent. However, they also felt that Targeted Wage Subsidies (TWS) could be more flexibly designed to enable the creation of sustainable employment. The seasonal nature of the Island economy must be considered in developing long-term labour market strategies. Employers, like the Management Committee, saw implementing EBSMs designed for the PEI context as crucial for long-term employment creation. Clients, employers and service deliverers indicated customer service issues concerning quality and consistency of information and referral services. Again, this group "... indicated that the EBSMs need to be further refined and adapted to meet the unique economic needs of Islanders." Efforts are being made to develop a balance of flexibility and consistency in these areas. Services are being enhanced through increased labour market and client tracking information, public awareness and customer service. PEI specific issues are considered when making funding decisions. The Management Committee continues to endeavour towards PEI-economy-specific programming. The LMDA Management Committee accepts concerns expressed by service deliverers, employers and Islanders in general for the labour market needs of those ineligible to receive EBSMs. Individuals who are not EI eligible, in particular "...employed workers, youth and those with weak labour market attachments," do not have access to valuable programs which could improve their labour market outcomes. The situations of these Islanders need to be addressed, though not necessarily through this Agreement, to allow full labour force participation. Results of this evaluation are consistent with the findings of the Pan Canadian Formative Evaluation of Employment Benefits and Support Measures completed in 1998, but they focus with more detail on Island-specific issues. The LMDA formative evaluation will provide a solid foundation for the upcoming summative review as the partners strive towards positive labour market impacts and outcomes through this co-managed model. The Formative Evaluation of the Canada-PEI Labour Market Development Agreement has provided federal and provincial partners with valuable recommendations for improving the effectiveness of labour market programs and services for EI eligible Islanders. The Management Committee is committed to providing relevant and high quality labour market programs and services designed to fit the specific needs of Islanders.
| |||||||||
![[To Top]](/web/20060127015432im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127015432im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127015432im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127015432im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}