|
|
     |
||
 |
     |
 |
|
Formative Evaluation of EI Part II Under the Canada/Prince Edward Island Labour Market Development AgreementFinal ReportEvaluation and Data Development
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060127015416im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif) |
| Shelley Borys | Principal Investigator and Project Manager |
| Mike Callahan | Senior Consultant |
| Norman Leckie | Senior Consultant |
| Bill Blois | Research Analyst |
| Simon Roy | Consultant |
| Natalya Kuziak | Research Analyst |
| Benoît Gauthier | Senior Advisor |
| Ken Innes | Database Manager |
| Lina Danileviciute | Database Analyst |
| Mark Anderson | Survey Manager |
While EKOS Research accepts responsibility for any errors contained in this report, we would nonetheless like to acknowledge the important contribution of a number of individuals to the successful completion of this project. The members of the Joint Evaluation Committee (JEC) for the Formative Evaluation were, in no particular order, Bob Charlton, Co-Chair from Human Resources Development Canada (HRDC), PEI Region; Janet Wood, Co-Chair on behalf of the Department of Development and Technology; Betty Newson of the Department of Health and Social Services; Faye Martin of the Department of Education; and Karen Johnson and Suzanne Marshall of HRDC National Headquarters. The JEC provided valuable advice and guidance to the research team throughout the project. We would also like to thank all respondents to the surveys, interviews, focus groups and the case study who took time out of their busy schedules to participate in this evaluation. Without their cooperation, this research would not have been possible.
|
Under the terms of the Canada/Prince Edward Island Labour Market Development Agreement (LMDA), which entered into effect on April 26, 1997, the federal and provincial governments agreed to share responsibility for active labour market measures for Employment Insurance (EI) eligible unemployed in the province. In accordance with this Agreement, the two governments are responsible for designing and delivering programs that have goals and objectives similar to the active Employment Benefits and Support Measures (EBSMs) identified in the Employment Insurance Act. Under this flexible partnership or co-management model — wherein Canada and PEI have equal roles in design, management and evaluation — the federal government retains responsibility for the financing and delivery of Employment Insurance benefits, measures, and services. The employment benefits reviewed during the formative evaluation were the Targeted Wage Subsidies (TWS) program, the Self-Employment (SE) program, the Job Creation Partnerships (JCP) program, Purchase of Training and Skills Development. The support measures that were evaluated included Employment Assistance Services (EAS) and Local Labour Market Partnerships (LLMP).
The formative evaluation was conducted between December 1998 and October 1999, though the final analysis and reporting continued until March 2000. The evaluation employed multiple lines of evidence to assess issues pertaining to the rationale, design and delivery, impact and success of EBSMs, as well as EBSM-specific issues. Data collection methods included:
The evaluation evidence indicates that, for the most part, the Canada/PEI LMDA and EBSMs are relevant to the needs of the targeted industry sectors, employers, communities and Islanders. The LMDA focuses on resource and seasonal industries (e.g., agriculture, fishing, and tourism), the staples of the Island, as well as on targeted high-value industries such as information technology and aerospace, which have the potential to create long-term jobs for the province. Still, there were some opinions expressed that the LMDA needs to be better focused on the unique needs of PEI (e.g., programs should be better adapted to the seasonal economy, high level of unemployment, and low annual earnings of Islanders). Moreover, there was a pervasive concern that the relevance of the EBSMs is limited by the restrictive program eligibility criteria imposed by the EI Act. This concern appears to stem from a widely held misperception that the mandate of the LMDA encompasses the entire PEI labour force, when in fact the Agreement is designed to serve EI clients only. In any case, the focus of EBSM assistance on EI eligible clients was thought to leave some major gaps in needed programming. In particular, many respondents felt that small business development, skills upgrading for currently employed or under-employed Islanders (in part to supply the skills required in new targeted industries), and the needs of youth and persons with a weak attachment to the labour market are not being adequately addressed by the LMDA or any other initiatives.
There was some conjecture among key informants on the issue of focusing on seasonal industry. On the one hand, taking measures to extend the duration of the work season in seasonal industries was seen as acceptable given that these industries will be a reality on the Island for the foreseeable future. Still, respondents holding this view also noted that seasonal jobs should be seen only as stepping stones to longer term employment, that seasonal workers should be equipped with multiple skills to make them portable, and that there is a need to promote industries complementary to seasonal industries. On the other hand, the majority of key informants suggested that the emphasis for the LMDA should be on developing new year-round industries so that the Island can end its dependence on seasonal industry.
On the issue of the complementarity of LMDA and other programming, federal and provincial officials at the senior management level generally felt that the EBSMs are fairly well harmonized with other programs, though at the middle management and front-line levels many respondents perceived there to be work needed to resolve issues related to duplication, overlap and a lack of coordination among federal and provincial programs (e.g., wage subsidies, self-employment and youth programs). It should be noted, however, that these areas of overlap existed before the signing of the LMDA in 1997.
Considering the complexity of the task, the implementation of the Canada/PEI LMDA has gone reasonably well to date, though further development and improvements will be needed. On the positive side, the high degree of cooperation and collaboration among all LMDA players, the delivery of successful initiatives such as adult basic education/literacy and aerospace, and HRCC staff's understanding of their role in EBSM delivery were all noted by interview and focus group respondents as successful aspects of LMDA implementation. On the other hand, a lack of reliable, timely labour market information and client tracking information for LMDA planning and management, excessive administrative requirements and the associated delays in project approvals, and somewhat of an HRCC staff shortage were regarded as key weaknesses that will require attention. Also, both federal and provincial managers identified the lack of technical resources to support needed LMDA information systems as a problem. Although productive partnerships with community organizations have been developed and despite the fact that local-level consultations with stakeholder organizations had been conducted during the development of the 1998-1999 LMDA Business Plan, stakeholders as well as front-line HRCC staff felt that consultations and partnerships with "grass roots" community organizations still need to be improved. Moreover, there is a need to increase awareness of the LMDA and EBSMs at the community level.
The EBSMs were generally regarded as flexible and responsive to local and client needs. Interview and focus group respondents had some reservations, however. In particular, respondents identified a need to further refine and adapt the programs to the unique economic needs of PEI (e.g., many seasonal industries, relatively high unemployment, and low annual earnings), to harmonize LMDA and social assistance programming, and to better serve and inform clients about available programs. The perceived strengths of Human Resources Development Canada (HRDC's) approach to delivering the EBSMs include a cooperative and positive effort on the part of highly experienced HRCC staff, flexible and decentralized program delivery, and a reasonable amount of paper work for clients/funding recipients. On the other hand, some clients and stakeholders perceived that service delivery is inconsistent from one HRCC to another and that the service from HRCC staff (and the HRCC environment as a whole) is unwelcoming and lacking in empathy.
In the survey, clients indicated being most satisfied with the quality of education or training they received and with the knowledge of Employment Counsellors, but comparatively less satisfied with the quality of referral services and with the information available to help them choose suitable programs. In addition, LMDA programs and services are being successfully delivered in both official languages as intended. Only a small minority of survey respondents (three percent or 28 clients) indicated that they were not able to get program information in their preferred language.
The federal-provincial partnership has been working reasonably well, though some "growing pains" are still being experienced. Both federal and provincial managers identified several strengths of the partnership and the co-management approach. In particular, these respondents perceived a high degree of cooperation between partners and noted that co-management facilitates collaborative decision-making, mutual understanding and coordination of federal and provincial initiatives. They acknowledged that the partnership does take a lot of effort and compromise, however. For example, front-line HRCC staff observed that the partnership has been difficult and frustrating at times, and that the degree of cooperation between the two levels of government could have been better in the pilot of Skills Development. Moreover, as already noted, most respondents observed that co-management adds another layer of bureaucracy and complexity to the LMDA, resulting in delays in project approvals.
Many HRDC staff have been reluctant to embrace the new co-management approach due to their concern that program delivery may eventually be fully devolved to the Province, which causes them anxiety over their job security. Although most senior managers felt that the LMDA is generally compatible with broader government objectives, provincial key informants asserted that the EBSMs need to be further adapted to better match the objectives of the provincial government.
The results of the accountability target attainment computations indicated that the EBSMs exceeded both the 1998/99 return-to-work and unpaid EI benefits targets. Despite exceeding the return to work target, questions were raised about whether or not all returns to work were being captured by the information systems.
Qualitative evidence of impacts on participants indicated favourable employment outcomes, if not immediately, then expected in the long term because of positive skill and psychological (e.g., self-confidence, self-esteem) impacts. Still, less than one half of survey respondents said their EBSM intervention was important in attaining their current job although this reflected a more positive perceived impact relative to the comparison group. Employers consulted in focus groups agreed that skill and psychological impacts are occurring, although a number said they were often unable to retain their wage-subsidy workers because of financial difficulties.
Quantitative evidence from the survey indicated that at this stage of evaluation, only Self-Employment led to consistently positive employment and income-support outcomes. TWS and Training/Feepayer (TFP) increased the likelihood of being employed for 12 consecutive weeks. EAS, TWS and JCP produced negative earnings outcomes. More conclusive evidence of EBSM impacts on employment and income support will be available at the summative stage of evaluation.
Results were mixed for perceived employer impacts. On the one hand, employers in the focus groups said that TWS relieved some employers of cash-flow pressures in hiring and that training unskilled workers enabled some businesses to be sustainable and even competitive. On the other hand, some employers were unable to retain workers because of cash flow problems and others said the EBSMs would be unable to channel workers into sectors and occupations where they were in demand. In addition, some employers in certain areas (e.g., seasonal, information technology, new/small businesses) did not think the LMDA as implemented could meet their specific needs because of perceived shortcomings in the rules. Some employers believed the EBSMs would be insufficient to effect a change in attitudes in PEI with respect to valuing training and dependence on seasonal industries and income support. Finally, some employers had problems with the exclusion of certain groups (e.g., non-EI eligible unemployed persons) who would have been able to fill their needs.
For impacts on the community, some key informants and focus group participants felt it may be too early to address this issue. Among those who were able to, there were mixed results. Some respondents felt the LMDA may have helped in addressing short-term needs of communities and Islanders, but there was concern that the exclusion of non-EI eligible persons may limit the LMDA's ability to address long-term needs. Others mentioned that, to truly benefit the province, there needs to be greater emphasis on interventions providing labour market development than on those providing job creation and wage subsidies. On the other hand, some respondents spoke of the benefits of the LMDA for the community, including the extension of the "shoulder" season, community learning centres, lifelong learning and adult literacy training measures, the technology mentoring program, lower social assistance (SA) caseload, and partnerships between communities and the government.
A review of the qualitative and quantitative evidence indicates that there were few clear rural-urban differences in terms of perspectives on the LMDA and in its impacts. With respect to the latter, multivariate analysis indicated that the EBSMs had positive impacts on rural residents in terms of the likelihood of seasonal employment, the percentage of weeks employed and three consecutive months of employment, but were beneficial for urban residents in terms of full-time employment. In addition, they produced negative effects for rural residents in terms of earnings. We also attempted to contrast the views of focus group participants located in urban, rural and isolated rural centres, but were again unable to observe any clear pattern.
|
The Management Committee of the Canada-PEI Labour Market Development Agreement (LMDA) has reviewed the Formative Evaluation of the Canada-Prince Edward Island Labour Market Development Agreement and is very pleased to have received this feedback. The Committee will use the evaluation results to improve its decision-making processes and to enhance labour market programs and services available to Islanders.
The Formative Evaluation focuses on the partnership model initiated to design and manage the LMDA and on the Employment Benefits and Support Measures (EBSMs) available to Employment Insurance (EI) eligible clients. A greater portion of the data collected concentrated on the five employment benefits which are those programs designed for short term outcomes. Qualitative information was also gathered on the support measures, such as the Local Labour Market Partnerships (LLMP). The evaluation provides a preliminary review of LMDA implementation based on information collected from December 1998 until October 1999. Further patterns and program outcomes will be captured in the summative evaluation to follow.
Generally, the formative evaluation concludes that LMDA programs are relevant to the labour needs of Islanders, communities, employers and growth sectors of industry. EBSMs can be used to extend the shoulder season of seasonal industries and to help diversify the Island economy into non-seasonal industries. As well, strengths of the partnership approach are confirmed, but some areas for improvement are also highlighted.
One identified strength of the co-managed model is the collective knowledge and perspectives the labour market partners bring to planning and decision making. The combination of HRDC's extensive experience in labour market programming and service delivery, and the Province's expertise in economic development has proven effective for labour market development on PEI.
For example, the focus on industry sectors with potential for growth has successfully created employment for many Islanders. By considering labour market development within the PEI economic context, labour market funding can be invested strategically.
A principle established in the LMDA to guide the labour market partners in the implementation of their partnership arrangement is to harmonize employment initiatives "...to ensure that there is no unnecessary overlap and duplication." The evaluation uncovered that having representatives from both governments on the Management Committee has served to decrease duplication of services. However, management continues to place programs strategically to minimize overlap and to address gaps in service delivery.
While cooperation among the labour market partners is significant, some difficulties with the management structure were identified in the evaluation. Concern was expressed that the co-managed model causes delays in some approval processes. As well, the on-going operation of this strengthened model was found to be resource intensive.
The Management Committee acknowledges that working together has been a learning experience, but that improvements have been made over time. Initial delays have been addressed and the Committee continues to strive towards timely and efficient processes.
Evaluation of the EBSMs demonstrates that targets are being consistently met. Multivariate analysis of EBSM participation results showed a difference in outcomes between rural and urban participants—rural clients achieved longer periods of employment while urban clients were more likely to move into full-time employment. In addition, positive qualitative employment outcomes, including self-confidence and esteem building, were attributed to EBSM participation.
Employers viewed the EBSMs as making their business more sustainable and consistent. However, they also felt that Targeted Wage Subsidies (TWS) could be more flexibly designed to enable the creation of sustainable employment. The seasonal nature of the Island economy must be considered in developing long-term labour market strategies. Employers, like the Management Committee, saw implementing EBSMs designed for the PEI context as crucial for long-term employment creation.
Clients, employers and service deliverers indicated customer service issues with quality and consistency of information and referral services. Again, this group A... indicated that the EBSMs need to be further refined and adapted to meet the unique economic needs of Islanders.
Efforts are being made to develop a balance of flexibility and consistency in these areas. Services are being enhanced through increased labour market and client tracking information, public awareness and customer service. PEI specific issues are considered when making funding decisions. The Management Committee continues to endeavour towards PEI economy specific programming.
The LMDA Management Committee accepts concerns expressed by service deliverers, employers and Islanders in general for the labour market needs of those ineligible to receive EBSMs. Individuals who are not EI eligible, in particular "...employed workers, youth and those with weak labour market attachments, do not have access to valuable programs which could improve their labour market outcomes. The situations of these Islanders need to be addressed, though not necessarily through this Agreement, to allow full labour force participation.
Results of this evaluation are consistent with the findings of the Pan Canadian Formative Evaluation of Employment Benefits and Support Measures completed in 1998, but with a more detailed focus on Island specific issues. The LMDA formative evaluation will provide a solid foundation for the upcoming summative review as the partners strive towards positive labour market impacts and outcomes through this co-managed model.
The Formative Evaluation of the Canada-PEI Labour Market Development Agreement has provided federal and provincial partners with valuable recommendations for improving the effectiveness of labour market programs and services for EI eligible Islanders. The Management Committee is committed to providing relevant and high quality labour market programs and services designed to fit the specific needs of Islanders.
|
The Canada/Prince Edward Island Labour Market Development Agreement (LMDA) is a comprehensive agreement that specifies the details of an arrangement made by the federal and Prince Edward Island (PEI) governments regarding the delivery of employment benefits and support measures. Under the terms of the agreement, the federal government retains responsibility for the delivery of Employment Insurance (EI) benefits; for aspects of labour-market development that are national in scope such as inter-provincial mobility, sector councils and national labour market information and exchange; and for the financing and delivery of benefits and measures and the National Employment Service (NES). This is a co-managed agreement where Canada and PEI share responsibility for the design, management and evaluation of benefits and measures for the eligible population of PEI.
This document presents the results of a formative evaluation undertaken of the Canada/PEI LMDA. Formative evaluations are carried out fairly early in the life of an initiative and, as such, focus on issues related to design and implementation rather than on definitive results, which are more appropriately assessed at a later point in time (in what is called a summative evaluation).
In late 1995 and during 1996, several actions by the federal government in the area of employment insurance and labour market development were announced that extended options for provinces/territories to take an expanded role in delivering human resource development services. On November 27, 1995 the Prime Minister announced federal withdrawal from labour market training. This commitment was made in response to the expressed desire by provincial/territorial governments to assume greater control over employment development programs. Subsequently, in May 1996 the federal government made formal offers to the provinces/territories to assume such responsibility (discussed below as it relates to PEI). In July 1996, it enacted Bill C-12 — the Employment Insurance Act — that called for agreements with the provinces/territories regarding the administration of active labour market measures for the unemployed and the evaluation of these programs. This Act commits the federal government to work more closely with the provinces/territories in labour market activities, and authorizes it to enter into agreements on the design and delivery of new active employment benefits and measures.
Part I of the EI Act provides for EI benefits for people temporarily out of work and maintains the national system of temporary income support for EI claimants while they search for employment. The Government of Canada, through Human Resources Development Canada (HRDC), continues to be responsible for providing Employment Insurance income support and for delivering labour market development programs consistent with national interests. Part II of the Act provides for a range of active employment benefits and measures that assist unemployed people in returning to work as quickly and efficiently as possible. These measures, which can be tailored to meet individual needs and local labour market realities, are intended to provide unemployed Canadians with opportunities to obtain and maintain employment and to be productive participants in the labour force.
Under the terms of the Canada/PEI Labour Market Development Agreement (LMDA), entered into effect on April 26, 1997, the federal and provincial governments agreed to share responsibility for active labour market measures for the unemployed in the province. In accordance with this Agreement, based on the co-managed model —wherein Canada and PEI have equal roles in design, management and evaluation — the federal government retains responsibility for the financing and delivery of Employment Insurance benefits, measures, and services. The legislative changes to the federal EI Act reflect a movement away from the purchase of blocks of seats in various labour market training programs and towards client-driven training programs (i.e., Skills Development), job creation and other measures integrating the unemployed into the labour market, thereby increasing the important strategic linkage of labour supply and demand.
The arrangement between Prince Edward Island and Canada is to take place according to the following principles:
Further, the signatory ministry for the Province is the Department of Development,1 which signals the intent of the Province to tie labour market development to areas of the economy where the potential for job growth exists.
The 1997/98 primary results targets of the program were that 4,400 EI claimants would participate, 1,800 claimants would be returned to employment and $7.8 million would be saved on the EI account.2 In the first year of the agreement 2,170 clients returned to employment and $4.1 million was saved on the EI account.3,4 Primary results targets for 1998/99 were therefore adjusted to 2,000 clients returned to employment and a $4 million savings to the EI account.5 The 1998/99 LMDA budget was $21.0 million, an increase from the 1997/98 budget of $17.8 million. The available funding increased to $22.8 million in 1999/2000.
From the initial year of the LMDA, the Canada/PEI LMDA Business Plan has been tied to priority sectors for economic growth and critical issues affecting labour force development. In the first two years of the LMDA, the priority sectors were as follow: aerospace; information and communication technology, tourism, small business, and primary industries. The critical issues were literacy and the employed worker. For 1999/2000, the LMDA's Business Plan has been modified to expand the priority sectors of industry to include Food and Value-Added Products. As well, a new level of strategic focus has been added: strategic support initiatives are sectors where shorter-term support may enable them to become priority sectors and include the food industry, the film industry, forestry products, wood manufacturing, the craft industry, and community development. Critical issues affecting labour force development have been expanded to address not only literacy and the employed worker, but also skills training (through Skills Development6), seasonality, administrative client information data, and labour market information.
The Canada/PEI LMDA is designed to serve those people defined as EI clients under the federal EI Act and not to meet the needs of all members of the labour force. According to the federal EI Act and therefore in the LMDA, an EI client is defined as an unemployed person requesting assistance who:
The latter two groups are referred to collectively as reachback clients.
The Agreement outlines five employment benefits and three support measures, as described below. More complete descriptions of the Employment Benefits and Support Measures (EBSMs) are presented in Appendix A.
Each of these EBSMs, except Research and Innovation, was included in the evaluation, although Local Labour Market Partnerships does not have participants and hence was not part of the survey of participants (described in Section 1.5.)
A review of published literature and government documents was carried out as part of this evaluation to provide a description of the general labour market and social conditions in PEI and a context for the evaluation of the LMDA. The literature and document review also identified specific employment issues and challenges to be addressed by the LMDA. An overview of the conclusions of this technical document is presented in this section. Prince Edward Island is divided into three counties which each encompass roughly one third of the province's 5,660 square kilometres. Just over half of residents live in Queens County, while a third live in Prince County and about 15 percent live in Kings County. Queens County, located in the central portion of the Island, has four centres with a population greater than 1,000 people: Charlottetown (32,530), Stratford (5,870), Cornwall (4,290) and Miltonvale Park (Milton-Winsloe Area, Queens County), (1,240). On the western side of the Island, Prince County has three such centres: Summerside (14,530), Kensington (1,380) and Alberton (1,080). Kings County, comprising the eastern part, has two centres with a population greater than a thousand people (Montague with 2,000 and Souris with 1,290).7 The Island's urban population is about 44 percent, primarily comprised of residents of Charlottetown and Summerside, which means that more than half of the population resides in rural and isolated rural areas which has implications for dependence on seasonal industries and limited other employment opportunities.
PEI's economy depends largely on seasonal industries such as agriculture, tourism, fishing and primary industries. The province has seen modest economic growth in the latter part of the 1990s in these areas, particularly in tourism, as well as in others such as sales and exports. Seasonal employment provides many jobs for Islanders, and as such, the majority of seasonal workers depend on EI benefits in the off-season. Other industries are emerging, however, such as aerospace, telecommunications, and information technology. Higher-skilled positions are expected to grow substantially in the next 20 years. Since these sectors are more likely to lead to long-term employment than seasonal industries, it will be important to invest in training the work force for these areas. In addition, small business has proven to be successful on PEI and merits labour market investment to help this sector grow.
The literature review pointed to a number of negative economic indicators. For the timeframe examined in the literature review, PEI was experiencing higher than average unemployment, as shown in Exhibit 1.1. In 1998 the unemployment rate in PEI was 13.9 percent compared to 8.3 percent for Canada (and 17.9 percent for Newfoundland, 12.1 percent for New Brunswick, and 10.7 percent for Nova Scotia). As well, employees make lower than average earnings and the number of involuntary part-time workers in PEI is also high compared to other provinces.
A contributing factor to unemployment, lower earnings, and underemployment is a lack of education. Compared to the Atlantic Provinces and Quebec, the educational attainment levels on PEI tend to be somewhat higher, but compared to Ontario and points west, education levels tend to be lower (Exhibit 1.2). Although post-secondary enrolment of Islanders has been increasing since the 1980s, particularly at community college, education levels on PEI are not perceived to be keeping pace with the demand for skilled workers, especially in information technology and other high technology sectors. Further, the youth population, which typically has higher levels of education than the rest of the population, has been declining in PEI over the last 15 years. A fear of the out-migration of PEI youth and a resulting "brain drain" may be well-founded. In 1981 the population aged 15 to 24 numbered 23,500, but declined to less than 20,000 in 1996.8 The number of youth aged 15 to 24 is projected to decline further, to less than 18,000 in 2011 and to less than 16,000 in 2021. Youth aged 20-24 have the highest out-migration rates of all age groups. In addition, out-migration is highest for individuals with high levels of education, including university and college.
A number of specific employment issues and challenges to achieving employment goals for Islanders were identified in the literature review. For example, low literacy levels in some areas/populations in the province pose obstacles to training and retraining workers for the emerging knowledge-based industries and should be addressed in conjunction with job specific training. The Atlantic provinces have the lowest rates of literacy in Canada, as shown in Exhibit 1.3. In PEI, 40 percent of the adult population have difficulty with the written word in varying degrees.9 Data from the 1995 International Adult Literacy Survey, presented in Exhibit 1.4, show averaged literacy levels in Prince Edward Island to be lower than Canadian averages. Low literacy skills levels may in part be explained by the fact that in Canada, rural populations typically have weak literacy skills. In addition, two or three decades ago many Islanders left school early to pursue careers in fishing and farming. With the decline of these industries and the transformation of the Island society and economy by technology, there are large numbers of displaced adult workers.10
| Level One: Great difficulty reading or non-reading | Level Two: Reading, but not well | Level Three: Reading but have problems with more complex tasks | Level 4/5: High literacy levels | |
| Canada | 11 | 26 | 44 | 20 |
| Atlantic Provinces1 | 11 | 32 | 42 | 15 |
| Quebec | — | 22 | 56 | 14 |
| Ontario | 17 | 24 | 39 | 21 |
| Western Provinces2 | — | 28 | 41 | 25 |
| From Reading the Future: A Portrait of Literacy in Canada. Statistics Canada, 1996. Note 1: New Brunswick, Newfoundland, Nova Scotia and Prince Edward Island. Note 2: Alberta, British Columbia, Manitoba and Saskatchewan. — Sample size too small to produce reliable estimates |
||||
| Level One: Great difficulty reading or non-reading | Level Two: Reading, but not well | Level Three: Reading but have problems with more complex tasks | Level 4/5: High literacy levels | |
| Canadian average | 17.2 | 25.4 | 34 | 23.3 |
| PEI average | 28.6 | 34 | 29.4 | 7.7 |
| Note 1: Projections based on IALS data, Statistics Canada, 1997; table from N. Smitheram, Local Community Needs Analysis (March 1998). Note 2: The percentages noted here are the mean results of the Quantitative, Document and Prose results for each level. |
||||
According to A Strategy for Adult Literacy/Education in Prince Edward Island (1998; p. 3.), "functional literacy skills are a minimum requirement for adults to fulfil their roles as workers, parents and community members." Low literacy skills pose a formidable obstacle to training and retraining workers for the emerging job market, particularly in knowledge-based industries. Training, skill advancement, the attraction of economic investment, and the introduction of technological innovations are all affected by the literacy and education levels of the workforce.
Since mid-1996, a number of literacy initiatives have been established in the province.11 These include the harmonization of federal and provincial funding to target literacy and basic education needs, a three-level adult learner curriculum, the establishment of a Diploma of Adult Education from the University of Prince Edward Island, the establishment of a Prior Learning Assessment Recognition to aid community learning centres, the development of provincial standards for adult educators; the availability of an assessment service to all residents, and the development of program standards and evaluation mechanisms.
With respect to seasonality, efforts to train seasonal workers in the off-season or to extend the shoulder seasons will have to take into account the history of some workers who have tended to work only the minimum number of days required to collect EI. There also are concerns that training programs may produce an over-supply of skilled labour in communities where an insufficient number of opportunities for employment exist. Further, there tends to be skepticism about the government promoting skills upgrading and whether this leads to direct career improvement. Finally, a lack of public transportation in PEI presents an employment barrier to some residents.12
From the literature review it appears that the LMDA and its Business Plan closely match the economic realities of PEI, providing support for established primary industries, seasonal industries and emerging high tech industries. The LMDA recognizes the importance of seasonal industries in PEI such as tourism and has provided funds to assist extending the tourist season or develop employment in the off-season of industries like fishing. At the same time, however, the LMDA acknowledges that investment has been made and should continue to be made in those industries that are more likely to lead to long-term employment, such as aerospace, telecommunications and small business.
Education and training (including adult basic education, literacy training, post-secondary education and specialized workplace skill training) have figured prominently and should be expected to continue to do so in labour market development plans for PEI, according to the needs documented in the literature review. Indeed, from the beginning, the LMDA's Business Plan has made education and training a priority, attempting to link training as closely as possible to employability. The LMDA thus provides support for training facilities, programs and individual program entrants. Knowledge-based skills training in the priority industry sectors are particularly emphasized. The shift from Purchase of Training to Skills Development reflects an effort to make training more relevant to individuals' employment and career paths. In addition, the LMDA recognizes literacy as a critical issue and the Adult Basic Education Literacy initiative has been developed to address the literacy gap.
Exhibits 1.5a and 1.5b profile the population of participants in LMDA programs and services in Prince Edward Island, as well as the population of all clients eligible for LMDA programs during the time period under investigation.
Overall, as the dark bars of Exhibit 1.5a indicate, LMDA clients were slightly more likely to be male than female (52 versus 48 percent respectively). Participants were also slightly more likely to be over 30 years of age than under 30 years of age (53 versus 47 percent respectively). Relative to the population of all eligible clients, participants were more likely to be women (48 versus 43.9 percent) and to be under the age of 30 (47 versus 28.4 percent). Thus, a disproportionate number of women and younger clients are making use of LMDA programs and services. Participants were also slightly more likely than the population of all eligible clients to be unilingual French or bilingual (2.6 versus 1.4 percent).
Few participants self-identified as being persons with disabilities (5.2 percent), Aboriginal (1.1 percent), or members of a visible minority group (0.6 percent). One in ten clients, however, self-identified as a social assistance recipient. Relative to the population of eligible clients, participants were more likely to be members of an equity group.
Generally, as Exhibit 1.5b (darker bars) indicates, PEI LMDA participants had not engaged in post-secondary education. A sizeable majority (81.5 percent) of participants had not been formally educated beyond high school. Just less than one third of all participants had between grade 7 and 11 as their highest level, while half of all participants had completed high school.13 Five percent of participants had university degrees, most of whom had completed undergraduate study only. Relative to the population of eligible clients, participants were more likely to have completed high school and to have engaged in post-secondary education.
The most current administrative data available indicate that participants' average annual incomes were increasing prior to their intervention, while the average annual EI benefits they have received declined (see Exhibit 1.6). Between 1992 and 1997, the average total income of participants increased from $15,399 to $17,171. At the same time, the average value of Employment Insurance benefits received steadily declined, with average benefits received in 1997 more than $600 lower than benefits received in 1992. Social assistance (SA) benefits showed an equivalent drop over this same period of time, falling from an average of $5,464 in 1992 to $4,768 in 1997. While the general trend in income, EI and Social Assistance (SA) benefits was the same for the population of eligible clients as was observed for participants, income and EI benefits were higher for eligible clients and SA benefits tended to be lower for eligible clients.
Between April 1997 and October 1998, 6,180 interventions were recorded.14 The most frequently used program was Purchase of Training, which accounted for 53.2 percent of all interventions recorded. Targeted Wage Subsidies (19.6 percent) and Employment Assistance Services (18.5 percent) were other commonly used interventions. Job Creation Partnerships (5 percent of all interventions) and Self-Employment (3.8 percent of all interventions) were participated in less frequently (Exhibit 1.7).
There are two client groups eligible for EBSMs: individuals who are current EI claimants and reachback clients (as described in section 1.2, those who have claimed regular EI within the past three years or claimed maternity/paternal benefits within the last five years). The administrative data file could not provide a reliable indicator of EI claimant/reachback status of participants and so an indicator was constructed based on participants' EI profiles.
In Exhibit 1.8, we observe the extent to which the share of claimant and reachback participants varies across programs, based on the administrative data for the full population. The results in row one indicate, first, that reachbacks represent roughly one quarter (23.1 percent) of all participants. Second, there is a good deal of variability in reachback share between different interventions, with the largest share of reachbacks as a proportion of all program participants observed for the Self-Employment program (44.7 percent), and the lowest observed for Enhanced Feepayers (8.9 percent).
In Exhibit 1.9, we present the distribution of reachbacks according to the length of time that elapsed between the end of the most recent EI claim and the start of the most recent intervention. The largest proportion of reachbacks participated in their intervention within three months of the end of their EI claim (37.3 percent) and this fact was particularly true of JCP (56.4 percent) and TWS program participants (48.9 percent).
The finding that reachback clients had participated in some of the programs, such as the Enhanced Feepayer program, is perhaps problematic because the rules and guidelines for the administration of these programs are such that reachbacks should not be able to access these EBSMs. For example, Enhanced Feepayer participants are by definition clients who receive EI benefits while paying for their own training. The fact that just fewer than one in ten Feepayers were found to be reachback clients implies that there may be problems in the delivery of this program such that reachback clients are accessing these services. Alternatively, current information systems may not capture the necessary information for all or part of the participant population in order to accurately identify program participants as claimants or reachbacks. Considering the difficulties that were found to exist with tracking and monitoring systems (as described in Section 3.6), the latter explanation would seem to be the more plausible.
The objectives of the formative evaluation of the Canada/PEI LMDA were fourfold:
An evaluation framework was prepared which identified a set of issues to guide evaluation activities. Given that this was the formative stage of the evaluation, the research objectives were focused on examining the relevance, implementation, design and delivery and the shorter-term outcomes of the EBSMs as well as the federal-provincial partnership. In broad terms, the purpose of the formative evaluation was to supply information indicating what improvements, if any, are required to the LMDA design, delivery and infrastructure that would permit it to better meet its objectives. The formative evaluation was also aimed at providing a foundation for the future summative evaluation by collecting baseline information on participants (e.g., early outcomes, sociodemographic profile information). In addition, the evaluation examined the extent to which current management information systems are sufficient for the conduct of the summative evaluation. The summative evaluation will provide information on the longer-term impacts of the EBSMs and will respond to questions regarding the cost-effectiveness of programs.
In this section, we briefly describe the five methodologies or lines of evidence employed in the formative evaluation. Multiple methods were utilized to capitalize on the advantages of each. Specifically, quantitative methods (i.e., surveys) were used to obtain early results which could be applied to conclusions about the full population of participants in the EBSMs and qualitative methods (i.e., interviews, focus groups) were used to gather particular in-depth perspectives on the various research questions. More details on each of these five methodological components of the evaluation are provided in Appendix C.
A total of 30 in-person interviews were conducted with key informants. These individuals were identified by the Joint Evaluation Committee as possessing specific expertise or experience which would allow them to respond to the evaluation questions. As most of the interviews involved two respondents, a total of 52 key informants were consulted. The average interview was one and a half hours long. Interviews were conducted in Charlottetown (24 interviews), Summerside (four interviews) and Montague (two interviews) with informants from these cities as well as some respondents from other areas of the Island. The interview locations were set to be convenient for most respondents.
The key informant interviews fell into the following three categories:
In presenting the results of the interviews, majority views are generally reported. The nature of qualitative research, however, is such that some individuals were interviewed for their particular expertise or as a representative of a larger group. In these cases, the individual perspective may be reported even though the finding is coming from only one person and not a number of key informants.
A total of 12 focus groups were conducted with four types of participants: HRCC and provincial delivery/front-line staff (one group); stakeholders16 (two groups); clients (six groups), and employers (three groups as well as one of the stakeholder groups which included some employers). The combined stakeholder/employer group was targeted to the francophone community and was conducted in French, and all other discussions were held in English.17 At least one group with clients was held in each of the five Human Resource Centres of Canada (HRCC) regions of the Island — in Charlottetown, Summerside, O'Leary, Montague and Souris. The sixth client focus group was targeted to francophone clients and was offered in French. These various client focus groups were conducted in order to assess the views of clients in the urban, central rural and more remote rural areas of the province and in the different employment situations in each area.
A review of program-related documentation and literature provided context for the formative evaluation of the Canada/PEI LMDA and assisted in addressing some of the evaluation issues, including the compatibility of the EBSMs with the objectives of HRDC and the PEI government and the relevance of the EBSMs to the documented needs of individuals, employers and communities in PEI.
Members of the Canada/PEI LMDA Joint Evaluation Committee provided most of the written sources consulted, including the LMDA 1998-99 Business Plan (June 1998), Local Community Needs Analysis (March 1998), and Securing Our Future: An Employment Strategy for Prince Edward Island (December 1998). In addition, we consulted the pan-Canadian EBSM formative evaluation report18 and drew references from it in this report where appropriate. Additional information was gathered by conducting a bibliographic search for PEI-related information on labour market issues and programs, including a search utilizing the Internet. An electronic labour market database provided quantitative data.
Socio-demographic information about participants as a group and as individual program users was analysed in order to capture a fuller understanding of LMDA program and service participants in Prince Edward Island. The administrative data analysis examined profile data for participants involved in LMDA interventions between April 26, 1997 and October 31, 1998, as well as income data from 1992 to 1997 for this same group. All demographic information, including self-identified variables, was derived from the National Employment Service System (NESS) file received from HRDC (PEI) in May 1999. Following receipt of the final data for the population of LMDA participants, analyses were conducted to determine where gaps in the administrative data existed and the extent to which the lack of certain types of information impacted on the ability to evaluate EBSMs and to provide tracking, monitoring and accountability information.
The participant data file was developed from one file containing information for 5,409 participants who participated in LMDA employment programs and services at any time between April 26, 1997 and October 31, 1998, and five administrative data files containing additional information about these cases. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 3,744 individuals. For all groups except EAS and Enhanced Feepayers, there were not enough cases available to obtain a three to one ratio of sample to survey completions; thus for an expected total of 1,164 survey completions, a total final sample of 2,483 cases was drawn from the data file of 3,744 program participants.
The comparison group sample was drawn from a file of EI claims that were active in 1998 and dormant EI claims (i.e., individuals who were eligible for EI but who had not activated their claim) from 1994 to 1998. This produced a file of 41,549 claimants from which to draw the comparison group sample. The comparison group data file was matched to the participant data file such that the comparison group sample frame reflected the proportion of participants who participated in an LMDA program during each of three equal time periods between April 26, 1997 and October 31, 1998, as well as the proportion of participants who were at different stages of their EI claim at the end date of their intervention.
Both the participant and comparison group surveys were designed in the fall of 1998 and were reviewed by the Joint Evaluation Committee (JEC) in January 1999. Fieldwork for the participant survey began on April 5, 1999 and was completed on June 10, 1999. Fieldwork for the comparison group survey began on May 10, 1999 and was completed on June 11, 1999. A more detailed description of the participant and comparison group survey methodologies is presented in Appendix C, section 3.
The integrated findings of the formative evaluation of the Canada/PEI LMDA, incorporating all lines of evidence in the research, are presented in this document. Chapters Two to Seven present the findings pertaining to the major evaluation issues addressed in this research, as introduced in Section 1.4. Finally, in Chapter 8, the evaluation conclusions are drawn and recommendations presented.
|
Evaluation findings pertaining to the relevance of LMDA programs are presented in this chapter, drawing on qualitative evidence from the key informant interviews, focus groups and literature review. At this formative evaluation stage, it is essential to establish if the Employment Benefits and Support Measures (EBSMs) continue to be relevant to the needs of industry, employers, communities and individuals in PEI. The identification of any weaknesses in these regards will give LMDA management the opportunity to fine-tune the programs so that they are as relevant as possible and LMDA funds are put to the best use.
The major findings are as follows:
The priority sectors targeted for labour market development in the 1998/1999 LMDA Business Plan included tourism, aerospace, information/communications technology and small business, while primary resource industries such as agriculture and fishing have received an ongoing commitment. These priorities were identified by the Department of Development and other provincial government departments, based on the economy of the Island. These priorities were elaborated on and critical issues identified through a series of community consultations with external partners at the five Human Resource Centres of Canada (HRCC) sites across the Island. The literature and document review (which was conducted in the present evaluation) illustrated that these priority sectors are highly relevant to the PEI economy.
The literature indicated that the LMDA has prioritized those industry sectors that greatly support the PEI economy and which have the best potential to grow, providing more jobs for Islanders. For example, the ongoing commitment to the primary resource sector recognizes the fact that agriculture and fishing are PEI's first and third largest industries. However, given that these industries are highly seasonal and will not provide substantial year-round employment, it stands to reason that other industries have been targeted for increased development. The literature indicated that tourism, PEI's second largest industry, has good potential for growth despite being seasonal in nature. The Confederation Bridge, increased visitation from Quebec, predictions that tourism will grow due to an aging baby boom population, and an extensive tourism infrastructure upon which PEI can build are all reasons why targeting tourism for growth is relevant to PEI's economy. Aerospace and telecommunications are growing rapidly in PEI and employment growth is anticipated to be high. Given that both industries require highly skilled labour, the LMDA's mechanisms to provide financial support for training for these industries responds, in theory, to the needs of both industry and the provincial economy.
While the literature review found small business to be common on PEI and more successful than in the rest of the Atlantic provinces, businesses tend to be quite small (two thirds employ fewer than five employees). Even though this sector may need labour market investment to help it grow, it is not clear precisely how the LMDA Business Plan will address the needs of small business. Several participants in the focus group and interview components of the evaluation made this observation (for example, HRCC staff members indicated that the Business Plan does not adequately describe what it will do for small business) and other participants expressed some dissatisfaction with the capacity of the LMDA to assist small business.
Based on the feedback from interviewees and focus group participants, senior management from HRDC and the PEI government, as well as delivery staff, felt that the LMDA programs are relevant to the targeted industry sectors. Nevertheless, some respondents felt that other industry sectors should not be excluded nor should one expect a dramatic shift in PEI's economy away from seasonal employment; in their opinion, seasonal industries are a reality for PEI.
Industry stakeholders were not as confident that the right industry sectors had been identified as priorities. Some felt that tourism should not be targeted because the majority of jobs in this industry are short-term and underpaid. Those stakeholders in regions of PEI where there are no high-tech businesses believed that the LMDA priorities are not relevant to their local economy. For example, some employers thought that the LMDA places too much emphasis on information technology, while a stakeholder argued that the construction industry should have been targeted because it contributes a substantial number of jobs and significant wealth to the economy.19
With respect to the relevance of the EBSMs to the needs of the targeted industry sectors and employers, Skills Development (formerly Skills Loans and Grants or SLG) was seen to be relevant by provincial senior managers because of the flexibility with which it can be used to fund various types of training. However, business stakeholders worried that private sector training programs (usually short-term programs of two to six weeks in duration) would lose their funding as a result of the implementation of Skills Development, which they perceived (incorrectly) to apply to one and two-year programs.20
Some gaps in the LMDA with respect to training in general were identified. Staff felt that the LMDA Business Plan lacked provisions for industry-specific training and interventions that will upgrade the skills of currently employed workers (especially women) and move them from minimum wage jobs into better paid positions. According to both front-line staff and management, the secondary and post-secondary education systems in PEI are not sufficiently well coordinated with the needs of the labour market. Industry stakeholders and employers were of the opinion that training continues to be given in fields where there will not likely be jobs available for the trainees (i.e., not all training is relevant to the needs of employers and emerging industries). Small business is one area where stakeholders felt funding for training has been insufficient, as already noted. As well, youth representatives wanted to see the Province develop priorities for training programs and implement career counselling services in high schools and universities to help channel students into the targeted industries.
Targeted Wage Subsidies (TWS) was the EBSM most frequently commented on by interviewees and focus group participants. This EBSM was deemed to be helpful to small business by HRCC staff, but only in the later stages of a business once employees are hired and not during the start-up stage. This was not seen as an improvement to pre-LMDA programs in which a new business could qualify for assistance before hiring employees. While the Self-Employment (SE) program exists to help EI clients establish their own small business, there currently is no mechanism to assist non-EI clients with the start-up costs of establishing a business. As already noted, however, the suggestion that the EBSMs should assist non-EI clients represents a misunderstanding of the LMDA's mandate to serve EI-eligible clients.
TWS met employers' and industry stakeholders' expectations in general. Non-governmental organizations have used TWS to assist them in organizing festivals to attract tourists, which has a positive impact on the tourism industry and other small businesses. TWS was even considered critical to some employers, who felt that it would have been difficult for them to pay full salaries to new employees going through on-the-job training. One limitation of TWS for employers is the eligibility requirement that restricts participation to EI clients. The focus groups revealed that employers were also frequently dissatisfied with the skills and work ethic of employees eligible for wage subsidies. They also found that retaining employees after the initial job training can be a challenge because participants' new skills may lead them elsewhere or because some participants do not want to stay on for full-time work. Employers in information technology thought that wage subsidies should be available for a longer duration because of the lengthier training period (a minimum of one year) associated with this industry.
EI eligibility criteria are considered by respondents in this evaluation to be the principal limitation of the LMDA in addressing the needs of employers and the targeted industry sectors. The federal EI Act restricts the EBSMs to EI claimants and this is thought to be a huge limitation to developing the PEI labour force to meet the challenges of the new industries such as information technology. Many interviewees, including both federal and provincial managers and staff, believed that the capacity of the LMDA to develop the priority industry sectors is constrained by the lack of mechanisms to assist currently employed individuals to upgrade their skills and to assist employers in upgrading or retraining their employees.
Based on the feedback from members of LMDA Committees and Working Groups, HRCC managers and staff, and provincial administrators and project officers, the LMDA and EBSMs are relevant to individual and community needs for the segment of the PEI population that is EI eligible. Training interventions and wage subsidies were deemed the most useful types of EBSMs, and the flexibility of LMDA programs was highly valued. However, the overriding concern of interviewees is that the EBSMs under the LMDA are restricted to a particular client group — EI claimants — and this restriction is a significant limitation to developing the PEI labour force. This view was shared by both federal and provincial government respondents.
Most front-line HRDC staff consulted in focus groups also felt that LMDA programs were of some relevance to PEI, just as the previous HRDC programs had been. HRCC managers and staff who were interviewed perceived that to address long-term needs of communities and Islanders, however, LMDA programs should be redesigned to provide assistance to currently employed workers, the unemployed not on EI, and employers. LMDA programs should also address the education and literacy needs of all Islanders. Key informants offered suggestions for change. For example, Job Creation Partnerships (JCP) would be more relevant to Islanders if they accommodated initiatives that lead to transferable job skills for participants (rather than being restricted to initiatives that lead to immediate jobs). Targeted Wage Subsidies (TWS) require more flexibility to increase the subsidy in instances where employers can hire clients for more than one season. To take advantage of the flexibility of the EBSMs and use them to achieve "made-in-PEI programs," some cultural change is required at HRDC so that staff are not afraid to be as creative and flexible as possible within the formal program guidelines.
Stakeholders consulted in interviews and focus groups had a number of positive things to say about the EBSMs. SE and JCP were perceived to be effective and very relevant to the needs of participants. Skills Development (formerly SLG) was perceived to be very relevant to clients' training needs as well as to private training institutions because it has made the training market more competitive. TWS was found to be extremely beneficial for both community organizations and the workers who gain work experience with these organizations. Literacy programs and the community learning centres were seen to be highly relevant to communities and Islanders. Employment Assistance Services (EAS) has been the most relevant program for organizations that provide services to the disabled. Community education organizations were pleased that the LMDA had formalized agreements with community learning centres across the Island in terms of establishing a long-term vision and planning.
Stakeholders also made numerous recommendations they thought would improve the relevance of the LMDA to PEI. One stakeholder felt that Local Labour Market Partnerships (LLMP) proposals should not have to be submitted to Charlottetown, because officials may not fully understand local dynamics and the potential of LLMP projects, and because this causes delays.21 Francophone stakeholders thought that the needs of francophone communities, which they believed to be different from those of anglophones, should be targeted. In particular, there should be room in the EBSMs to support small-scale training and literacy programs delivered in French (though these stakeholders acknowledged that access to courses in French has improved with Skills Development). Life-skills training would make programs like TWS more relevant to client participants who are often low skilled and inexperienced at keeping a job.
Focus group respondents would like to see more specialized training offered through SE for entrepreneurs. In addition, programs that help integrate youth into the labour market of PEI would be relevant to communities to keep educated youth from leaving the Island and stem the "brain drain." Stakeholders also perceived that the LMDA would be more consistent with long-term community labour market plans if the majority of funds were spent on human resources development rather than on wage subsidies.
Like many respondents, stakeholders were also concerned about the eligibility criteria of LMDA programs. For example, representatives of disability organizations felt that eligibility criteria and assistance should be more flexible to recognize that clients with disabilities may have difficulty working full-time or for extended periods and may require continual support to maintain employment. There is also a fear within the literacy community that LMDA programs such as Skills Development will reduce the number of people who qualify for literacy training due to the EI eligibility criteria.22 In addition, some stakeholders asserted that the restriction of EBSMs to EI clients has limited the effectiveness of the programs and that the proportion of unemployed who qualify for EI has been reduced as a result of changes to the Employment Insurance Act. Lastly, representatives of community education organizations felt that while LMDA funding meets the needs of EI eligible client groups such as seasonal workers, it does not meet the needs of youth and employed people. Similar results were obtained in the 1998 national EBSM evaluation.23 In this study, community organizations expressed concern that the EI eligibility criteria are too restrictive and consequently the needs of people with weak attachments to the labour force, such as youth and persons with disabilities, are not being met.
Some employers consulted in focus groups felt that the LMDA has the potential to produce programs relevant to the needs of Islanders because one of the partners, the Province, is more familiar with local needs. However, other employers felt that there has been a lack of flexibility and HRDC staff need to be more flexible in their approach.
For clients, the biggest frustrations lay not with the EBSMs per se, but with the perceived lack of jobs in PEI and the low pay of those jobs available. Some acknowledged that a lack of experience, a lack of skills, being out of the system too long, and technological change were other barriers to finding employment. As such, training programs (especially in technology) and work experience programs like TWS would be potentially relevant to Islanders to some degree. However, even though many of the clients consulted in focus groups were appreciative of the assistance they had received, they stressed that their main barrier to employment has been a lack of decent employment opportunities in PEI.
The literature review lends credence to this perception by clients. It documented that, despite relatively high rates of job growth and labour force participation as well as falling unemployment in recent years, PEI still had the second highest rate of unemployment among Canadian provinces in 1998 (13.9 percent compared to the national average of 8.3 percent). Moreover, based on 1996 figures, the average annual earnings of Islanders were the lowest in Canada ($19,333 compared to the national average of $27,089). The PEI economy depends largely on seasonal industries, which leave many Islanders unemployed during the off-season. Given the relatively high proportion of involuntary part-time workers in PEI, however, it appears that many Islanders do want full-time work. Although comparatively low levels of literacy and education in the province (as well as in other Atlantic provinces) present a challenge for the development of new industries such as information and communications technology, PEI has been proactive in meeting this challenge with the 1996 Strategy for Adult Literacy/Education.
The LMDA has targeted both PEI's traditional seasonal industries for expansion and off-season training initiatives and its new industries such as information technology, communications, and aerospace, which have more potential to lead to year-round employment. In this manner the broad goals of the LMDA appear to closely match the economic realities of PEI, that is, the need to enhance the seasonal industries while simultaneously developing other year-round industries.
The delivery of training programs during the off-season may, however, face certain challenges according to the literature review. Seasonal employees are often residents of rural communities where other employment opportunities do not exist. Finally, insufficient literacy and education levels often need to be addressed before low skilled workers can be trained for higher skilled and longer-term employment. To help address this need, adult literacy/education has been identified as a critical issue for the LMDA.
Key informant interviews revealed that those consulted were divided on the issue of how to deal with the seasonal nature of employment in PEI. The majority of respondents felt that the LMDA should focus on reducing the Island's reliance on seasonal employment, while the remainder felt that extending the duration of seasonal employment, in the tourism or fishing industries for example, is an appropriate strategy for the LMDA. Reactions of the focus group participants were very similar, the majority of whom thought the goal of the LMDA should be to reduce PEI's dependence on seasonal employment and "get people working year-round."
First, views on the strategy of expanding seasonal industries and employment are discussed. Both HRCC managers and staff and provincial administrators felt that where feasible, seasonal employment should be extended as long as employees are trained during the off-season to work in other industries. Training in the off-season, however, should be considered a stepping stone for clients to move into other employment opportunities with more long-term potential. Both federal and provincial managers and staff thought that programs such as TWS could be used to develop special projects to extend the "shoulder seasons." Extending the duration of seasonal employment could create positive economic spin-offs, such as higher incomes, a decrease in Social Assistance payments and an increase in taxes paid.
Many community and industry stakeholders also felt that extending the seasonal economy is an appropriate strategy for the LMDA. It was noted that the seasonal economy is a reality from which PEI will not be able to escape. For employers consulted in the focus groups, the objective of the EBSMs should be to get people working year-round, but they thought that seasonal industries would continue to be important to PEI's economy and would require skilled workers just as non-seasonal industries would. Nonetheless, stakeholders in general cautioned that the LMDA must find ways to develop businesses that are complementary to seasonal industries (e.g., production of crafts and souvenirs to be marketed in the tourist season) and that help individuals find jobs to complement their seasonal employment. This means that training should be provided to help seasonal workers become multi-skilled and more mobile. Some stakeholders also suggested that the EI eligibility requirements restrict the capacity of the EBSMs to expand the tourism industry, which requires better qualified workers than those available in the pool of EI claimants.
LMDA committee members and provincial staff felt that other medium term performance measures should be incorporated in the upcoming summative evaluation and future LMDA accountability requirements in order to capture the positive impacts of extending the work season in seasonal industries. For example, the number of jobs created, the contribution to GDP, and annual growth in the PEI economy might be good indicators of the success of the LMDA in expanding the seasonal economy.
Second, turning to the opposing view on how to deal with the seasonal economy of PEI, a larger portion of key informants argued that the objectives of the LMDA should be to reduce seasonal employment and focus on structural solutions to diversifying PEI's economy. This view was shared by stakeholders, provincial LMDA committee members, provincial administrators, and HRCC managers and staff. For these respondents, extending seasonal employment has limits and is not realistic for business. New industries (such as knowledge-based industries) and other economic activities, which can provide year-round employment should be developed instead. Youth stakeholders pointed out that extending seasonal industries would not assist students returning to school to improve their summer employment earnings (wage subsidies would be more useful). In addition, representatives of the literacy community felt that extending seasonal employment would limit the amount of time workers have available for training.
According to federal LMDA committee members, provincial staff and stakeholders, PEI needs to change the historical pattern of dependence on EI (which will require attitudinal change as well as structural change), and to focus on increasing the skills of Islanders. According to provincial personnel, many of the EBSMs are not appropriate for breaking this cycle of dependence and for helping Islanders to find year-round employment. This sentiment was echoed by most program delivery staff and stakeholders consulted in focus groups. For example, it was noted that the labour market programs are not suitable for overcoming the seasonal nature of the economy because they do not provide incentives for seasonal workers to engage in off-season work.
A number of key informants, including both federal and provincial managers and staff, suggested that the focus on expanding seasonal industries in PEI has created a conflict between two of the principal LMDA performance objectives — to help individuals obtain employment and to decrease expenditures on EI benefits. They predicted that this focus on seasonal employment would increase expenditures on EI benefits.
Senior managers on LMDA Committees and Working Groups, particularly those at the federal level, perceived that the LMDA programs are fairly well harmonized with other federal and provincial programs and that there are no major problems of duplication and overlap. These key informants felt that this harmonization is due largely to the LMDA co-management structure whereby all players sit at the same table and can coordinate their respective efforts. The delivery of the adult basic education/literacy initiative through community learning centres was offered as an example of good harmonization of the objectives of many provincial departments (Health and Social Services, Education, Development) and the LMDA.
There are some minor problems of duplication or overlap between LMDA programs and other federal and provincial initiatives. For instance, there is some overlap between the LMDA Self-Employment program and Atlantic Canada Opportunities Agency (ACOA) and provincial programs aimed at assisting individuals to start a business.24 Provincial officials felt that the Province's entrepreneurship programs and job creation programs should be better harmonized with the LMDA. They also perceived there to be a lack of coordination between the Canada Student Loan and PEI Student Loan programs and the LMDA Skills Development program. The two different philosophies of Skills Development and the student loan programs were thought to work at cross purposes concerning student debt levels. Moreover, often students can receive both types of assistance.
Below the senior management level and particularly among provincial government respondents, the views are less positive and respondents saw much room for improvement in the harmonization of related federal and provincial programs. HRCC managers and staff thought there was duplication in wage subsidy programs. They also thought that federal and provincial programs are not compatible when it comes to providing capital/loans/start-up costs to new businesses. HRDC's goal to serve clients most in need is not compatible with the goals of industry to hire the most qualified people or to upgrade the skills of employees.
In the 1998 national EBSM evaluation, although minimal duplication was observed, the potential for duplication between TWS and SE programs was suggested. Consistent with the present evaluation findings, the national study noted that in the Atlantic provinces, Industry Canada, ACOA and provincial departments all have self-employment programs.
Similarly, in the present evaluation, provincial respondents felt that there is much duplication of employment programs among the various departments in the federal government (e.g., Industry Canada, the Department of Fisheries and Oceans, ACOA) and in the provincial government (e.g., Department of Health and Social Services, Department of Development, Department of Tourism). In their opinion, some employers are "playing the system" and obtaining funding from several different programs, or appealing to the Island federal member of parliament (MPs) or provincial member of the legislative assembly (MLA) offices for support if HRDC does not approve their request for funding based on LMDA criteria. What is needed to avoid overlap, according to these respondents, is the complete specification of what each department (federal and provincial) is funding and their associated performance targets, so that all players can see the total picture.
Many front-line delivery staff at HRDC (consulted in the focus groups) perceived there to be a lot of overlap between the EBSMs and other federal and provincial programs, as well as inefficiency in delivery. For instance, federal and provincial staff are currently unable to share information on the same client who has received both EI and Social Assistance, and clients often have to see various staff of different orders of government to address basic needs.
Stakeholders consulted in interviews and in the focus groups generally thought the LMDA programs were complementary with other related programs. Interviewees, however, perceived some minor duplication. They thought that overlap exists between: the provincial Employment Enhancement Program and the LMDA Targeted Wage Subsidies program; programs for new businesses offered by ACOA, HRDC, third-party deliverers and Enterprise PEI; Canada Student Loans and the LMDA Skills Development program; and federal and provincial youth programming. Some stakeholders added that the Province's efforts to get Social Assistance recipients onto EI and then into training programs work at cross purposes with the LMDA results targets.
Clients and employers consulted in focus groups were generally unaware of any duplication or overlap of programs from other orders of government or other organizations, with the exception of employment programs offered in the community.
Evaluation evidence from the key informant interviews, focus groups and literature review indicates that, for the most part, the EBSMs are relevant to the needs of the targeted industry sectors, employers, communities and Islanders. Due to the restrictive program eligibility criteria imposed by the EI Act, however, respondents perceived that some individuals in need of assistance were not being served. This view appears to stem from a widely held misperception that the mandate of the LMDA encompasses the entire PEI labour force, when in fact it is designed to serve EI clients only. In particular, many respondents felt that small business development, skills upgrading for currently employed (and underemployed) Islanders, and the needs of non-EI eligible clients are not being adequately addressed. For the seasonal PEI economy, most respondents noted that the EBSMs are relevant in that these programs can be used to extend the shoulder seasons of seasonal industries (e.g., the fishery, tourism) and, more importantly, to help diversify the economy and develop new year-round industries (e.g., aerospace and information technology). On the issue of the complementarity of LMDA and other programming, federal and provincial officials at the senior management level generally felt that the EBSMs are fairly well harmonized with other programs, though at the middle management and front-line levels many respondents perceived there to be work needed to resolve issues related to duplication, overlap and a lack of coordination among federal and provincial programs (e.g., wage subsidies, self-employment and youth programs).
|
Views on the degree of success of Labour Market Development Agreement (LMDA) planning and implementation, as expressed in key informant interviews and focus groups, are presented in this chapter. It is important to identify the strengths and weaknesses of LMDA implementation to date so that needed improvements can be identified. This information will allow LMDA management to respond to problems early in the implementation phase so that the Agreement and associated Employment Benefits and Support Measures (EBSMs) are ultimately as successful as possible in achieving their objectives.
Key findings are as follows:
In the view of many key informants with both the federal and provincial governments, the major strength of the LMDA implementation has been the high degree of cooperation, communication and collaboration among all LMDA players, including good federal-provincial cooperation in the co-management of the Agreement. Government managers and staff as well as stakeholders cited some examples of successful LMDA initiatives, including the adult basic education/literacy initiative, the aerospace initiative, Self-Employment and Targeted Wage Subsidies. In addition, front-line HRCC staff noted that they have a fairly good understanding of their role in the delivery of the EBSMs because these benefits and measures are similar to the former Human Resources Development Canada (HRDC) programs.
Key informants perceived there to be numerous difficulties with the implementation of the LMDA to date. Chief among federal and provincial managements' concerns was a lack of reliable, timely information (i.e., useable labour market information and data tracking Employment Insurance (EI) clients) to support LMDA planning, management and evaluation. In addition, managers expressed concern about the somewhat excessive administrative requirements and the slow, cumbersome approval process due to the fact that major decisions under the co-managed LMDA require negotiations and agreement between federal and provincial partners on the Management Committee and on the Operations Committee.25
HRCC managers and staff had some unique concerns about the LMDA implementation. HRCC managers are having some difficulty adjusting to the federal-provincial approval process and to the fact that they no longer have the authority to approve programming exceeding $75,000 in their local area. On occasion, HRCC managers have felt that these decisions, made by the Management Committee, have not been ideally suited to the needs of their area.
For front-line delivery staff at HRCCs, the administrative requirements (and the added level of bureaucracy) associated with the LMDA have created some confusion and have increased their workload. Also, the interviews and focus groups indicated that many HRCC staff are anxious about the possibility that EBSM delivery will eventually be fully devolved to the provincial government (as it has been in some other provinces), and they are concerned that this could result in the loss of their jobs. As one key informant put it, "full devolution hangs over our heads." Front-line employees' anxiety about their job security is a serious issue, considering their crucial role in the delivery of services at HRCCs. It will be important to ensure that staff anxiety does not interfere with the delivery of services under the EBSMs.
In addition, front-line staff in the focus groups made the following observations about LMDA implementation: not enough emphasis is placed on proper employment counselling (to screen clients and point them in the right direction) and on encouraging clients to take responsibility for their own progress toward employment; and a belief that the Management Committee is not sufficiently aware of client service issues at the "grass roots" level and of the possibility of overlap between EBSMs and other programs.
Similar to the government respondents, stakeholders expressed some confusion and frustration over the slow, complicated application and approval processes associated with the co-managed LMDA. Many are uncertain about the respective responsibilities of each level of government. Moreover, consistent with the findings of the 1998 national EBSM evaluation, key informants representing a variety of community/stakeholder organizations (i.e., youth, persons with disabilities, business and education) observed that awareness of the LMDA is quite low in communities, suggesting that communications and promotion of the EBSMs could be improved. In focus group discussions, some stakeholders perceived that the public and business community on the Island are somewhat cynical about the ability of government to "get it right" with respect to delivering the EBSMs and meeting clients' needs. For instance, employers using Targeted Wage Subsidies expressed a need for better matching of clients with suitable occupations, and for more background on a client's work history so they can better assess the client's capabilities and suitability. Also, some stakeholders were of the opinion that there were insufficient local-level consultations regarding the LMDA (despite the fact that consultations had been held in all local areas for input into the development of the LMDA Business Plan).
Both federal and provincial key informants observed that there are several productive partnerships between government and community organizations related to the LMDA. Some factors that have hindered the development and maintenance of good partnerships were also identified, however. In particular, LMDA-related changes in the decision-making and approval procedures for program funding, downsizing and staff turnover at the Human Resource Centres of Canada (which can result in inconsistent program delivery), and changes to the Employment Insurance Act have created some confusion and difficulties for community organizations in their efforts to work with government. Also, due to some confusion about the respective responsibilities of the two levels of government, community organizations have sought assistance from both federal and provincial offices, sometimes attempting to play one off against the other to their advantage (though it was noted that this was also done prior to the LMDA).
Although some stakeholders acknowledged that there have been productive partnerships and that local community organizations (including the francophone community) have been consulted for their input into the LMDA Business Plan, several stakeholders identified some problem areas that hinder partnerships. Many stakeholders (representing youth, industry associations, community education groups, and people with disabilities) identified a need for further improvements in local-level consultations and more community input into LMDA planning and implementation. Similarly, in the national EBSM evaluation, an overall finding was that community partners felt they had been consulted on local labour market plans but not on the mix of benefits and measures to be delivered. In the present study, organizations representing persons with disabilities also called for more flexibility in EBSM eligibility criteria to accommodate different client groups with unique barriers to employment. The adult basic education/literacy community felt inadequately consulted by government, and was not supportive of the fact that Holland College was awarded the contract to administer community-based programs for the delivery of adult literacy training.
In the focus group discussions, some stakeholders observed that good partnerships have existed with HRDC for some time and that they have continued under the LMDA. On the other hand, others perceived that there is room for improvement in partnerships between HRDC and "grass roots" community organizations, an observation also made by some front-line HRCC staff. In the view of stakeholders, partnerships have been hindered by: a resistance on the part of HRDC to treat stakeholders as equal partners; a lack of trust between government and community groups; LMDA "growing pains" and confusion concerning areas of responsibility; a lack of communication with stakeholders regarding program objectives; and insufficient bilingual resources/services at HRCCs, which has meant that some francophone stakeholders are not always served in their preferred language.26 In the national EBSM evaluation, community organizations also indicated that more effort needs to be devoted to the development of partnerships.
In the view of both federal and provincial key informants, the primary resource issue with the LMDA pertains to the lack of technological resources to support the information systems needed for proper planning, management and evaluation. With respect to human resources, it is perceived that federal downsizing has created somewhat of a staff shortage for delivering the EBSMs — an observation also made in the 1998 national EBSM evaluation. The number of staff was reduced on the promise of more technology (to compensate for having fewer staff), but the technology has yet to be delivered. Moreover, the existing technology does not have the capability of real-time management information as requested by the LMDA Management Committee nor will it permit easy input, tracking or evaluation.
Regarding financial resources for the EBSMs, federal respondents felt that, although the budget is very tight, it is probably sufficient at present. On the other hand, provincial officials argued that more financial resources would be helpful, particularly for training. In addition, some provincial respondents questioned why LMDA funds lapsed in one province could not be transferred to another province (like PEI) where the funds could be put to good use. There are no mechanisms at this time to permit the interprovincial transfer of funds.
Stakeholders representing persons with disabilities felt that EBSM funds provided for hiring staff and client monitoring are insufficient, and that the one-year program contracts are too short for them to plan and provide a sustainable service to clients. These respondents preferred the previous three-year contracts. In addition, community education representatives perceived that HRDC has less flexibility to fund community organizations under the LMDA than it had in the past.
In the key informant interviews, senior and middle managers as well as staff with both the federal and provincial governments agreed that the current information monitoring systems are inadequate and that the difficulty in obtaining reliable, practical information on the labour market/labour force and on EI clients' progress impedes proper planning, management and evaluation of the LMDA. The LMDA includes a provision for the creation of a working group to identify the information needs and system connectivity requirements for implementation of the Agreement, but no funds were specifically allocated for these purposes. As an illustration of this issue, some provincial respondents described a case where they asked for some LMDA performance measurements and received three different estimates within a one-year period. It was also discovered during the development of the survey samples that a reliable flag to distinguish client and reachback participants does not exist in the administrative database. As a result, provincial respondents have little confidence in the validity of the currently available performance measures due to shortcomings in HRDC's information systems. Key informants noted that these limitations with information systems are not unique to PEI, but are a problem across the country.
Several aspects and causes of this problem were identified, including the following:
A number of third-party stakeholder organizations (representing youth, self-employment clients and the training community) claimed in interviews that they do collect some useful data on client characteristics and outcomes. Other stakeholders perceived that there are problems with the information monitoring, however. For instance, it appears that some organizations, depending on the nature of their activities, are required to complete more extensive monitoring than others and this can result in inconsistencies. Some stakeholders also argued that LMDA monitoring should be more comprehensive, incorporating the results of follow-up surveys with clients, narrative reports of client progress, information on the full range of interventions in which clients participate (not just those delivered by HRDC), and a broader range of outcome measures (e.g., educational achievements, lifestyle changes and volunteer as well as paid employment).
On balance, given the complexity of implementing the LMDA, things have gone reasonably well so far. The high degree of cooperation and collaboration among all LMDA government partners, the delivery of some successful initiatives (e.g., adult basic education/literacy initiative, aerospace initiative), and HRCC staff's good understanding of their role in EBSM delivery were all noted as successful aspects of LMDA implementation to date. On the other hand, a lack of reliable, timely information to support LMDA planning and management, excessive administrative requirements, and somewhat of an HRCC staff shortage were regarded as key weaknesses that will need to be addressed. In addition, although many respondents acknowledged that productive partnerships with community organizations had been developed, stakeholders as well as front-line HRCC staff felt that consultations and partnerships with "grass roots" community organizations need to be improved. Moreover, there is a need to increase awareness of the LMDA and EBSMs at the community level.
|
In this chapter, findings pertaining to program design and delivery are presented, drawing on both quantitative and qualitative evidence. Numerous issues related to the design and delivery of the Employment Benefits and Support Measures (EBSMs) were examined in order to determine if the programs are sufficiently responsive to client needs and if they are being delivered as intended and in a satisfactory manner.
Findings from the key informant interviews, focus groups and survey of clients indicate the following:
Managers with both the federal and provincial governments perceived that the LMDA and EBSMs are reasonably flexible and responsive to the needs of clients and communities. The wide range of benefits and measures, which has a significant degree of flexibility built into them, have been utilized to fund a variety of initiatives considered to be worthwhile. Moreover, the citizen-centred approach to service delivery through the five HRCCs, coupled with the fact that HRCC managers in each local area have spending authority up to $75,000, helps to ensure that the EBSMs are responsive to local needs. Similarly positive results regarding the local-level flexibility of programs across the country were obtained in the 1998 national EBSM evaluation.
These key informants also expressed some reservations, however. For instance, provincial officials argued that better harmonization between LMDA programming and the provincial Social Assistance programming is needed to improve responsiveness to local needs. They also observed that programming through the five HRCCs, while responsive locally, is inconsistent from one region of Prince Edward Island to another and this causes confusion for clients (a problem also identified in the national EBSM evaluation). Of course, it must be recognized that there are trade-offs — the more local flexibility and responsiveness, the less consistency in program delivery from one local area to another.
Stakeholders representing industry and education felt that LMDA projects such as those under Job Creation Partnerships would be more beneficial to clients if they moved beyond short-term assistance/job experiences and adopted a longer-term focus whereby clients would be helped to develop job skills that would serve them well over the long run. On the positive side, stakeholders identified literacy programs and Targeted Wage Subsidies as highly responsive to client needs.
Most of the employers consulted in focus groups expressed positive views about the responsiveness of the EBSMs. For instance, participants were satisfied with the Targeted Wage Subsidies (TWS) because they enable employers in seasonal industries to extend their season. This also began to satisfy client needs for additional weeks of work. Involuntary part-time workers (i.e., those who would prefer to work full-time) on PEI were more likely to have looked for full-time work than involuntary part-time workers in all the other provinces (Statistics Canada, Labour Force Update, Catalogue No. 71-0005-XPB, p. 19-20). In addition, the Self-Employment benefit was viewed as relevant and responsive to needs, as long as the clients start their business in an industry where there is not too much competition. Some employers argued, however, that the EBSMs could be better adapted to the needs of individual clients. For example, older clients who have worked in the same industry for decades but who now find themselves needing to be retrained for a new type of work have difficulty making this transition. The EBSMs need to be flexible enough to accommodate such unique needs.
In focus group discussions, many clients complained about inadequate promotion of the LMDA programs and the fact that some HRCC staff could be unwelcoming and lacking in empathy. Some clients also argued that insufficient funds are available to help cover travel costs to attend training programs. On the other hand, a minority of clients noted that their expectations for service and for LMDA-related training programs had been met.
Similar to the findings pertaining to information monitoring systems (see Section 3.6), both federal and provincial senior managers felt that the currently available information on the PEI labour market and labour force is inadequate and of limited use for LMDA planning and management. At the client service level, although a great deal of labour market information (LMI) is collected and available at the HRCCs, the retrieval of useful, focused information and analysis to address clients' questions appears to be a problem. In the view of some HRCC managers, the available LMI is adequate but it is not being well utilized at present because it is difficult to keep staff trained and current in the optimal use of these data (e.g., how to retrieve LMI from the Internet).
Senior management with both the federal and provincial governments observed that a strength of the EBSM delivery to date has been the positive outlook and cooperative effort of all involved, including front-line HRCC staff who are highly experienced in delivering these sorts of programs. There has been a productive focus on searching for the best ways to deliver and improve client service through the EBSMs.
Stakeholders consulted in interviews and focus groups identified a number of strengths of EBSM delivery: the emphasis on skills training; simplified application procedures for Local Labour Market Partnerships (LLMP) funding; a reasonable amount of paperwork involved in HRDC programming and follow-up; and decentralized service delivery through the five HRCCs, with much productive activity at the local community level. In the group discussions, some employers also noted an appropriate amount of paperwork (e.g., in Targeted Wage Subsidies). In addition, employers expressed satisfaction with the flexibility in the EBSMs to accommodate the realities of seasonal employment and business cycles, as well as flexibility in the number of hours employees must work to qualify for wage subsidies. Moreover, some employers praised a few exceptional HRDC staff who have exhibited creativity in their LMDA programming.
Although the clients in focus group discussions tended to concentrate on their complaints about service, a few respondents did express satisfaction with the service provided by employment counsellors and case managers.
Some weaknesses or problems with HRDC's delivery of the EBSMs were also identified in the key informant interviews and focus groups. Federal respondents felt that the difficult adjustment required of some HRDC delivery staff (e.g., having to do EBSM-related jobs that would not have been their first choice), coupled with staff anxiety over the possibility of EBSM delivery eventually being devolved to the Province and the associated threat to their job security has been a problem to date. Provincial key informants identified the inconsistency in service delivery among the five HRCCs on the Island, the apparent poor communications between HRDC senior management and front-line delivery staff, and the negative impression some clients have of the HRCCs (e.g., that HRCC staff are unreceptive and reluctant to provide continuing support to unemployed Islanders) as problem areas. Also, they observed that the EBSMs are not yet fully compatible with provincial objectives because they came as pre-existing programs designed and run by HRDC, and because HRDC staff who were experienced with the programs may be having some difficulty making the switch to the new LMDA approach. Front-line delivery staff acknowledged that there is some inconsistency from one HRCC to another and that the EBSMs need to be better adapted to the particular needs of PEI.
Stakeholders perceived some weaknesses with EBSM delivery too. These respondents criticized the lack of face-to-face contact with HRDC staff, the unwelcoming atmosphere in some HRCCs, the delays in processing some applications, and a lack of transparency in the decision-making such that applicants for EBSMs are not always given an adequate explanation for why some projects are accepted while others are rejected. Some stakeholder organizations had particular concerns. A representative of an organization for persons with disabilities argued that there are not enough employment counsellors in regional HRCCs to handle the volume of clients. In addition, representatives of industry associations expressed concern about the lack of continuity in the HRCC staff with whom they deal (which may be due partially to HRDC downsizing).
Like the other stakeholders, employers felt that the EBSMs are not adequately promoted. They observed that many clients and established businesses have poor awareness of available programs, though new businesses may be better informed because they tend to search for information on programs to help them get established. Employers also suggested that some HRDC employees might adopt a more proactive approach and develop a better understanding of the labour market in PEI so that good opportunities are not missed. Finally, some employers expressed dissatisfaction with the limited eligibility criteria and with the business hours of the HRCCs (e.g., it can be difficult to get service late in the day).
The clients who participated in focus groups appeared to be quite dissatisfied with several aspects of service delivery. In all five regions, clients' most common complaint was that they have been inadequately informed by HRCC staff of the range of benefits and measures available to them. As one participant put it, it is like "pulling teeth" to get any information out of HRCC staff. Clients consulted in the 1998 national formative evaluation were similarly frustrated about the lack of information on EBSMs. Another frequent criticism heard in the present evaluation was that some HRCC staff can be inconsiderate and lacking in empathy. Additional client concerns included the following: there is not enough follow-up from HRCC staff; funding provided under the EBSMs such as Skills Development is inconsistent from one region to another and even from different HRCC staff; there is too long a wait at reception; and some HRCCs have an excessive number of security features, creating a cold and unwelcoming ambiance.
In the view of many key informants, the transition from Purchase of Training, which was formally terminated on June 30, 1999, to the new Skills Development program (initially called Skills Loans and Grants) has been difficult. Both provincial and federal managers as well as front-line delivery staff felt that the process of implementing Skills Development has been slow and frustrating, with much resistance to change.
Both advantages and disadvantages of the new program were identified. On the positive side, federal and provincial key informants observed that the termination of Purchase of Training (and the introduction of competition from private training organizations) will probably force provincial educational institutions to improve the design and marketing of their courses, which will ultimately benefit clients. In addition, some stakeholders noted that Skills Development has the potential to be a positive and flexible approach for highly motivated clients with clear career goals, though they cautioned that the repayable loan portion may deter clients with limited job prospects beyond seasonal work. Also, stakeholders from the francophone community felt that Skills Development will help to improve the opportunities for clients to find courses in their preferred language. They felt that the former Purchase of Training approach had favoured community colleges that offer courses in English.
On the other hand, numerous difficulties were identified, including the following:
EBSM participants who responded to the survey were asked to indicate what resources or services they used on their own at a Human Resource Centre of Canada. As shown in Exhibit 4.1, respondents were most likely to indicate having used job bank kiosks or job board listings (64 percent) and information on training and education programs (36 percent). Respondents were least likely to have asked for resumé writing assistance or assessment services (14 and nine percent respectively).
When asked if they had spoken to HRCC staff to help them plan their strategy to return to work, roughly two in five participant respondents indicated that they had (42 percent), and 43 percent of these respondents indicated having developed an action plan with an employment counsellor. When asked if they had completed the activities in their action plan, fully 85 percent of these respondents indicated that they had. The most common reasons for non-completion of action plan activities among the remaining 15 percent of these participants (n=31) were that they found a job (37 percent), their action plan was currently in progress (21 percent) or they changed direction and were no longer interested in pursuing the action plan (10 percent). No clear pattern was observed with respect to subgroup differences by program type or by respondents' socio-demographic characteristics.
Participant survey respondents were asked to rate their satisfaction with various aspects of the programs and services they received. As shown in Exhibit 4.2, program participants were most likely to indicate that they were satisfied (responded with a 5, 6 or 7 on a 7-point scale) with the quality of education or training (88 percent) and the knowledge of the employment counsellors (87 percent). Participants were least satisfied with the quality of referral services (68 percent) and the information available to choose programs (69 percent). These latter results are consistent with the findings from client focus groups. As discussed in Section 4.4, clients' most common complaint about service delivery was that HRCC staff did not inform them adequately about the full range of available benefits and measures.
The following subgroup differences were observed:
An important evaluation issue involved the degree to which LMDA programs and services are being delivered in both official languages. Evidence from the formative evaluation suggests that this requirement has been very well met. All LMDA program participants in the survey were asked if they had been able to receive information about employment programs in the official language that they prefer, and only three percent of respondents indicated that they had not (n=28). Of these respondents, 58 percent had been unable to receive information in French and 26 percent in English (sixteen percent n=5] did not or could not respond to the question.)27
Respondents to the participant survey were asked if they had used employment-related services from an organization other than a HRCC, and just less than one third of participant respondents indicated that they had (68 percent indicated they had not used any such services). Respondents were most likely to indicate that they received job search or resumé writing assistance (12 percent), used job bank kiosks or listings (9 percent), had access to a computer (9 percent), and received information on training and education programs (9 percent) from the other organization (Exhibit 4.3). Subgroup differences are presented in Appendix E.
Of those respondents who used services from another organization, they were most likely to have received these services through friends or personal resources (40 percent), a community college or university (21 percent), or the provincial or federal government (12 percent) (Exhibit 4.4).
No clear pattern was observed with respect to subgroup differences by program type or respondent socio-demographic variables.
In commenting upon future challenges and directions for the LMDA, senior managers with the federal government stressed the need to address the problem of relatively low literacy (and computer literacy) among the PEI labour force, including both EI clients and currently employed Islanders; the need to regularly re-assess the highest priority industry sectors for development; and the ongoing challenge of achieving seamless service delivery — whether through the current co-managed approach or through the eventual devolution of service delivery to the Province (should this option be selected in the future). Some provincial senior managers suggested that an apprenticeship or internship program for PEI youth, coupled with a wage subsidy to introduce youth interns to jobs in targeted sectors, would facilitate the development of a skilled labour force while at the same time helping to reduce the migration of young people away from the Island. Provincial key informants added that the ultimate success of the LMDA must be assessed over the long term because it is unrealistic to expect the Agreement's objectives to be fully realized within a five-year period.
At the middle management level, key informants with HRDC made a number of suggestions for improving the LMDA: the Operations Committee's decision-making criteria for evaluating local projects should be more transparent and the turn-around time for project approvals should be reduced;28 the LMDA business plan should be better adapted to local community needs while ensuring that realistic expectations are communicated; information systems, connectivity and information exchange (i.e., between HRDC, the Province and third-party delivery organizations) need to be improved; LMDA roles and responsibilities need to be clarified and assigned for all HRDC staff; there is a need for public communications regarding HRDC's new role under the LMDA; and the federal and provincial governments need to become better informed about each other's mandates and parameters for implementing the co-managed LMDA. Provincial middle managers repeated the same suggestions regarding the need to improve information systems and performance monitoring and to better adapt the EBSMs to local community needs. They also suggested that programs (and the associated federal and provincial dollars) should be better harmonized to ensure the seamless delivery of services that address the needs of EI clients as well as currently employed Islanders, such as has been done with the adult basic education/literacy initiative. Others argued, however, that it is most productive at this point to accept the fact that the LMDA is intended to serve EI clients and that the needs of employed Islanders are best met through other program vehicles, either by themselves or in combination with LMDA funds.
In focus groups, front-line delivery staff reiterated some of the suggestions made by management. For example, they expressed a need for better LMDA information systems and for the sharing of information between federal and provincial staff. In addition, they identified a need to better coordinate the secondary educational system with the needs of the labour market so that PEI youth are encouraged to remain in school, receive high school education of relevance to the labour market, and gain some exposure to the working world (e.g., through field trips to businesses).
Representatives of stakeholder organizations made a number of suggestions. The most common suggestion was to better promote the LMDA and EBSMs (e.g., through the delivery of presentations to the public and the distribution of promotional materials to high schools, colleges and universities) so that the public as well as community partner organizations are better informed. Industry associations also expressed a wish to be informed about the LMDA evaluation results.
In addition, the following suggestions were made:
In focus group discussions, employers also identified a need for better promotion of the EBSMs to both clients and employers (e.g., through public presentations and employer association bulletins). In addition, employers expressed a need for HRCC staff to make themselves more available to clients, for more relevant training programs that are clearly linked to needs in the PEI labour market, and for more flexibility in the Targeted Wage Subsidies (e.g., offering an extended subsidized period with a diminishing percentage of wages subsidized and offering wage subsidies to help employed persons in establishing or changing careers).
Similar to other key informants, the clients consulted in focus groups suggested that they would like to be better informed about the EBSMs. Additional suggestions from clients included the following:
The evaluation findings indicate that, although the EBSMs were generally regarded as flexible and responsive to local and client needs, some reservations were expressed. In particular, respondents identified a need to further refine and adapt the programs to the unique needs of PEI (e.g., many seasonal industries, relatively high unemployment and low annual earnings), to harmonize LMDA and Social Assistance programming, and to better serve and inform clients about available programs. Also, as discussed earlier, the EI eligibility criteria imposed by the EI Act and delays in project approvals were thought to limit the responsiveness of EBSMs.
The perceived strengths of HRDC's approach to delivering the EBSMs include a cooperative and positive effort on the part of highly experienced HRCC staff, flexible and decentralized program delivery, and a reasonable amount of paper work for clients/funding recipients. On the other hand, some clients and stakeholders perceived the service from HRCC staff (and the HRCC environment as a whole) to be unwelcoming and lacking in empathy. In addition, clients did not feel they were adequately informed about all available programs. Also, the transition to the Skills Development program was perceived to be difficult, with much resistance to change.
Turning to the survey results, clients indicated being most satisfied with the quality of education or training they received and with the knowledge of employment counsellors, but least satisfied with the quality of referral services and with the information available to help them choose suitable programs. In addition, LMDA programs and services are being successfully delivered in both official languages as intended, though a small minority of survey respondents (three percent) indicated not receiving program information in their preferred language.
|
One of the objectives of this formative evaluation is to evaluate the partnership between the two orders of government in delivering the Employment Benefits and Support Measures (EBSMs). It is important to examine this partnership because it affects the management, delivery and ultimate effectiveness of the EBSMs. The views of key informants and focus group participants on the success of the federal-provincial partnership in the co-managed Labour Market Development Agreement (LMDA) are presented in this chapter.
The key findings are as follows:
Key informants among LMDA committees and working groups perceived a high degree of cooperation among the various federal and provincial players involved in the LMDA, such as those on the Management Committee and the Operations Committee. This has greatly facilitated the implementation of the LMDA. Similarly, managers and staff with both Human Resources Development Canada (HRDC) and the Province found that the LMDA has enhanced partnerships between the Province and HRDC and that these partnerships are improving as officials on both sides learn to adjust to the LMDA and changes to the Employment Insurance (EI) Act. Nevertheless, the partnership arrangement is not free of problems under the new decision-making structure. Some federal government respondents felt that the devolution of some responsibilities to the Province through the LMDA has created unnecessary complexity in the delivery of labour market programs. For example, HRCC managers now have to obtain approval from the Management Committee in Charlottetown for projects valued beyond $75,000, which requires more time for projects to be approved and implemented. As well, focus group discussions revealed that front-line delivery staff felt that the partnership between HRDC and the Province has been difficult and frustrating, with "people from Ottawa telling us what to do."
Senior managers and executives on LMDA Committees and Working Groups generally felt that the LMDA and EBSMs are compatible with broader government objectives. Federal government respondents thought that the LMDA and EBSMs are compatible with the priorities of federal departments such as Industry Canada (IC) and the Atlantic Canada Opportunities Agency (ACOA), in addition to the priorities of HRDC. In particular, they are consistent with the goal of preparing Islanders to work in the industries targeted for growth by the Department of Development. The same respondents felt that the LMDA is consistent with the mandate of HRDC in the area of community capacity building (e.g., providing good labour market information, steering youth into streams of education relevant to the labour market, strengthening adult education and encouraging youth to stay in school). They did feel, however, that there is a need to enrich HRDC's programming to ensure that there is a skilled labour force in the province to fill the jobs emerging in the industries targeted for growth, such as information technology and aerospace.
Provincial government managers, on the other hand, argued that the EBSMs still need to be refined to better match the needs of PEI and the objectives of the PEI government. In their opinion, even though the LMDA objectives were jointly developed by HRDC and the Province, the EBSMs came as pre-designed former HRDC programs. Some officials felt that, because there was no provincial department with significant expertise in labour market development, the Province let HRDC play a strong role in identifying objectives for the first year of the LMDA. As the Province learns, it will play an increasingly important role in developing the annual LMDA business plans and setting its own objectives. Other provincial officials identified a need in the LMDA for more emphasis on training and a better process of connecting training initiatives with job creation strategies and economic development. Finally, senior provincial officials expressed concern about the ability of the EBSMs to "get to the bottom of the problem," that is, to get people off the cycle of EI and into long-term employment. The overall LMDA strategy may require more flexibility to go beyond individual program criteria as well as broader objectives to develop the labour market beyond the parameters of the Employment Insurance Act.
All key informants felt that there are many strengths of the co-management approach. Senior managers felt that in particular, having both levels of government working together collaboratively should improve the quality of decisions and facilitate the coordination of all players' initiatives. Having all players sitting around the same table (i.e., the Management Committee and Operations Committee) facilitates mutual learning/understanding and the coordination of initiatives. The small size of PEI and the limited number of players have also facilitated the success of a co-managed approach.
HRCC managers and staff felt that the co-management approach has created checks and balances that would make for less duplication of services. Community stakeholders in general felt that the federal-provincial partnership and co-management were positive developments. Like the HRDC respondents, they also felt that this approach has the potential for single window delivery and would reduce duplication of services. Other important strengths of co-management, according to stakeholders, are the focus on the PEI economy as a whole, the understanding of business that the Province brings to the partnership, the sharing of common goals such as literacy training, and the recognition of the importance of partnerships with community organizations.
However, co-management is also difficult and takes time, effort and compromise on the part of both sides. This was the principal challenge identified by senior managers with the federal and provincial governments. A weakness noted by several HRCC managers and staff was that co-management adds another layer of bureaucracy and can slow down processes such as the review and approval of project applications.
In addition, HRDC staff members felt that there has been much confusion surrounding co-management and how it will impact on case management and accountability. Many HRDC staff have been reticent to embrace the co-management approach because they fear the eventual devolution of program delivery to the Province. HRDC staff also perceived that a lack of information sharing on Social Assistance clients and EI clients between federal and provincial departments is a barrier to achieving long-term solutions for both types of clients.29
Some provincial officials perceived (incorrectly) that a weakness in the co-management approach stems from the fact that EBSM delivery and the tracking of expenditures and results are entirely in HRDC's hands, while the Province is held accountable for achieving results.30 This problem is compounded by the fact that HRDC can make decisions on its own regarding any expenditures under $75,000. In addition, some provincial officials felt that another weakness of co-management is the "influence of Ottawa" on decisions or projects that may have an impact on the LMDA. In fact, the expenditures in the HRCCs are driven by initiatives laid out in the LMDA management plan, which is the vehicle that enforces accountability to the co-managed agreement at the HRCC level.
Although most stakeholders saw co-management as a positive development, they also voiced concerns. The most frequently identified weakness of co-management echoed both federal and provincial government officials' concerns, namely that the additional layer of bureaucracy (i.e., a new tier in the decision-making process) involves delays and communication problems between applicants and decision-makers. This added layer consequently results in slower service to the public. Representatives of industry associations indicated that they would prefer to deal with only one office rather than two orders of government. These observations suggest that co-management may need to be streamlined in order to achieve the goal of single-window service to the public. In addition, many stakeholders feared that co-management would allow local, provincial or federal politics to influence the decisions regarding projects. Some respondents suggested that better long-term results could be achieved if LMDA programs were co-delivered by the Province and HRDC.
Qualitative findings indicate that the federal-provincial partnership in the co-managed LMDA has been reasonably successful to date. Both federal and provincial managers identified several strengths of the partnership and the co-management approach. In particular, these respondents perceived a high degree of cooperation between partners and noted that co-management facilitates collaborative decision-making, mutual understanding and coordination of federal and provincial initiatives. They acknowledged that the partnership does take a lot of effort and compromise, however. Moreover, as already discussed in Chapter Three, most respondents observed that co-management adds another layer of bureaucracy and complexity to the LMDA, resulting in delays in project approvals. Although most senior managers felt that the LMDA is generally compatible with broader government objectives, provincial key informants argued that the EBSMs still need to be further adapted to better match the needs of PEI and the objectives of the provincial government.
|
Evidence is presented in this chapter on several measures of EBSM success to date based on multiple lines of evidence, including focus groups with participants and employers, key informant interviews, administrative data, and surveys of participants and non-participants. First, we consider the degree to which EBSMs have met accountability targets set out in the Canada/Prince Edward Island Labour Market Development Agreement (LMDA). Second, the results of descriptive analysis of EBSM outcomes are then presented. Finally, some of the EBSM outcomes revealed through the descriptive analysis could be explained by pre-existing differences in the characteristics of participants and non-participants and not necessarily by the EBSMs themselves, we present the results of multivariate analysis which is able to control for these differences.
Though early in the process, it is important to monitor and assess how the EBSMs are working and provide early warning signals if expectations are not being met. This information will allow LMDA management to make changes to the EBSMs that would facilitate their ultimate success. It is important to note, however, that EBSM participation may not result immediately in jobs, but may be expected to lead to employment and other outcomes in the future, beyond the scope of the current evaluation. The future summative evaluation will provide the opportunity to identify incremental impacts and results.
The findings follow:
To meet the objective of accountability for the EBSMs, the Canada/PEI LMDA, like all LMDAs with provinces and territories, specified results targets in three areas: active EI claimants served, returns to work, and unpaid EI benefits. As set out in the Agreement, these primary result measures are to be monitored and adjusted annually based on the past year's performance and changing circumstances. The purpose of setting the targets and checking to see if they are being met each year is to determine the extent to which the EBSMs are successful in meeting basic objectives of integrating EI clients into the workforce. Divergence from targets could signal the need for adjusting the EBSMs to better meet the needs of clients.
A component of this evaluation is to consider the extent to which the accountability targets are being met.33 In this section, the results of the assessment exercise are summarized, while Appendix C presents a detailed discussion of the methodology and the results of the analysis.
For the number of EBSM participants returning to work target, computations based on the administrative data indicate that the 1998/99 return to work target (2,000) was exceeded by almost 50 percent, reaching a level of 2,902 returns to work. In 1998/99, there were over $4.6 million in unpaid EI benefits, exceeding the target ($4 million) by over 12 percent.
It is noteworthy that between 1997/98 and 1998/99, the return-to-work target was increased while the unpaid benefits target was reduced. These changes likely partly due to the fact that, in the previous fiscal year, returns to work exceeded the target while unpaid EI benefits were well short of it. This change in targets indicates a principle of the Canada/PEI LMDA is being followed, whereby these targets are to be adjusted each year to reflect changing circumstances, particularly as experience and familiarity grow with time. The latter might also be reflected in the fact that returns to work rose considerably between the two fiscal years, indicating rising EBSM effectiveness. However, this also could be due to a growing economy providing increasing job opportunities.
Finally, other computations based on comparisons between the administrative data and the evaluation survey indicate that the administrative data do not capture all actual returns to work. Some of these differences can be partly attributable to the different datasets and methodologies used to compute the results. Once again, the interested reader is referred to Appendix C for a discussion of the different data sets and methodologies.
In this section, we present descriptive findings relating to the impacts of six EBSMs: Self-Employment (SE), Targeted Wage Subsidies (TWS), Job Creation Partnerships (JCPs), Employment Assistance Services (EAS), Training and Feepayers.34 Two approaches to measuring impacts are used in this section: (1) clients' own subjective ratings of the importance of the help they received in obtaining employment; and (2) objective measures of labour-market outcomes as revealed by participants' post-intervention status in a number of areas, including employment and attitudes, as compared to non-participants.
It is important to note that, in these simple (bivariate) descriptive analyses, observed differences in outcomes between participants and the comparison group cannot account for differences in the characteristics of the two groups. For example, if participants had greater education or were more highly motivated than non-participants, this could explain their possibly more favourable outcomes, irrespective of the role of the intervention. This is the reason that the descriptive analysis was followed by multivariate analyses, which can control for the pre-existing differences; the results of this multivariate analysis are presented in Section 6.3. Still, results of the descriptive analysis are presented here in order to provide basic information on a number of outcomes, overall and for various population sub-groups. Due to difficulties in interpreting the results of descriptive analyses, the results of a number of other labour-market outcomes, such as joblessness, job-search behaviour and utilization of income support, were examined in the multivariate modelling and are presented in Section 6.3. The interested reader is referred to Appendix E for descriptive results pertaining to these labour-market outcomes.
We start with participants' own assessment of EBSM importance and go on to objective observations of participants' post-intervention status.
Survey respondents were asked to indicate the extent to which they thought the assistance they received from Human Resources Development Canada (HRDC) was important in helping them to get a job. Twenty-two percent of participant survey respondents who were employed in the post-program period felt that the program was very important (responded with a 6 or 7 on a 7-point scale). Sixteen percent rated the HRDC assistance as somewhat important (responded with a 3 to 5 on a 7-point scale) and 61 percent rated the assistance as not important (responded with a 1 or 2 on a 7-point scale) in helping them to get a job. Training and EAS respondents were the most likely to feel that the employment program was very important in helping them find a job (29 and 23 percent, respectively), whereas TWS participants were the least likely to feel this way (12 percent).
It should be noted that this question was addressed only to those respondents who had a job in the post-intervention period and to respondents who were not continuers (i.e., Self-Employment participants who continued operating their businesses or wage subsidy participants who were hired on by the host employer in the post-program period). If we assume that these continuers would have provided a response of "very important" to this question had it been asked of them, the proportion of respondents who rated the HRDC assistance as very important rises dramatically among Self-Employment (from 16 to 78 percent), TWS (from 12 to 70 percent) and JCP participants (from 19 to 44 percent) (Exhibit 6.1).
While participant ratings of the importance of HRDC assistance are modest, a comparison of participant and comparison group survey responses reveals that, overall, participants are more likely than comparison group respondents to rate the services they received as very important in helping them to get a job (Exhibit 6.2). This implies an advantage for participants in LMDA programs relative to non-participants (note that participants were asked about HRDC services whereas non-participants were asked about employment services in general). Among EBSM participants, reachbacks were somewhat more likely than claimants to rate the HRDC assistance as very important. An examination of participant demographics did not reveal any significant sub-group differences.
Participants who continued in their wage subsidy jobs were also asked specifically about the importance of their wage subsidy program or job creation project in helping them to obtain the job with their program employer (Exhibit 6.3). JCP participants were more likely than TWS participants to rate their employment program as very important in helping them to get this job (68 versus 42 percent, respectively).
Qualitative evidence on impacts was also obtained from focus groups with clients and employers. Focus groups with clients revealed a number of perceived impacts that LMDA programs have had to date. To begin, the acquisition of skills through training has been instrumental in allowing some participants to find employment. Even for those who had not found employment at the time of the focus groups, a number mentioned that the programs had contributed to an improved sense of self-confidence, self-esteem and optimism that they will find employment because their job prospects are better. These findings are somewhat consistent with results of the Pan-Canadian Formative Evaluation which found that the perceived impacts of EBSMs were primarily related to skills acquisition and, even more importantly, to work experience — not necessarily jobs. This study also found that EBSMs had had an impact on client attitudes through increased self-confidence.
Self-reported positive benefits of wage subsidy programs include a salary increase for one respondent, as well as the acquisition of employment for a number of others. One factor that may contribute to post-intervention unemployment, noted by employers and participants alike, was that often employers do not keep employees on after the 12-week subsidy. Employers felt that they are often unable to do so because of financial restrictions.
In this section we present survey evidence on employment status and retention. It is important to reiterate that observed differences in outcomes between participants and the comparison group as presented in this section do not control for the pre-existing characteristics of the different groups. Once again, due to problems with interpretation (i.e., the inability to control for pre-existing differences between participants and comparison group members), the descriptive results for a number of employment outcomes, such as employment rates and employment stability, are not presented in this section. These employment outcomes were analyzed through multivariate modelling and are presented in Section 6.3. Again, the interested reader is referred to Appendix E for the descriptive results pertaining to these employment outcomes.
It should be pointed out before proceeding that a different pattern of results was observed between reachbacks and active EI claimants for many of the employment outcome measures (as shown here and in Appendix E). Specifically, reachbacks appeared to have more positive outcomes than active EI claimants. A possible explanation for this is a pre-existing strong labour force attachment among reachbacks which would mean greater employment "outcomes" for this group. In most cases here, the different findings between reachbacks and claimants, as well as between participants and the comparison (non-participant) group, can be attributed to positive employment outcomes specifically among comparison group reachbacks. This group would have received no form of assistance, whereas all of the participants and the comparison group claimants would have received some form of assistance, either through income assistance (EI), employment programming (EBSMs), or both. It may be that comparison group reachbacks' apparent lack of use of any form of assistance denotes an already strong labour force attachment and, consequently and apparently, more positive results in terms of employment outcomes. While existing data do not indicate significant differences in profiles between comparison group reachbacks relative to the other groups, it is likely that these data do not capture all differences between the groups that could account for the more positive employment outcomes observed for the reachback comparison group.
i) Pre- /Post-Employment Status
The impact of EBSMs on participants' employment status was measured in two ways. First, we compared participants' employment status in the week prior to the intervention (or reference date) to participants' employment status at the time of the survey. This measure provides a sense of the absolute change in employment status between these two points in times. Generally, this measure shows positive shifts in employment from the pre- to post-intervention periods for all EBSMs and larger shifts relative to the comparison group. Second, participants' employment status in the first week following the intervention and at the time of the survey was compared. This measure provides a sense of the extent to which employment status outcomes persist following an intervention. Descriptive analyses for this comparison are presented in Appendix E.
As shown in Exhibit 6.4, there was a larger overall increase in employment from the pre-to post-program periods among participants relative to the comparison group and participants' post-program employment rates were slightly higher than those of the comparison group for both claimants (56 versus 47 percent, respectively) and reachbacks (57 versus 51 percent, respectively). It is important to note however, that the comparison group had a stronger attachment to the labour force prior to the reference date, thus the smaller observed increases in employment rates for this group were at least partly attributable to a ceiling effect caused by higher rates of pre-reference date employment.
Unemployment rates were roughly equivalent for both participants and the comparison group overall but participants exhibited considerable declines in unemployment rates (55 to 29 percent and 64 to 32 percent) compared to the comparison group (30 to 34 percent and 38 to 32 percent).
| Participants by Claim Status | Comp. Group by Claim Status | |||||||
| EI Claimant | Reachback | EI Claimant | Reachback | |||||
| Pre | Post | Pre | Post | Pre | Post | Pre | Post | |
| Employed | 27% | 56% | 17% | 57% | 55% | 57% | 38% | 51% |
| Self-employed | 0 | 3% | 1% | 6% | 3% | 4% | 2% | 3% |
| Full-time year-round | 8% | 25% | 4% | 27% | 12% | 12% | 12% | 27% |
| Part-time year-round | 4% | 7% | 4% | 8% | 7% | 6% | 4% | 6% |
| Full-time seasonal | 8% | 11% | 4% | 7% | 17% | 11% | 11% | 5% |
| Part-time seasonal | 4% | 3% | 2% | 5% | 7% | 3% | 5% | 4% |
| Contract | 1% | 4% | 0 | 1% | 2% | 3% | 1% | 2% |
| Casual | 2% | 3% | 2% | 3% | 7% | 8% | 3% | 4% |
| Unemployed and looking | 55% | 29% | 64% | 32% | 30% | 34% | 38% | 32% |
| Student/in school | 4% | 6% | 4% | 3% | 2% | 6% | 14% | 8% |
| Out of labour force | 10% | 7% | 13% | 8% | 10% | 11% | 6% | 9% |
| DK/NR | 3% | 0 | 1% | 0 | 3% | 0 | 3% | 0 |
| No. of cases | 852 | 311 | 289 | 196 | ||||
| Note: Pre-intervention refers to the week prior to starting intervention/reference date and post-intervention refers to the time of the survey. | ||||||||
Positive shifts in employment were observed for all the EBSMs under study as well (Exhibit 6.5). All groups at least doubled their employment rate and the largest positive shift in overall employment occurred for self-employment (SE) participants (19 to 83 percent). Unemployment was also much lower at the time of the survey, dropping considerably for all EBSMs, but particularly for SE.
ii) Retention
A more direct measure of the contribution of EBSMs to positive employment outcomes is the extent to which wage subsidy participants were hired on by their host employers following the completion of the wage subsidy. Only participants who completed their wage subsidy were asked this question. As shown in Exhibit 6.6, the majority of wage subsidy participants who completed the full period of their subsidy were hired on by their host employer (55 percent), although considerably more TWS participants were hired on than JCP participants (65 versus 30 percent, respectively). This latter finding is not surprising if we consider the nature of these employment programs (i.e., TWS is geared toward job placement whereas JCP is designed simply for work experience). TWS participants were also more likely to have been hired into the same job they had during the wage subsidy (86 versus 73 percent of JCP participants). Little difference existed in the extent to which claimants and reachbacks were hired on after their program; however, claimants were slightly more likely to have been hired into the same job they had during the wage subsidy (88 versus 80 percent, respectively). Rates of retention also varied according to client demographics (not shown). The incidence of participants being retained in full-time year-round jobs rose with education level, whereas retention rates for full-time seasonal jobs rose with age and declined with education.
With respect to the types of jobs into which wage subsidy participants were hired by their host employers, TWS participants were four times as likely to have been hired into year-round full-time positions than JCP participants (43 versus 11 percent), whereas JCP participants were somewhat more likely to have been hired into full-time seasonal (47 versus 35 percent) or casual and contract positions (21 versus two percent). Again, these findings are not unexpected, given the nature of the programs. Differences in the types of post-program jobs participants held with their host employers were also observed in relation to clients' claimant status. While reachback clients were more likely to hold year-round full-time jobs (57 versus 30 percent), claimants were more likely to hold full-time seasonal (40 versus 26 percent) or part-time seasonal positions (12 versus five percent). The future summative evaluation will provide the opportunity to identify incremental impacts and results.
| Total | Wage Subsidy Type | Claimant Status | |||
| TWS | JCP | EI Claimant | Reachback | ||
| Percentage of participants hired by host employers* | 55 | 65 | 30 | 55 | 57 |
| Percentage of those hired by host employer who were hired into same job as wage-subsidy job | 84 | 86 | 73 | 88 | 80 |
| Percentage distribution of those hired back by host employer, by type of job hired into | |||||
| Full-time year-round | 39 | 43 | 11 | 30 | 57 |
| Part-time year-round | 6 | 6 | 4 | 6 | 3 |
| Full-time seasonal | 35 | 35 | 47 | 40 | 26 |
| Part-time seasonal | 9 | 8 | 10 | 12 | 5 |
| Casual/contract | 5 | 2 | 21 | 4 | 8 |
| DK/NR | 6 | 6 | 6 | 9 | 1 |
| TOTAL | 100 | 100 | 100 | 100 | 100 |
| No. of cases | 387 | 269 | 124 | 242 | 144 |
| * Only wage subsidy participants were asked this question; those who left the program before completion of wage subsidy were coded as "not hired." Source: Canada/PEI LMDA Participant Survey |
|||||
iii) Job Search Activity
Those who reported actively searching for work in the post-program period were asked to specify the job search methods they used while looking for work. As shown in Exhibit 6.7, the types of job search activity were somewhat similar across all participant groups. The most commonly cited job search methods were sending in resumes or applications (64 percent), checking job banks (64 percent), personal networks to friends and family (56 percent), and personal visits to employers (52 percent). A number of sub-group differences were also observed for participants in different programs:
Other notable sub-group differences emerged for participants and the comparison group. Overall, participants were more likely than comparison group respondents to send applications and resumes, whereas comparison group respondents were more likely than participants to check the newspaper. Furthermore, claimants in both the participant (55 versus 49 percent) and comparison groups (51 versus 45 percent) were more likely than reachbacks to make personal visits to employers.
iv) Interest in Entering the Labour Force and Willingness to Move
Participants' motivations to enter the labour force were measured through their rated interest in entering the labour force in the next 12 months and their willingness to travel for employment. Of those respondents who were jobless at the time of the survey, the vast majority of participants rated themselves as very interested (responded with a 6 or 7 on a 7-point scale) in entering the labour force in the next 12 months (92 percent) (Exhibit 6.8). Little variation existed in the extent to which participants in different EBSMs were interested in entering the labour force, although SE participants were least likely to be very interested (81 percent) and Training participants were most likely to be very interested (95 percent). Participants were more likely to be interested in entering the work force than comparison group respondents.
Similar to rated interest in entering the labour force, participants were somewhat more likely to indicate that they would be willing to travel for the right job (responded with a 5, 6 or 7 on a 7-point scale) than comparison groups respondents (70 versus 59 percent, respectively),35 and were willing to do so for a lower average hourly wage ($12.50 versus $14.10, respectively) (Exhibit 6.9). Together, these findings imply that participants are more motivated to enter the labour force than comparison group respondents.
Sub-group differences were observed according to client demographic characteristics (not shown). Willingness to travel was highest among males, younger respondents and single respondents. The minimum hourly wage to travel for a job was higher among men than women and rose as a function of the respondent's level of education.
Overall, the descriptive analysis of program impacts reveals some advantages for EBSM participants in terms of employment. Qualitative evidence from the focus groups suggests favourable employment outcomes were perceived to have occurred for participants, notably in terms of indirect employment measures, such as positive skill impacts and increased self-confidence. Survey evidence suggests the perceived impact of EBSMs on participants' post-intervention employment status is more modest, although EBSMs had a more positive impact on employment relative to services accessed by comparison group members. However, it is only through multivariate analysis, which takes into consideration the characteristics of participants which may pre-dispose them to successful outcomes, that definitive statements about program impacts can be made. This will be done in the next section.
For now, the following points summarize the findings for different outcome indicators, based on the descriptive analysis.
In the preceding section, participants in six EBSMs (EAS, SE, TWS, JCP, Training, and Feepayers) were compared to non-participants (the comparison group) in terms of a large number of post-intervention outcomes. As mentioned, however, the results from this simple descriptive analysis may yield a biased estimate of the impact of EBSM participation because of pre-existing socio-demographic and labour-market differences between non-participants and participants, favouring the latter. For example, the fact that one intervention out-performs another may have more to do more with the fact that its participants are highly educated and motivated relative to non-participants and participants in other interventions, than it has to do with the intervention itself. To ensure that differences in measured outcomes were not simply the effect of these pre-existing differences, therefore, multivariate modelling analyses were conducted. In these analyses, we control for differences between participants and non-participants, so that a "net" effect of EBSM participation can be measured.
The idea behind multivariate analysis is to use a statistical technique to explain a particular outcome (for example, post-program employment) in terms of a set of factors. The "set" of factors used in the analysis includes both the variables of interest (in this case, participation in EBSM interventions) and a number of other "control" variables (for example, background characteristics of participants) which might also explain the differences in the employment outcome. We also add what is called a "Heckman Correction" factor or "Inverse Mill's Ratio," which controls for any unobservable characteristics that distinguish participants from non-participants. (See Appendix C section 4 for a full explanation of the methodological approach used.)
In addition, because the models indicated that particular control variables influence the outcomes, we ran the models separately for segments defined by gender, age, claimant status, prior employment status, and rural/urban status. Models for the latter were run because rural-urban differences were of particular interest to the evaluation committee. The purpose of this exercise was to observe how paired segments differed in terms of how the interventions impact on outcomes, for example, how men and women differ in terms of being employed following an intervention.
The specific variables used in the analysis are as follows. The outcomes are a set of eleven employment, earnings and income-support measures, many of which were examined above in the descriptive analysis. These outcomes reflect the basic increased employment and reduced income-support dependence objectives of the Canada/PEI LMDA and include seasonal employment as an outcome, which is particularly relevant in the PEI context. The intervention variables in the models cover participation in the five EBSMs of EAS, SE, TWS, JCP, and Training and Feepayers (TFP) combined.36 The control variables capture the time since the intervention; socio-economic and labour-market characteristics of individuals such as sex, age, education, residence, past labour market history, EI claimant status, and past use of income support; and use of assistance services such as self-serve products, counselling and action plans — all of which are variables that could account for differences in outcomes between participants and non-participants over and above the impact of the interventions. See Appendix F for a complete list of the variables used in the analysis and their means and frequencies, indicating the differences between participants and non-participants.
In this section, a summary of the results of the multivariate modelling analysis is presented, including the segmented analysis. The summary is in three parts. We provide, first, the overall findings in each main outcome area, then a summary of the intervention-by-intervention results, and finally a summary of the characteristics of participants who tend to benefit from each intervention. Please see Appendices G to J under separate cover.
In the presentation and discussion of the results, a number of important points should be borne in mind. First, lower in the regression analysis we mention only variables having a "statistically significant" impact on the outcome variable. Significance is measured at the five percent level, which means that we are 95 percent confident of the result presented. Second, in this formative evaluation, the impacts of interventions have been measured only over a short period of time (one year or less). Truer measures of these outcomes are obtainable over the longer term, a need that to some extent will be addressed in the summative evaluation. Third, for this reason as well, we attempt to point out only patterns of findings rather than focus on specific outcomes for specific groups and characteristics of participants. Presenting all the detail would not only obscure the main findings, but would also not be useful at this early stage in the process when the focus is on design, delivery and implementation issues. We do, however, discuss specific differences between rural and urban participants because of the above mentioned interest in this issue among committee members. Note that we also mention specific participant characteristics that appear related to success so as to guide program officers in targeting of their assistance efforts.
The analysis was organized and the results are presented according to the four main intervention outcomes: employment, job search, earnings, and income support use. Where applicable, multivariate results are compared to the results of the descriptive analysis from the previous section and to the national results presented in the pan-Canadian EBSM formative evaluation report. In the exhibits presented in this sub-section, we show only the results for the EBSM interventions, indicating the direction of the impact each EBSM had on each outcome where the measured impact was significant in the models. In addition, based on the results presented in the appendices, we comment on the significant impacts of the control variables among all participants, as well as the differences in program impacts between paired segments (e.g., males compared to females) where patterns arise, while emphasizing rural/urban differences.
i) Employment Outcomes
In Exhibit 6.10, we present the impacts of the EBSMs for five (post-intervention) employment outcomes: the likelihood of being currently employed, the likelihood of being currently full-time employed, the likelihood of being currently seasonally employed, the likelihood of being employed for three consecutive months, and the percentage of weeks employed since the intervention. Non-significant impacts are indicated by "NS" in the exhibits.
The results show that SE led to positive employment outcomes for participants compared to non-participants for most employment outcomes. The likelihood of seasonal employment, a particularly important outcome from the perspective of the Island, was reduced only by participation in SE. Additionally, TWS and TFP had a positive effect on the likelihood of three consecutive months of employment. Other EBSMs had negative employment effects: JCP reduced the chances of current employment while EAS reduced the percentage of weeks working.
These results run counter to results from the descriptive analysis, which had indicated positive employment outcomes for participants compared to non-participants in a wide set of interventions. Controlling for differences between participants and non-participants, as was done here, has eliminated the apparent positive employment effects of the several interventions identified in the descriptive analysis. It would seem that participants in a number of interventions were predisposed to realize greater employment gains than non-participants, even before participating in the EBSM interventions.
| Intervention (vs. Non-Participant in intervention) | Likelihood of Being Currently Employed | Likelihood of Being Currently Full-Time Employed | Likelihood of being Currently Seasonally Employed** | Likelihood of Being Employed for Three Consecutive Months | Weeks Working as a Percentage of Weeks Since Intervention |
| Employment Assistance Services (EAS) | NS | NS | NS | NS | Negative |
| Self-Employment (SE) | NS | Positive | Negative | Positive | Positive |
| Targeted Wage Subsidies (TWS) | NS | NS | NS | Positive | NS |
| Training/Feepayers (TFP) | NS | NS | NS | Positive | NS |
| Job Creation Partnerships (JCP) | Negative | NS | NS | NS | NS |
| No. of cases | 1,261 | 1,261 | 1,261 | 1,265 | 1,253 |
| * Shown is the direction of the impact (positive or negative) of the respective EBSM on the outcome if the impact was found to be significant, resulting from regressions explaining the outcome. Not shown are the results for control variables entered into the model, including variables capturing the time since the intervention, socio-demographic characteristics, employment history, employment assistance services used, and the Heckman Correction. Specific variables entered are described in the text as are the regression results for these variables. See Appendix G for more details. ** A negative result is considered a favourable finding. NS Not significant at the 5 percent level or less. |
|||||
The complete results according to different groups (or segments) of individuals presented in Appendix G indicate few patterns in employment impacts of EBSM interventions with respect to differences between paired segments. For example, in comparing males and females, depending on the intervention and employment outcome, sometimes it would be male participants who were affected by a particular intervention and sometimes it would be female participants.
With respect to specific urban-rural differences in employment outcomes of the interventions, they are the following:
The analysis also identified certain client characteristics contributing to employment success beyond the role played by the interventions (see Appendix G). These "success factors" include having a post-secondary education, earning over $20,000 in the year before the intervention and being employed one year before the intervention. Negative factors are being less than 45 years of age and being in a minority group.
A factor contributing to sustained employment (i.e., as evidenced by an increase in full-time employment, three consecutive months of employment, or a greater proportion of weeks employed) is the length of time since the intervention. The descriptive analysis in the previous section showed that, even over the relatively short period of time between the intervention and the survey, the percentage of EBSM participants who were unemployed fell, while that of those in full-time, seasonal jobs increased. This result lends support to the notion that some interventions (such as EAS and TFP) may have longer post-intervention "gestation" periods that exceed the time horizon of this evaluation. This interpretation is further supported by focus-group responses which indicated that participants were realizing skill gains, which were seen as contributing to positive employment outcomes down the road. Similarly, at the national level, the pan-Canadian EBSM evaluation report did not report information on employment impacts per se, undoubtedly owing to the short-term nature of the evaluation period. Instead, only skills and job-experience impacts were observed, which were seen as contributing to employment outcomes in the future.
ii) Job Search
In Exhibit 6.11, we present the intervention results from modelling job-search intensity, defined as the number of weeks in job search as a percentage of the weeks since the intervention while jobless. The results indicate that participation in SE reduced job search, a finding that is true for most sub-groups of the population. That EAS, which includes job-search assistance, did not increase job-search intensity is somewhat surprising.
As indicated in the complete results presented in column 1 of Exhibit H.1 of Appendix H, factors associated with job-search intensity while jobless can be identified. A large number of personal characteristics increased the intensity of job search, including being male, having a pre-intervention interest in being trained, starting a business and entering the labour market, having received 105 or more weeks of EI before the intervention, and having used self-serve employment assistance products.
Looking at differences within pairs of segments, all groups experienced reduced job search intensity following participation in SE, except claimants, the not-employed and urban residents. Only urban residents' job search intensity was negatively affected by JCP and TWS.
iii) Earnings
Enhanced earnings is another goal of the Canada/PEI LMDA. Exhibit 6.12 presents the modelling results for the EBSM interventions for three earnings outcomes: current weekly earnings, absolute change in weekly earnings from before the intervention to the time of the survey, and percentage change in weekly earnings over that period. The earlier descriptive analysis identified no relationship between EBSM participation and earnings. Note, however, that after controlling for other factors affecting earnings, the multivariate analysis reveals that TWS, JCP and EAS lead to negative earnings outcomes. Additionally, current weekly earnings and percentage change in earnings were reduced by participation in SE in a number of segments. In general, it is rural not urban residents who experience negative earnings outcomes following their interventions, particularly EAS, TWS and JCP.
Negative earnings outcomes may be due to the fact that, as shown in the descriptive results, participants often change employment status. This suggests that participants may be changing careers and finding themselves near the "bottom of the career ladder" following their intervention. Once again, time will tell if positive earnings outcomes will eventually occur and possibly show up in the summative evaluation.
Finally, the complete modelling results presented in Appendix I indicate that having a post-secondary education, being younger, being male, and having three or more job separations contribute significantly to positive earnings outcomes. Also, earning $10,000 or more in the year prior to the intervention increases current weekly earnings and percentage change in earnings, while earning $30,000 or more reduces the absolute change in earnings. This finding implies again, that participants are moving to entry-level jobs following their intervention.
| Intervention (vs. Non-Participant in Intervention) | Current Weekly Earnings | Absolute Change in Weekly Earnings | Percentage Change in Weekly Earnings |
| Employment Assistance Services (EAS) | Negative | Negative | Negative |
| Self-Employment (SE) | NS | NS | NS |
| Targeted Wage Subsidies (TWS) | Negative | Negative | Negative |
| Training/Feepayers (TFP) | NS | NS | NS |
| Job Creation Partnerships (JCP) | Negative | Negative | Negative |
| No. of cases | 1,245 | 1,171 | 1,015 |
| * Shown is the direction of the impact (positive or negative) of the respective EBSM on the outcome if the impact was found to be significant, resulting from regressions explaining the outcome. Not shown are the results for control variables entered into the model, including variables capturing the time since the intervention, socio-demographic characteristics, employment history, employment assistance services used, and the Heckman Correction. Specific variables entered are described in the text as are the regression results for these variables. See Appendix I for more details. NS Not significant at the 5 percent level or less. |
|||
iv) Income Support Use
In addition to the goals of sustained employment and increased earnings for EBSM participants, an objective of the Canada/PEI LMDA is to reduce dependence on Employment Insurance (EI) and Social Assistance (SA). In Exhibit 6.13, we present results for outcomes in these areas. Note that a negative result is really the sought-after outcome here. We first discuss participants' post-intervention income-support use relative to non-participants. Then we discuss the effects of prior use of income support on post-intervention use, which are indicative of the extent to which income support use is being reduced.
We start with comparisons of EBSM participants' use of EI. The results in Exhibit 6.13, column 1, indicate that only SE participants had lower EI use compared to non-participants (the comparison group). This result is contrary to the descriptive evidence which showed that participants in all interventions had lower EI use. Once again, controlling for differences between participants and non-participants has eliminated the apparent advantage for participants. However, the segmented results in Exhibit J.1 in Appendix I do show that interventions reduced EI use for specific segments, as follows:
Finally, the analysis also found that TWS increased EI use for rural participants alone.
As for the effects of the control variables, the complete results presented in Appendix J also indicate that having a post-secondary education reduces post-intervention EI use, as does having pre-intervention interest in starting a business and earnings of $20,000 or more. On the other hand, speaking a language other than French or English, having more than two job separations, and having received more than 105 weeks of EI in the five-year pre-intervention period increased the percentage of post-intervention weeks on EI. The latter implies no change in EI use among heavy users of EI. Also, the longer the time since the intervention, the greater is the percentage of weeks on EI, implying discouragement over time.
Turning to post-intervention SA use, Exhibit 6.13, column 2 indicates that no intervention had an impact, suggesting that the EBSMs may have difficulty in reducing use of an income support mechanism (SA) not tied to the labour market (as EI is). Indeed, the segmented analysis (Exhibit J.2 in Appendix J) indicates that EAS acted to increase SA use for younger, employed and urban segments. With respect to effects of the control variables, having a post-secondary education leads to lower likelihood of using SA, as with EI use. Being married also reduces the likelihood of SA use, but being in a minority group reduces it, as does using SA before the intervention.
| Intervention (vs. Non-Participant in Intervention) | Percentage of Weeks on EI Following Intervention | Percentage Receiving SA Following Intervention |
| Employment Assistance Services (EAS) | NS | NS |
| Self-Employment (SE) | Negative | NS |
| Targeted Wage Subsidies (TWS) | NS | NS |
| Training/Feepayers (FTP) | NS | NS |
| Job Creation Partnerships (JCP) | NS | NS |
| No. of cases | 1,379 | 1,263 |
| * Shown is the direction of the impact of the respective EBSM on the outcome if the impact was found to be significant, resulting from regressions explaining the outcome, with a negative result implying a positive outcome. Note that negative impacts are here considered to be favourable findings. Not shown are the results for control variables also entered into the model, including variables capturing the time since the intervention, socio-demographic characteristics, employment history, employment assistance services used, and the Heckman Correction. Specific variables entered are described in the text as are the regression results for these variables. See Appendix J for more details. NS Not significant at the 5 percent level or less. |
||
Turning to reliance on public income support over time, we showed above that pre-intervention use of SA and heavy pre-intervention use of EI were strong determinants of post-intervention use of the respective income support mechanisms. But this finding was based on the full sample including non-participants. Therefore, we focused on just EBSM participants to measure the impact of pre-intervention use of income support on post-intervention use. The results (not shown) indicate that pre-intervention users of EI and of SA were more likely than non-users or low-users to continue to use the respective income-support mechanisms in the post-intervention period. Among EBSM participants, then, there has been no reduction in income support dependence, at least within the short-term context of this formative evaluation. Conversely, the pan-Canadian EBSM formative evaluation report found that, nationally, EBSM participation did appear to reduce EI dependence in the short term.
To provide more detail on outcomes on an intervention-by-intervention basis, the modelling results have been summarized in Exhibit 6.14. In each cell, there are two lines of information:
A summary of the findings, based on the exhibit, follows:
In this section, we summarize the impacts of each EBSM intervention, this time emphasizing the characteristics of participants who tend to profit from their participation in the intervention. The material here is based on the results of the segmented analysis summarized in Exhibit 6.14 and presented in detail in Appendices G to J. For each intervention, where there are differences in outcomes within pairs of segments, the specific characteristics of clients benefiting from their participation in the intervention appear in italics.
For participation in SE, there are no real differences in intervention impacts on employment outcomes within pairs of segments, except that the employed one year before tend to do better than the not employed, and rural participants do better than urban participants. Other characteristics are associated with specific employment outcomes; for example, younger participants in SE do better than older participants with respect to seasonal employment; and reachback participants do better than claimants with respect to current employment; but the opposite is true with respect to three consecutive months of employment. No population groups derive any gains in job-search intensity or earnings or reduction in SA use as a result of their participation in SE. SE acts to reduce the relative length of post-intervention EI spells for male, female, younger, claimant, reachback, employed, not employed, and rural participants.
For participation in TWS, the ideal characteristics vary according to outcome measure and intervention. In general, though, those employed one year before participating do better than those not employed. Other ideal characteristics include being an urban resident or a reachback for current employment; being older, a reachback and living in an urban setting for full-time employment; being younger and living in a rural setting for seasonal employment; being anything but a reachback, not employed and living in an urban setting for three consecutive months of employment; and being older and a reachback for percentage of weeks employed. No population group benefits from TWS participation with respect to job search, earnings and income-support outcomes.
For participation in JCP, being a claimant, not employed and a rural resident benefits a participant from the standpoint of reduced chances of seasonal employment. Being a claimant also benefits participants in terms of three consecutive months of employment. These are the only positive employment results for JCP. No population group experiences increased job-search intensity, positive earnings outcomes or reduced SA use from JCP participation. For reduced post-intervention EI spells, ideal characteristics for JCP participants are being male, older, an active claimant and an urban resident.
For participation in TFP, the extent to which participants experience positive employment outcomes again varies by outcome measure and client characteristic. For example, for full-time employment, males do better than females; the employed do better than the not employed; and urban participants do better than rural participants. For seasonal employment, younger participants do better than older participants; claimants, better than reachbacks; employed participants, better than not employed participants, and rural participants, better than urban ones. And for the chances of three consecutive months of employment, ideal characteristics include being female, younger, a claimant, and employed one year before. No positive outcomes of TFP participation were observed for job search intensity, earnings, and SA use. Finally, TFP leads to reduced post-intervention EI spells for male and urban participants.
For participation in EAS, no population group enjoys positive employment outcomes, except urban participants in terms of current employment. Nor did groups participating in EAS see post-intervention job-search intensity or earnings increase. Finally, male, female, younger, and urban EAS participants enjoy declines in their post-intervention EI spells, but no group participating in EAS saw reduced SA use.
Evidence on several measures of EBSM success to date was presented in this chapter. Though it is early in the implementation process (year one), it was deemed important to monitor and assess how the EBSMs are working to provide early warning signals of expectations not being met and to provide some insight into potential evaluation issues and design for the summative evaluation phase. There are three main findings of the analysis presented here.
First, with respect to accountability results targets set for the Canada/PEI Labour Market Development Agreement, this evaluation found that in 1998/99 targets were exceeded in two areas: EBSM participants returning to work and unpaid EI benefits resulting from a return to work before the end of the EI claim. While this is indicative of EBSM success to date, this could also possibly reflect targets that may be set too low or an economy that is improving. It was also found that the return-to-work indicator in HRDC administrative data may under-estimate actual returns to work, implying the need for improvement in the information systems and procedures used to track and monitor participants' progress.
Second, this evaluation has shown that it is important to control for differences in characteristics of EBSM participants and non-participants. These differences could contribute to success beyond the influence of the interventions themselves. Whereas results of the descriptive analysis presented early in this chapter indicated a range of positive employment and other outcomes for participants in most EBSMs compared to non-participants, the multivariate analysis, which controlled for differences between participants and non-participants, revealed that it was mainly Self-Employment (SE), and to a lesser extent Targeted Wage Subsidies (TWS) and Training Feepayer (TFP) participants, who benefited from EBSM participation to date in terms of employment gains and reduced EI use.
Third, it was determined that it may be too early to make definitive statements about the effectiveness of the EBSMs. Qualitative evidence gathered in this evaluation did indicate some job gains, but, more importantly, real gains in skills and confidence, which may be expected in the long term to materialize in jobs. This is supported by the survey evidence gathered for this evaluation indicating that the chances of job gains rose with the time since the intervention. The fact that the multivariate analysis pointed to job gains mainly for SE participants and to some extent TWS and TFP participants does not necessarily mean that the other EBSMs are ineffective. Instead, it means that it may take longer to see the payoff from these EBSMs, beyond the scope of this formative evaluation.
|
Evidence on the impacts of the Canada/Prince Edward Island Labour Market Development Agreement (LMDA) and Employment Benefits and Support Measures (EBSMs) on employers and communities is presented in this chapter, based on evidence gathered in focus groups with employers and key informant interviews with community stakeholders and government officials. It is important to find out not only how Employment Insurance clients are doing as a result of the interventions, as was done in the previous chapter, but also how other stakeholders are being affected.
The findings follow:
For participants, the EBSMs may have resulted in the acquisition of new skills, competencies and work experience and the development of transferable skills, such as independence, time management and responsibility, which ultimately may affect the sustainability of an employer's business. Nonetheless, it should also be noted that employers, consulted in focus groups, generally believed that the employment programs are not as effective as they could be. Employers were less inclined than government respondents to believe that the Canada/PEI LMDA had the potential to achieve employment results and to channel more people into the industry sectors with the most potential for growth. Generally speaking, while labour programs were considered valuable, employers thought that they did not always have the desired impact, perhaps due to an inadequate consideration of the needs of the individual client.
Focus groups revealed that employers generally believed that those programs that clearly have a link between the client and the desired end-result have been the most successful. The Targeted Wage Subsidies (TWS) is an example of such a program, as it has contributed the most to positive client outcomes. TWS has also benefited those employers who hesitate to hire by allowing them to hire unskilled workers and train them to be productive employees while being relieved of the pressures of cash flow. Wage subsidy programs were considered helpful, if not critical, for some employers in helping their enterprises stay competitive. Several respondents pointed to the positive impact labour market programs had on improving the sustainability of their business, particularly where wage subsidies and training were involved.
Employers pointed out two examples where TWS does not meet the needs of particular industries, however. First, while TWS was seen to be successful in meeting the needs of the seasonal industries, it is unclear to employers whether or not this success will lead to a positive or negative impact on the PEI labour market in the long run. For example, while the program allows employers to extend their season and increase their visibility by undertaking more activities, there is a lack of financial incentive for employees to engage in off-season work activity when Employment Insurance (EI) is available. Second, employers in the information technology sector felt that, although advantageous, TWS needs to be extended in time to benefit their industry due to the extensive amount of training required.
Employers consulted in interviews were doubtful that the LMDA would be able to strengthen attitudes in PEI toward education and training, which government respondents hoped the LMDA could achieve. Employment programs were still perceived by employers as stop-gap measures to fill positions that would not be filled otherwise and that are generally lacking in substance. In addition, some employers were skeptical that TWS would contribute to creating long-term employment because they felt that participants often engage in training on-the-job training simply to qualify for EI.
Employers in the focus groups perceived that the EBSMs are the least effective where there has been an inadequate match of the program or service with the real needs of the clients and their desired outcomes. From an employer perspective, Human Resources Development Canada staff need to learn more about the labour market supply and demand in PEI so that programs can be geared towards potential fields of employment. More emphasis needs to be placed on matching clients with suitable occupations. Employers noted that there was a general lack of consultation by authorities to determine current employer and community needs. It was felt that employers should be able to discuss with an HRCC staff member the specific staffing needs of the community and that the HRCC in turn would attempt to fill that job vacancy.
The biggest problems identified by employers relate to clients who are not served by the LMDA: unemployed people not eligible for EI and currently employed people needing skills upgrading. While employer key informants recognized that employment programs were designed to accommodate the realities of seasonal employment and peaks of business, many respondents expressed frustration with the program requirements that provide funding for the hiring of EI clients only. It is sometimes very difficult for employers to find good workers from the pool of candidates who are EI eligible at a particular time, thereby losing the potential positive impact of the employment program in question on their business.
Interview respondents also commented that it was difficult, particularly for new and small businesses, to benefit from TWS due to program requirements. One burdensome requirement is that employers must hire a client before applying for assistance. Prior to the LMDA this was not required. Also, training days under the programs were thought to be too long: shorter training days were suggested to reduce the amount of time employers had to spend training participants and to allow them to cover a wider range of activities. Employers must invest a substantial proportion of their own time in the actual training and employees in training are less productive.
From the point of view of employers consulted in focus groups, the marketing of the EBSMs was seen as inadequate. They did not feel that Human Resources Development Canada (HRDC) had done a good job of informing employers about the programs to which they had access and about which ones would be most beneficial. Most respondents were informed of the programs through word of mouth. Employers felt that public presentations where they could learn about various programs and how to apply would be beneficial. Bulletins boards could also be utilized as a form of advertising.
Despite numerous recommendations for change, however, employers reported that the general impact of the labour market programs was positive. Problem areas were, for the most part, viewed as areas for potential growth. Many employers noted that subsidized programs are a way of life in PEI and indicated they would continue to participate in labour market development programs in the future.
Community stakeholders consulted in this evaluation felt that it was still early at this point to assess the impacts of the LMDA on communities, although qualitative data gathered through interviews and focus groups provide some evidence of preliminary impacts.
Senior officials with the federal government and the Province felt that the LMDA and EBSMs have been successful in addressing the short-term needs of communities and many Islanders. Wage subsidies, for example have been successful in extending seasonal employment. To thoroughly address longer-term needs, however, some key informants felt that the programs may need to be re-designed to provide assistance for other groups besides EI claimants. As mentioned previously, a key perceived limitation of the LMDA in affecting the community is the fact that unemployed Islanders and the currently employed in need of training and skills upgrading are not eligible for EI. To maximize their impact on the community, the EBSMs need to be tailored to distinct target groups among residents of the province. For example, there is a need for programs to keep educated youth in PEI and integrate them into the local market. Further, representatives of disability organizations stated that the EI eligibility criteria have put certain population groups, such as persons with disabilities, at a distinct disadvantage because they are often incapable of achieving the number of weeks of required work to qualify for programs that would help them enter the labour market.
Despite these perceived limitations, the LMDA was perceived to have had beneficial impacts on individuals and communities. According to senior federal and provincial officials, these include the extension of the shoulder seasons, the community learning centres, the Technology Mentoring Program, life-long learning measures and a lower Social Assistance case load (due to clients being maintained in the labour force for longer periods). One federal official perceived that participants in training programs seem to value the training and work experience they are gaining through EBSMs and he speculated that, contrary to what some employers felt, the LMDA may be strengthening values in PEI regarding education and training.
Like government representatives, community stakeholders consulted in interviews felt that the LMDA programs are meeting the needs of those people eligible for EBSMs and have had a positive impact on PEI. However, they felt the LMDA would have a greater positive impact if more funds were spent on human resources development than on wage subsidies and job creation initiatives. One respondent thought that the LMDA should do more to enhance training — a key community-based solution.
Community stakeholders also felt that, as a result of the LMDA, partnerships with community organizations had improved. Representatives of disability organizations, in particular, thought that their relationships with federal and provincial departments had improved. Community education representatives thought that PEI had made much progress in developing a knowledge-based economy and that the LMDA may perhaps be credited with this. Stakeholders felt that the LMDA had formalized agreements with community learning centres, establishing long-term vision and planning. These centres address a key need for adult literacy training in PEI. Literacy representatives felt that the LMDA has helped literacy development on the Island, and hoped that the Province would continue to take literacy training very seriously as higher literacy rates will have a broader, long-term positive influence on communities, the economy and the lives of Islanders.
Community stakeholders also discussed the delivery of specific EBSMs. One community representative felt that SE is an excellent program that meets the needs of clients, but that eligibility was a major problem (because most new businesses are established by employed people, not by those on EI). Job Creation Partnerships (JCP) was seen as an effective, well-delivered program, as it has allowed for special projects such as infrastructure for tourism and has provided both skills and a feeling of pride to workers. Also, though TWS was valued, employers would like to play a more important role in client selection, because the match of workers and employers in the TWS program is not always appropriate. Community stakeholders also expressed some frustration with the delays in approving Local Labour Market Partnerships (LLMP) project proposals, which must be submitted to Charlottetown. Representatives of private training institutions felt that Skills Development meets the needs of clients because it allows them to choose their own training program and start their courses anytime, and of the institutions because it has made the training market more competitive. This view was not shared by stakeholders in public education, however, who indicated that they have had to cut back on training programs due to the elimination of the Purchase of Training program as part of the Canada/PEI LMDA.
Evidence on the impacts of the Canada/PEI LMDA and EBSMs indicated that employers believed that their participation in the wage-subsidy EBSMs was beneficial to their businesses. However, employers often could not retain participants after funding had ended owing to poor economic conditions and the seasonal nature of their business. Moreover, while employers' participation in the EBSMs often enabled them to extend their season, they were sceptical that this would lead to long-run changes in this respect on the Island.
Community stakeholders consulted, including representatives of training institutions, felt that the LMDA had contributed positively to the community. Specific outcomes observed included extension of the "shoulder" season, improved literacy and the means to enhance literacy, and the potential to strengthen attitudes among Islanders toward education and training. However, it was felt that it was still too soon to tell whether or not certain hoped-for outcomes such as changes in attitudes and reduced dependence on seasonal industry had taken effect.
|
The evaluation evidence indicates that, for the most part, the Canada/Prince Edward Island Labour Market Development Agreement (LMDA) and Employment Benefits and Support Measures (EBSMs) are relevant to the needs of the targeted industry sectors, employers, communities and Islanders. The LMDA focuses on resource and seasonal industries (e.g., agriculture, fishing and tourism), the staples of the Island, as well as on targeted high-value industries such as information technology and aerospace, which have the potential to create long-term jobs for the province. Still, there were some opinions expressed that the LMDA needs to be better focused on the unique needs of PEI (e.g., programs should be better adapted to the seasonal economy, high level of unemployment, and low annual earnings of Islanders). Moreover, there was a pervasive concern that the relevance of the EBSMs is limited by the restrictive program eligibility criteria imposed by the Employment Insurance (EI) Act. This concern appears to stem from a widely held misperception that the mandate of the LMDA encompasses the entire PEI labour force, when in fact the Agreement is designed to serve EI clients only. It was believed that the focus of EBSM assistance on EI eligible clients meant that some individuals in need of assistance were not being served. In particular, many respondents felt that small business development, skills upgrading for currently employed or underemployed Islanders (in part to supply the skills required in new targeted industries), and the needs of youth and persons with a weak attachment to the labour market are not being adequately addressed by the LMDA or any other initiatives.
There was some conjecture among key informants on the issue of focusing on seasonal industry. On the one hand, taking measures to extend the duration of the work season in seasonal industries was seen as acceptable given that these industries will be a reality on the Island for the foreseeable future. Still, respondents holding this view also noted that seasonal jobs should be seen only as stepping stones to longer term employment, that seasonal workers should be equipped with multiple skills to make them portable, and that there is a need to promote industries complementary to seasonal industries. On the other hand, the majority of key informants suggested that the emphasis for the LMDA should be on developing new year-round industries so that the Island can end its dependence on seasonal industry.
There were also conflicting views with respect to the degree of overlap between the EBSMs and other employment and training assistance measures. Senior managers and stakeholders felt that the EBSMs are well harmonized with other federal and provincial programs mainly because of the wide range of representation on LMDA committees and working groups. On the other hand, managers below the senior management level and provincial government representatives perceived that there is a fair amount of duplication and a lack of coordination among programs.
Considering the complexity of the task, the implementation of the Canada/PEI LMDA has gone reasonably well to date, though further development and improvements will be needed. On the positive side, the high degree of cooperation and collaboration among all LMDA players, the delivery of successful initiatives such as adult basic education/literacy and aerospace, and Human Resource Centre of Canada (HRCC) staff's understanding of their role in EBSM delivery were all noted by interview and focus group respondents as successful aspects of LMDA implementation. On the other hand, a lack of reliable, timely labour market information and client tracking information for LMDA planning and management, excessive administrative requirements and the associated delays in project approvals, and somewhat of a HRCC staff shortage were regarded as key weaknesses that will require attention. Also, both federal and provincial managers identified the lack of technical resources to support needed LMDA information systems as a problem. Although productive partnerships with community organizations have been developed and despite the fact that local-level consultations with stakeholder organizations had been conducted during the development of the 1998-1999 LMDA business plan, stakeholders as well as front-line HRCC staff felt that consultations and partnerships with "grass roots" community organizations still need to be improved. Moreover, there is a need to increase awareness of the LMDA and EBSMs at the community level.
The EBSMs were generally regarded as flexible and responsive to local and client needs. Interview and focus group respondents had some reservations, however. In particular, respondents identified a need to further refine and adapt the programs to the unique needs of PEI (e.g., many seasonal industries, relatively high unemployment, and low annual earnings), to harmonize LMDA and Social Assistance programming, and to better serve and inform clients about available programs. Also, as already noted, the EI eligibility criteria of the Employment Insurance Act and delays in project approvals were thought to limit the responsiveness of EBSMs. The perceived strengths of Human Resources Development Canada's approach to delivering the EBSMs include a cooperative and positive effort on the part of highly experienced HRCC staff, flexible and decentralized program delivery, and a reasonable amount of paper work for clients/funding recipients. On the other hand, some clients and stakeholders perceived that increased flexibility at some HRCCs and not others meant that service delivery could be inconsistent from one HRCC to another.
In the survey, clients indicated being most satisfied with the quality of education or training they received and with the knowledge of employment counsellors, but comparatively less satisfied with the quality of referral services and with the information available to help them choose suitable programs. In addition, LMDA programs and services are being successfully delivered in both official languages as intended.
The federal-provincial partnership has been working reasonably well, though some "growing pains" are still being experienced. Both federal and provincial managers identified several strengths of the partnership and the co-management approach. In particular, these respondents perceived a high degree of cooperation between partners and noted that co-management facilitates collaborative decision-making, mutual understanding and coordination of federal and provincial initiatives. They acknowledged that the partnership does take a lot of effort and compromise, however. For example, front-line HRCC staff observed that the partnership has been difficult and frustrating at times, and that the degree of cooperation between the two levels of government could have been better in the pilot of Skills Development. Moreover, as already noted, most respondents observed that co-management adds another layer of bureaucracy and complexity to the LMDA, resulting in delays in project approvals.
Many Human Resources Development Canada (HRDC) staff have been reluctant to embrace the new co-management approach due to their concern that program delivery may eventually be fully devolved to the Province, which causes them anxiety over their job security. Although most senior managers felt that the LMDA is generally compatible with broader government objectives, provincial key informants asserted that the EBSMs need to be further adapted to better match the objectives of the provincial government.
The results of the accountability target attainment computations indicated that the EBSMs exceeded both the 1998/99 return-to-work and unpaid EI benefits targets. Despite exceeding the return to work target, questions were raised about whether or not all returns to work were being captured by the information systems.
Qualitative evidence of impacts on participants indicated favourable employment outcomes, if not immediately then expected in the long-term because of positive skill and psychological (e.g., self-confidence, self-esteem) impacts. Still, less than one half of survey respondents said their EBSM intervention was very important in attaining their current job although this reflected a more positive perceived impact than the comparison group. Employers consulted in focus groups agreed that skill and psychological impacts are occurring, although a number said they were often unable to retain their wage-subsidy workers because of financial difficulties.
Quantitative survey evidence indicated that at this stage of the evaluation, only Self-Employment (SE) led to consistently positive employment and income-support outcomes. Targeted Wage Subsidies (TWS) and Training/Feepayers (TFP) increased the likelihood of being employed for 12 consecutive weeks. Employment Assistance Services (EAS), TWS and Job Creation Partnerships (JCP) produced negative earnings outcomes. More conclusive evidence of EBSM impact on employment and income support will be available at the summative stage of evaluation.
Results were mixed for perceived employer impacts. On the one hand, employers in the focus groups said that TWS relieved some employers of cash-flow pressures in hiring and that training unskilled workers enabled some businesses to be sustainable and even competitive. On the other hand, some employers were unable to retain workers because of cash flow problems and others said the EBSMs would be unable to channel workers into sectors and occupations where they were in demand. In addition, some employers in certain areas (e.g., seasonal, information technology, new/small businesses) did not think the LMDA as implemented could meet their specific needs because of perceived shortcomings in the rules. Some employers believed the EBSMs would be insufficient to effect a change in attitudes with respect to valuing training and dependence on seasonal industries and income support.
For impacts on the community, some key informants and focus group participants felt it may be too early to address this issue. Among those who were able to, there were mixed results. Some respondents felt the LMDA may have helped in addressing short-term needs of communities and Islanders, but there was concern that the exclusion of non-EI eligible persons may limit the LMDA's ability to address long-term needs. Others mentioned that, to truly benefit the province, there needs to be greater emphasis on interventions providing labour market development than on those providing job creation and wage subsidies. On the other hand, some respondents spoke of the benefits of the LMDA for the community, including the extension of the "shoulder" season, community learning centres, lifelong learning and adult literacy training measures, the technology mentoring program, lower Social Assistance (SA) caseload, and partnerships between communities and the government.
A review of the qualitative and quantitative evidence indicates that there were few clear rural-urban differences in terms of perspectives on the LMDA and in its impacts. With respect to the latter, multivariate analysis indicated that the EBSMs had positive impacts on rural residents in terms of the likelihood of seasonal employment, the percentage of weeks employed and three consecutive months of employment, but were beneficial for urban residents in terms of full-time employment. Also, they produced negative effects for rural residents in terms of earnings. We also attempted to contrast the views of focus group participants located in urban, rural and isolated rural centres, but were again unable to observe any clear pattern.
Differences observed between rural and urban clients include the following:
In sum, no strong pattern of differences in impacts was observed between urban and rural participants, although the available evidence suggests that rural clients are more likely to be affected by the EBSMs than urban clients. Multivariate analyses of program impacts show negative effects for rural participants with respect to earnings outcomes, which may be explained by the fact that opportunities for entering better-paying jobs are fewer in rural areas. However, in light of the higher unemployment in rural areas, as indicated by the higher rates of Employment Insurance dependency37 in these regions, it is perhaps surprising that employment outcomes were not more consistently positive for urban residents.
Also, in the few areas where differences in satisfaction between urban and rural residents were observed, rural residents report higher satisfaction. This latter finding may imply that urban residents have higher expectations with respect to EBSM services.
The key lessons learned in the formative evaluation are presented in this section.
Lesson 1
There are serious information system problems that have been perceived to exist which hamper the ability to track both clients' progress and results in a timely fashion and to provide accurate and usable labour market information. A large number of problem areas, also noted in the pan-Canadian study, that contributed to the information system difficulties underline the severity of the problem. They are as follows:
The information difficulties identified had implications for the current evaluation. The administrative data problems, in particular the lack of accurate return-to-work indicators, compromises the ability to accurately assess the results targets. Further, the absence or inaccuracy of identifiers in the administrative dataset compromised our ability to accurately profile participants, to include all members of the population in the sample frame and therefore develop a sample that was truly representative, and to contact all members of the sample for the survey. Attempts to acquire the necessary information required a review of several different databases from several different sources. This process was far more complex than it should have been and still failed to yield complete data. If the data issues go unresolved, they will pose problems for the summative evaluation as well.
Lesson 2
In assessing the returns-to-work and unpaid EI benefits targets set for the Canada/PEI Labour Market Development Agreement (LMDA) EBSMs, EKOS' computation found that both targets were exceeded in 1998/99, particularly the former. The fact that the return-to-work target was exceeded by almost 50 percent, despite the fact that the target had been increased from the year before, would suggest increased EBSM effectiveness due to growing familiarity with processes. However, it could also reflect targets that were set too low and/or an economy that was generating increased job opportunities. Moreover, despite the fact that the EBSM generated returns-to-work exceeding targets, EKOS' analysis indicated a discrepancy between the survey and administrative datasets in terms of capturing returns-to-work. Moreover, EKOS raised concerns about how unpaid EI benefits were computed. This was a problem identified nationally in the pan-Canadian EBSM formative evaluation, as well as in evaluations of LMDAs in other parts of the country.
Lesson 3
Analysis of EBSM impacts indicated that SE was particularly effective in integrating participants into the labour market compared to non-participants. This was found even after controlling for differences between the characteristics of participants and non-participants (e.g., differences in education level) that could have accounted for the employment advantage in favour of participants over and above the intervention itself. The fact that other interventions were found, for the most part, not to play a significant role in participants' entering employment, despite positive effects initially found in the purely descriptive analysis, suggests that in measuring impacts the characteristics of participants must be taken into consideration. Also, because not all interventions are expected to have immediate employment outcomes like SE, the analysis also suggests that it may be too early to measure the true employment-generating properties of all EBSMs, i.e., within the timeframe of this formative evaluation. With respect to other outcomes, SE was found to lower EI use compared to non-participants, but no evidence was found for the reduction of dependence on income support, to date. No increase in earnings was detected for this or any other EBSM intervention.
Lesson 4
Although most clients appear to be quite satisfied with the programs and services that they have received, some are nonetheless unhappy about the way they are treated while accessing services. They are dissatisfied with the layout of the HRCCs and with not being adequately informed about programs and services.
Lesson 5
The EBSMs are largely responsive to local needs and in part this can be attributed to HRCC managers' spending authority of up to $75,000. There is, however, a perceived lack of consistency among the different HRCCs in the way programs are delivered, the amount of financial assistance granted to clients, etc. There are, of course, trade-offs involved here the more local flexibility and responsiveness, the less consistency in program delivery from one local area to another. While local flexibility is a desired attribute, clients perceive that there are too many inconsistencies across HRCCs and they feel that this is unfair treatment.
Lesson 6
For the most part, the LMDA is considered relevant to the needs of the targeted industry sectors, employers, communities and Islanders. Respondents also feel that the LMDA is relevant in the sense that the EBSMs can be used to extend the shoulder season of seasonal industries and to help diversify the Island economy into non-seasonal industries. It was also noted, however, that this diversification may require a change in the way some Islanders view their working lives and their economy in terms of dependence on seasonal work. Although some senior federal managers feel that the EBSMs are consistent with government priorities, objectives and HRDC's mandate, provincial officials suggest that the former HRDC programs need to be further adapted to meet the specific needs of the Island.
Lesson 7
There are mixed views and some confusion regarding the extent to which the EBSMs are harmonized and complementary with other provincial and federal initiatives. While senior managers tend not to see any major problems with overlap, middle management and front-line personnel perceive that there is some duplication and a lack of coordination among federal and provincial programs (e.g., wage subsidies, self-employment and youth programs). Moreover, in the 1998 national EBSM formative evaluation, the potential for duplication in Atlantic Canada was noted.
Lesson 8
Many respondents believe that the imposition of EI eligibility as a criterion for LMDA program participation creates a serious gap in programming because many Islanders (e.g., currently employed, self-employed, youth, chronically unemployed, persons with disabilities) are unable to access EBSMs as a result. Further, many respondents point to a perceived lack of services for currently employed individuals (i.e., for skills upgrading, particularly in priority sectors) as an example of a major gap in labour market programming. This evaluation focussed only on EBSMs and their associated client base and did not look at the specific needs of these other groups. Meeting their needs would require a comprehensive labour market development strategy for the whole province which goes far beyond the issues addressed in this evaluation.
Lesson 9
Although productive partnerships have been developed with community organizations, stakeholders and front-line HRCC staff feel that consultations and partnerships with "grass roots" community organizations need to be improved. Despite the fact that they were consulted for the 1998/1999 LMDA business plan, community groups would like more input into LMDA planning and implementation. Findings from the 1998 national EBSM evaluation also indicate a need for improved community consultations and partnerships.
Lesson 10
At senior management levels, co-management has had a number of positive impacts, such as better communication, mutual understanding and learning, collaborative decision-making and better coordination of joint initiatives. However, there are perceptions that management does not always communicate well with front-line staff; thus there has been some confusion about programs. Also, HRDC staff perceive that the lack of information sharing between federal and provincial partners on EI clients and Social Assistance clients limits the opportunities to achieve productive, long-term solutions for both types of clients. Further, there is much anxiety among staff about what will happen in the long term in labour market development (i.e., whether programs will be fully devolved to the Province and whether some federal staff will lose their jobs as a result).
Lesson 11
The co-management model for the LMDA has been quite helpful to both levels of government. When the Agreement was first signed in 1997, the Province, with the exception of the Health and Social Services Department, had little experience in delivering employment programs and services department, while HRDC benefited from the economic development expertise of the Province. The added layer of bureaucracy in the administration of the programming has, however, caused delays in reviewing and approving project applications.
Lesson 12
Many of the findings from this evaluation echo the pan-Canadian 1998 Formative Evaluation of the Employment Benefits and Support Measures. They follow:
The Management Committee of the Canada-PEI Labour Market Development Agreement (LMDA) has reviewed the Formative Evaluation of the Canada-Prince Edward Island Labour Market Development Agreement and is very pleased to have received this feedback. The Committee will use the evaluation results to improve its decision-making processes and to enhance labour market programs and services available to Islanders.
The Formative Evaluation focuses on the partnership model initiated to design and manage the LMDA and on the Employment Benefits and Support Measures (EBSMs) available to EI eligible clients. A greater portion of the data collected concentrated on the five employment benefits, which are those programs designed for short-term outcomes. Qualitative information was also gathered on the support measures, such as the Local Labour Market Partnerships (LLMP). The evaluation provides a preliminary review of LMDA implementation based on information collected from December 1998 until October 1999. Further patterns and program outcomes will be captured in the summative evaluation to follow.
Generally, the Formative Evaluation concludes that LMDA programs are relevant to the labour needs of Islanders, communities, employers and growth sectors of industry. EBSMs can be used to extend the shoulder season of seasonal industries and to help diversify the Island economy into non-seasonal industries. As well, strengths of the partnership approach are confirmed, but some areas for improvement are also highlighted.
One identified strength of the co-managed model is the collective knowledge and perspectives the labour market partners bring to planning and decision making. The combination of HRDC's extensive experience in labour market programming and service delivery with the Province's expertise in economic development has proven effective for labour market development on PEI.
For example, the focus on industry sectors with potential for growth has successfully created employment for many Islanders. By considering labour market development within the PEI economic context, labour market funding can be invested strategically.
A principle established in the LMDA to guide the labour market partners in the implementation of their partnership arrangement is to harmonize employment initiatives "...to ensure that there is no unnecessary overlap and duplication. The evaluation uncovered that having representatives from both governments on the Management Committee has served to decrease duplication of services. However, management continues to place programs strategically to minimize overlap and to address gaps in service delivery.
While cooperation between the labour market partners is significant, some difficulties with the management structure were identified in the evaluation. Concern was expressed that the co-managed model causes delays in some approval processes. As well, the on-going operation of this strengthened model was found to be resource intensive.
The Management Committee acknowledges that working together has been a learning experience, but that improvements have been made over time. Initial delays have been addressed and the Committee continues to strive towards timely and efficient processes.
Evaluation of the EBSMs demonstrates that targets are being consistently met. Multivariate analysis of EBSM participation results showed a difference in outcomes between rural and urban participants — rural clients achieved longer periods of employment while urban clients were more likely to move into full-time employment. In addition, positive qualitative employment outcomes, including self-confidence and esteem building, were attributed to EBSM participation.
Employers viewed the EBSMs as making their business more sustainable and consistent. However, they also felt that Targeted Wage Subsidies (TWS) could be more flexibly designed to enable the creation of sustainable employment. The seasonal nature of the Island economy must be considered in developing long-term labour market strategies. Employers, like the Management Committee, saw implementing EBSMs designed for the PEI context as crucial for long-term employment creation.
Clients, employers and service deliverers indicated customer service issues concerning quality and consistency of information and referral services. Again, this group "... indicated that the EBSMs need to be further refined and adapted to meet the unique economic needs of Islanders."
Efforts are being made to develop a balance of flexibility and consistency in these areas. Services are being enhanced through increased labour market and client tracking information, public awareness and customer service. PEI specific issues are considered when making funding decisions. The Management Committee continues to endeavour towards PEI-economy-specific programming.
The LMDA Management Committee accepts concerns expressed by service deliverers, employers and Islanders in general for the labour market needs of those ineligible to receive EBSMs. Individuals who are not EI eligible, in particular "...employed workers, youth and those with weak labour market attachments," do not have access to valuable programs which could improve their labour market outcomes. The situations of these Islanders need to be addressed, though not necessarily through this Agreement, to allow full labour force participation.
Results of this evaluation are consistent with the findings of the Pan Canadian Formative Evaluation of Employment Benefits and Support Measures completed in 1998, but they focus with more detail on Island-specific issues. The LMDA formative evaluation will provide a solid foundation for the upcoming summative review as the partners strive towards positive labour market impacts and outcomes through this co-managed model.
The Formative Evaluation of the Canada-PEI Labour Market Development Agreement has provided federal and provincial partners with valuable recommendations for improving the effectiveness of labour market programs and services for EI eligible Islanders. The Management Committee is committed to providing relevant and high quality labour market programs and services designed to fit the specific needs of Islanders.
|
|
The Canada/Prince Edward Island Labour Market Development Agreement (LMDA), like all provincial and territorial LMDAs, specifies accountability results targets for the Employment Benefits and Support Measures (EBSMs) in three areas: Employment Insurance (EI) clients served; participants returning to work, particularly active EI claimants; and unpaid EI benefits as a result of active EI claimants returning to work before their EI claim has ended. Under the terms of the Canada/PEI LMDA, these targets are to be monitored each year and possibly adjusted to reflect changing circumstances or imprecision in the setting of the targets. In this appendix, we use both administrative and evaluation survey data to assess the attainment of the latter two targets (returns to work and unpaid EI benefits) because LMDA management focus is no longer on the number of active EI claimants participating in the EBSMs.
In this appendix, we present the results of two exercises. In Exhibit B.1, there are the results of measuring attainment of the annual targets. Columns 1 and 2 present the targets for the 1997/98 and 1998/99 fiscal years for all active benefits and measures under the Canada/PEI LMDA.39 In Exhibit B.2, we show the results of measuring success using the same results measures, concentrating on the EBSMs considered in the survey (Targeted Wage Subsidies [TWS], Self-Employment [SE], Job Creation Partnerships [JCP], Training/Feepayer, and Employment Assistance Services [EAS]). We use survey and administrative data to consider the first result measure (returns to work)40 and administrative data to look at the other result indicators (unpaid EI benefits). Note that the figures in this exhibit cover a period from April 26, 1997 to October 31, 1998, which was the time frame for the five EBSMs covered by the survey. Also note that, because of the differing datasets and methodologies used, it is expected that differences in computed results will emerge.
It is also important to note that the figures in Exhibit B.1 apply to all active benefits and measures under the Canada/PEI LMDA including services such as individual counselling interviews and group services41 which EI clients can use independently of the benefits and measures being evaluated. On the other hand, the figures in the Exhibit B.2 apply only to the five benefits and measures under study in the survey and over the period in question (April 26, 1997 to October 31, 1998). For this reason, the figures in these two exhibits should not be compared.
In this subsection, we present results of our computations to assess attainment of the accountability results targets.
Panel 1 of Exhibit B.1 shows that, with respect to the targets for participants returning to work following intervention, column 3 indicates that this target was exceeded by over 20 percent (2,170/1,800) in 1997/98. Note that column 3 shows the result as published in the 1998 EI Monitoring and Assessment Report and we confirmed this figure (2,170) using the Human Resources Investment Branch (HRIB) Results File for 1997/1998 administrative data. Note as well, for 1998/99 the target (column 2) was increased over the previous year (column 1), likely partly reflecting the fact that it had been exceeded in that year. Finally, in column 4, using the HRIB Results File for 1998/1999, we show that the 1998/1999 target was exceeded as well, this time by 45 percent of the target (2,000) set for that fiscal year, indicating improved EBSM effectiveness and possibly further need to increase targets. This performance could also reflect an improvement in economic conditions, leading to more job opportunities.
In Panel 2 of Exhibit B.1 are presented the results of our computation of unpaid EI benefits. In columns 1 and 2, we show targets of $7.8 million for the 1997/98 fiscal year and $4 million for the 1998/99 fiscal year. In columns 3 and 4, we show the amounts of unpaid EI benefits, based on the administrative data. Note that these are amounts realized from active EI claimants because reachbacks cannot generate unpaid EI benefits. We produce these figures using HRIB Results Files for the respective fiscal years, adding up the unpaid benefits as entered on the file for those with a return to work (RTW) flag. For 1997/98, column 3 indicates a sum (for the 2,170 RTW cases) of $4,142,331 in unpaid benefits (which is somewhat over the figure of $4,120,672 as published in the 1998 EI Monitoring and Assessment Report.) This figure was well short of the target of $7.8 million for that year (column 1) and may be one reason why the target was reduced in 1998/99 (column 2). For 1998/99, column 4 of Exhibit B.1 indicates that (for the 2,963 returns to work in that year) there was a total of $4,537,329 in unpaid benefits, which this time exceeds the target for that year (column 2), by over 12 percent. This suggests targets have now been set at a level that is more attainable. The fact that returns to work increased from 1997/98 to 1998/99 to a greater extent than did unpaid EI benefits would suggest that many of the increased numbers of returns to work were reachbacks (and therefore would generate no unpaid EI benefits) and/or active EI claimants returning to work after exhausting their EI claim (therefore again generating no unpaid EI benefits).
| Targets* | Results | |||
| 1997/98 (1) | 1998/99 (2) | 1997/98** (3) | 1998/99 (4) | |
| Panel 1 Number of EI clients participating in EBSMs who returned to work | 1,800 | 2,000 | 2,170 | 2,963 |
| Panel 2 Unpaid EI benefits ($) as a result of active EI claimants participating in EBSMs who returned to work | 7,800,000 | 4,000,000 | 4,142,331 | 4,537,329 |
| * Targets as set out in the original Canada/PEI LMDA and 1998/99 Business Plan. ** Figures similar to those published in the 1998 EI Monitoring and Assessment Report. |
||||
In this subsection, we use survey and administrative data to look at measures of success for the just five EBSMs under study over the period of the evaluation (April 26, 1997 to October 31, 1998). Note that these figures cannot be compared to figures presented in the previous section because the latter apply to the full set of EBSMs and are shown by fiscal year. We concentrate on just these EBSMs because these were the focus of the evaluation survey, the employment results from which could be compared to the return to work indicator in the administrative data. The measures of success correspond to the accountability targets as discussed in the previous section. At the same time, we use the opportunity to look at the data and methods used by HRIB to measure success. Note that because different methods and datasets are being used, we expect results to differ as well.
Returns to Work
Exhibit B.2, Panel 1, presents the results of our computations for the return to work measure of success over the period in question. Here, we first show in the unnumbered row of that panel what the administrative data yield for returns to work and then in the next three rows (1 through 3) the number of returns to work based on the survey employment rates applied to participant counts in the administrative data. For these, we multiplied (i) the count of the total number of EBSM participants from the administrative data (5,731) by (ii) three employment rates derived from the survey results. The latter are (1) employed following the intervention (86 percent); (2) employed for 12 consecutive weeks following the intervention (74 percent); and (3) currently employed (59 percent). Note that the second employment measure — employed for 12 consecutive weeks —comes closest to the measure of employment expressed in the Canada/PEI LMDA.
Comparing the unnumbered row to the next three rows of Panel 1 of Exhibit B.2 indicates that returns to work as measured in the administrative data (2,741) appear to underestimate actual returns to work as reported in the survey (3,381 to 4,927). Taking 2,741 as a percentage of the figures in rows 1 to 3 indicates that just 56 to 81 percent of actual returns to work (as reported by survey respondents) are captured in the administrative data.42 The 1998 EI Monitoring and Assessment Report suggested that under-estimation might be due to the fact that the data systems used to track clients and the exchanging of data were not fully operational initially.
| Success Measure | Results |
| Panel 1 — Number of EI clients returning to work | 2,7411 |
| (1) 4,9272 | |
| (2) 4,2412 | |
| (3) 3,3812 | |
| Panel 2 — Unpaid EI benefits ($) | 4,713,4951 |
| (1) 4,591,6883 | |
| (2) 3,062,5733* | |
| * Figures apply only to the five EBSMs under study in the survey. For this reason, these figures should not be compared to figures in the previous exhibit which covered all EBSMs. 1.Produced straight from HRIB Results Files administrative data. 2.Based on participant count multiplied by three employment measures derived from survey data: (1) employed following intervention, (2) employed for 12 consecutive months (3) currently employed. See text for details. 3.Computations based on (1) EKOS and (2) ARC methods, as described in the text. |
|
In Exhibit B.2, Panel 2, we produce the results of our efforts to measure success of the five EBSMs with regard to generating unpaid EI benefits over the period in question. At the same time, we point out possible problems in data and methods.
We start with the amount of unpaid EI benefits straight from the administrative dataset. In computing this amount, we first merged the participants file with the Results Files and came up with matches for only 1,047 participants out of 2,170 RTW cases from Results File 1997/98 and 2,358 cases out of the 2,963 from Results File 1998/99. Filtering out the participants with an intervention start date earlier than April 26, 1997 or later than October 31, 1999 decreased the number of observations to a total of 2,741. Of these, only 1,285 had unpaid benefits greater than zero; the rest were paid all their entitled EI benefits before returning to work, thus resulting in no unpaid benefits being realized. Then, we summed unpaid benefits on the Result Files for these 1,285 cases. As the unnumbered row of Panel 2 of Exhibit B.2 indicates, this resulted in a total of $4,713,495 in unpaid EI benefits for the five EBSM interventions entered into between April 26, 1997 and October 31, 1998.
In the next two rows (1 and 2) of Panel 2 of Exhibit B.2, we show the products of two alternative approaches to computing unpaid EI benefits. The first approach is to sum the differences between EI benefits paid and EI benefits entitled across all individuals. The second approach is the so-called Applied Research Consultants (ARC) approach, which involves summing the differences between EI benefits paid after the RTW date and remaining eligible EI benefits across individuals. The discussion of each of these approaches follows.
In this approach, we first determined which of the individuals on the participants file had an RTW. The reason is that there are other reasons for unpaid EI benefits besides a return to work, and computing unpaid benefits for participants regardless of an occurrence of a return to work would overestimate actual unpaid benefits for purposes of the Canada/PEI LMDA.43 (Likewise, there are persons on the Results File who do not have a matching record on the participant file, if their return to work is associated with a non-EBSM intervention.44) We computed unpaid EI benefits for 1,407 EBSM participants from the participant file who also had an RTW tag from Results Files 1997/98 and 1998/99 and unpaid EI benefits greater than zero. For each of these individuals, we subtracted EI benefits paid from EI benefits entitled and summed these differences across all individuals. The sum of unpaid benefits associated with these individuals is $4,591,688, as shown in row 1 of Panel 2 of Exhibit B.2. This is $121,807 lower than the total sum of unpaid benefits taken straight off the administrative data ($4,713,495), as indicated in the un-numbered row.
There are several reasons for the discrepancy between the computations based on HRIB figures and EKOS computations which involve explicitly subtracting benefits paid from benefits eligible. First, only EI claims linked to an intervention were included in EKOS calculations, while unpaid benefits calculated using HRIB figures included some EI claims unrelated to an intervention captured in our population. Second, HRIB calculations were based on the straight multiplication of unpaid weeks by a fixed benefit rate, while EKOS also considered the amount of EI benefits actually paid, which in some cases is based on the actual benefit rate which could vary during the claim period. Third, in EKOS calculations, the existence of missing EI or intervention information means, by default, unpaid benefits being set equal to zero, thus decreasing the amount of benefits unpaid.
A second, alternative method of calculating unpaid EI benefits has been suggested by Applied Research Consultants (ARC) and we use it here. This method explicitly takes into account the timing of the return to work and the fact that benefits may be paid to a client after a return to work (at a reduced rate). This approach is based on the HRDC Status Vector file, which contains information on EI eligibility and benefits paid, and the Results File, which contains return to work (RTW) flags. The results of this exercise are presented in row 2 of Panel 2 of Exhibit B.2. In general terms, unpaid EI benefits were computed as the difference between: (1) EI benefits paid after the RTW date (often paid at a reduced rate), and (2) remaining eligible EI benefits. This differs from our approach which involved subtracting total benefits paid from total benefits eligible.
To elaborate, in order to calculate unpaid EI benefits under the ARC approach, the following detailed steps were taken for each individual:
The calculation was initially performed on the 2,741 persons on the Results File who participated in the five EBSMs under study (i.e., excluding services that were used exclusive of those EBSMs) within the defined time frame and matched with the participant population from the main file. Missing information on RTW date or status vector information reduced the number of cases to 1,281, however. These remaining individuals had "valid" dates, i.e., an RTW flag that was dated before the end of their entitlement period, and therefore could potentially generate "savings." Summing the computed unpaid benefits for these individuals (693 of whom had unpaid benefits greater than zero) produced a total of $3,062,573 in unpaid EI benefits, as shown in row 2 of Panel 2 of Exhibit B.2.
This is much less than the amounts shown in the other two rows of Panel 2. The main reason is that the ARC approach is the most rigid of the methods of calculating unpaid benefits depending on a variety of variables. Unfortunately, in Canada/PEI LMDA information systems, the availability of data on some variables retrieved from Results Files was poor. For instance, large proportions of cells for the RTW date were simply empty. Therefore, unpaid benefits of most of clients were assigned zero, thus significantly decreasing computed unpaid benefits.
|
A total of 30 in-person interviews were conducted with key informants. These interviews were held from March 8 to March 12, 1999, when three members of the study team travelled to Prince Edward Island. As most of the interviews involved two respondents, a total of 52 key informants were consulted. The average interview was one and a half-hours long. Interviews were conducted in Charlottetown (24 interviews), Summerside (four interviews) and Montague (two interviews) with informants from these cities as well as some respondents from other areas of the Island. The interview locations were set to be convenient for most respondents.
The key informant interviews fell into the following three categories:
Three interview guides — one guide designed for each of the three groups were utilized in the key informant interviews. All respondents were sent a copy of the appropriate interview guide in advance of their scheduled interview appointment. In addition, they were sent an introductory letter explaining the purpose of the interview, the fact that the interview is voluntary, and that the fact their comments are protected under the Federal Privacy Act. Key informants were interviewed in their preferred official language.
A total of 12 focus groups were conducted from March 8 to March 12, 1999 when members of the study team travelled to PEI. The discussions were held with four types of participants: HRCC and provincial delivery/front-line staff (one group); stakeholders45 (two groups); clients (six groups); and employers (three groups; one of the stakeholder groups also included some employers). The combined stakeholder/employer group was targeted to the francophone community and was conducted in French, and all other discussions were held in English.46 At least one group with clients was held in each of the five Human Resource Centre of Canada (HRCC) regions of the Island — in Charlottetown, Summerside, O'Leary, Montague and Souris. The sixth client focus group was targeted to francophone clients and was offered in French. These various client focus groups were conducted in order to assess the views of clients in the urban, central rural and more remote rural areas of the province, and the different employment situations in each area. Each discussion was two hours in duration, and was held in the evening (with the exception of the one staff focus group which was conducted in the afternoon). The focus groups were held at a focus group facility in Charlottetown and in hotel meeting rooms in the other four centres.
Using lists of potential participants developed with the assistance of the Evaluation Committee, interviewers recruited the participants by telephone. For employers and stakeholders, the lists were comprised of persons who had accessed Employment Insurance (EI) programs and who represented specific demographic and geographic areas of PEI. Clients were recruited at random from a client sample list provided by the Evaluation Committee. In these telephone contacts, the interviewers explained to prospective participants the purpose of the discussion and study sponsor; details on the time and location of the focus group; the fact that participation is voluntary; the fact that the discussion would be audio-tape recorded but that their comments would be kept strictly confidential; and the fact that an honorarium of $50.00 as well as travel expenses would be provided to all participants (except the front-line government staff). Persons agreeing to participate were given a reminder phone call a day before their scheduled discussion.
In order to ensure the participation of eight to ten people in each discussion, we endeavoured to recruit 12 confirmed participants for each focus group. Unfortunately, there were bad weather conditions in PEI on some of the evenings when groups were scheduled, so the participation was low for some of the focus groups (see Exhibit C.1). In an attempt to make up for this, the focus group questions were sent (by facsimile) to 31 persons who were unable to attend discussions during our visit to PEI. Only two people — both employers from Charlottetown — returned their responses and these comments are incorporated in the findings.
The distribution of the focus groups, including the number of participants per group, is summarized in Exhibit C.1.
Four focus group guides — one guide for each of the four types of participants — were developed for the group discussions. Following the appropriate guide, the moderator asked the group the questions in a non-directive way, probing for clarification and more detail when necessary, and intervening as appropriate to involve all participants and keep the discussion on topic.
Participant Survey
The participant data files were originally developed to include participants who participated in LMDA employment programs and services at any time between April 26, 1997 and October 31, 1998. These were compiled from a participant file (n=5,409) and five administrative data files (HRDC, Hull, Quebec, names and addresses file and T1 file (n=45,513); Status Vector file (n=45,963); Status vector file with Benefit Period Commencement (BPC) and BVT data (n=45,963); and Record of Employment (ROE) file (n=45,861)). These files were aggregated, yielding a single data file containing information for 5,409 participant cases, with the individual client as the unit of analysis. This was not equal to the sum of all the cases from the administration and data files because clients who had taken part in more than one intervention could appear in more than one file. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 3,744 individuals.
The survey sample was randomly drawn from the final data file using a three to one "sample to survey completion" ratio for each different participant group (i.e., three times as many participants were sampled as were expected to complete the survey). For all groups except Employment Assistance Services (EAS) and Enhanced Feepayers, there were not enough cases available to obtain this three to one ratio, thus for an expected total of 1,164 survey completions, a total final sample of 2,483 cases was drawn from the data file of 3,744 program participants.
Based on the matrix of issues and indicators developed for the evaluation, a survey was designed for clients who have participated in LMDA funded programs in Prince Edward Island. From the initial review of the instrument by the Joint Evaluation Committee (JEC) in January 1999, a number of changes were made, including wording changes; the addition of a number of questions related to respondents' work profiles, LMDA programs, attitudes, and use of income assistance; and modifications to response categories.
The pre-test was carried out in order to simulate the conditions to be encountered during the actual survey. The objective was to test the survey instrument in terms of the length of time required for the interview, as well as to ensure the sequencing and clarity of the questions and appropriate wording and flow were. On February 23, 1999, a total of six interviews were completed with an average length of 29.5 minutes. The pre-test results prompted several revisions of the instrument, such as changes to skip logic and wording changes. Notably, the pre-test results demonstrated that the instrument was longer than planned; thus a list of suggested changes to shorten the survey was developed and submitted to the JEC for their approval. Efforts to shorten the survey involved developing a list of questions to be deleted and/or merged with other questions (i.e., in order to collect the same type of information using fewer questions).
Following approval of the suggested changes and modification of the survey instrument to reflect these changes, another pre-test was conducted on March 11, 1999. The results of eight completed interviews showed that the average length of the survey was now 28.2 minutes. Additional efforts to shorten the survey were made and another pre-test was conducted with three survey completions the following day, on March 12. This pre-test yielded an average estimated time of 24 minutes for the survey. Although the final pre-test revealed that the instrument was somewhat longer than the time allotted for the survey, additional resources were supplied by the client to offset the costs of the longer survey.
Fieldwork for the survey began on April 5, 1999 and was completed on June 10, 1999. A major delay in the fieldwork occurred on April 14 when it was discovered that different protocols were used by HRDC in PEI and National Headquarters in Ottawa to extract the population of participants from the administrative data files. The consequent discrepancy in the population characteristics of the participants pulled using these two extraction protocols required that all fieldwork come to a stop until a resolution to the problem had been reached. The differences in the two different selection strategies were as follow:
On May 17, 1999, EKOS received the new participant data files from HRDC, including the population of all participants and all participant administrative data files. The new participant data file was rebuilt and matched to the old participant data file so that only new cases were pulled from the new file. This new file was also matched to the list of comparison group cases that had already completed the comparison group interview. Respondents who had responded to the comparison survey and were listed as participants in the new participant data files (n=48) were also ineligible for selection as participants in the new wave.
In order to limit the amount of time that elapsed between the first period of data collection and the continuation of the fieldwork, fieldwork for the participant survey resumed on May 12, 1999, prior to receiving the new participant data files. This early return to the field was also prompted by the fact that the research team felt it would be prudent to collect extra participant cases concurrent to the comparison group fieldwork, which began in the field at the same time. In this way, there would be a sample of completed participant surveys that could be compared to the earlier participant cases, as well as to the comparison group cases. These comparisons would provide information about any effect that the time delay might have had on participant responses.
An additional difficulty that arose from problems in defining the population of participants concerned the loss of time in the field. Specifically, fieldwork was quickly moving into the May long weekend, a weekend which typically marks the beginning of the tourist season on the Island and a return to work for many people, including perhaps significant proportions of the participant survey sample. Given this potential source of bias, all efforts were made to complete the survey fieldwork before the long weekend. When it became apparent that this was not going to occur, the wording for both surveys was modified so that questions concerning post-intervention employment status and activities referred clients to report on their employment history only up until April 10, 1999 (i.e., the mid-point of the original fieldwork for the participant survey). In this way, the confounding effect of a mass return to work heralded by the beginning of the tourist season was avoided.
Following the completion of fieldwork, it was also discovered that no reliable means existed to distinguish reachback clients from active EI claimants on the basis of the available administrative data. The administrative data lacked a reliable flag for reachbacks and claimants; thus reachback status was computed on the basis of the BVT and BPT variables derived from the Status Vector files. The BVT variable records the receipt of EI claimants' most recent report cards, while the BPT is the week code of the theoretical end of EI eligibility. At the actual end of a claim, the BVT and BPT codes are reconciled and will be equal. When the proportion of reachbacks participating in each EBSM was determined using the BVT variable, the true proportion of participating reachbacks was over-estimated because claimants for whom a most recent (but not last) report card had been received were categorized as reachbacks. The only way to make a positive determination of claimant status is to wait a sufficient period of time after the claim period begins for the BVT and BPT variables to be reconciled and for the quarter-annual data extraction to take place following this reconciliation for the data to become available to researchers. As such, at the time of this report the claimant status of roughly one in 10 participants was still in question.
The response rates and refusal rates for participants by program type are presented in Exhibit C.2. The response rate is the proportion of cases from the functional sample who responded to the survey, while the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey. The functional sample factors out the attrition in the survey, leaving only the sample which resulted in completions, refusals, and those numbers attempted but not reached before the completion of fieldwork (e.g., retired phone numbers, respondents who were unavailable for the duration of the survey, respondents who were unable to participate due to illness or some other factor, etc.). Attrition includes numbers not in service, duplicate phone numbers, respondents who do not speak either English or French and respondents who indicated no knowledge of the topic.
The overall margin of error for the survey is 
2.6 percent. That is, the overall survey results are accurate within 2.6 percentage points, 19 times out of 20. It should be noted that the response rate for the survey was fairly good, ranging from 77.4 percent among Purchase of Training participants to 33.7 percent for Enhanced Feepayers, with an overall response rate of 59.5 percent. The overall refusal rate was also quite satisfactory (4.2 percent) and ranged from 8.6 percent for Enhanced Feepayers to 0.7 percent for Purchase of Training participants.
The comparison group sample was drawn from a file of Employment Insurance claims that were active in 1998 and dormant EI claims (claims that were not being processed) from 1994 to 1998 that was provided by Human Resources Development Canada. This selection method produced a file of 41,549 claimants from which to draw the comparison group sample.
The comparison group data file was matched to the participant data file based on the time periods for which members of the comparison group were receiving EI. To accomplish this matching, three time periods were defined according to observed values for program end dates in the population of program participants. That is, within the population of program participants, the end dates of participants' most recent interventions ranged from April 26, 1997 to October 31, 1998 and this time period was divided into three time periods. For comparison group sampling purposes, the following time periods were derived: April 26, 1997 to September 30; October 1, 1997 to April 30, 1998; and May 1, 1998 to October 31, 1998. Reference data flags were then computed using the mid-point in each of these time periods (June 1, 1997, December 1, 1997, and July 1, 1998) so that if an individual in the comparison group was EI eligible at the reference date (at the mid-point of the time period), that individual would fall into the time period cohort. This meant, however, that these were not necessarily mutually exclusive cohorts because an individual could have been EI eligible at more than one of the reference dates.
Based on the participant population characteristics, for each time period cohort a listing was produced of the time in weeks between the end of the latest intervention and the start date of the most recent EI eligibility period. These time frames were further broken down into five categories based on the amount of time into the EI eligibility period that the participant's intervention came to an end. These categories were 13 weeks or less, 14 to 26 weeks, 27 to 39 weeks, 40 to 52 weeks, and 53 weeks or more. Each comparison group cohort was then similarly broken down into the same five categories based on the time in weeks between the reference date (the mid-point of one of the three time period cohorts) and the start date of the most recent EI eligibility period. (see Exhibit C.3).
The comparison sample was drawn in the same proportions as were observed for each of the three time period cohorts in the participant population. For example, if 13.3 percent of the population of participants fell into the first time cohort (i.e., the most recent intervention end dates were between Apri1 26, 1997 and September 30, 1997), the comparison group was sampled to ensure that 13.3 percent of cases were EI eligible at the mid-point of this time period (i.e., June 1, 1997). The comparison group sample was further stratified by the number of weeks between the end date of the latest intervention and the start date of the most recent EI eligibility period. For example, if 10.1 percent of the participant April to September time cohort had 14 to 26 weeks between the end date of their latest intervention and the start date of their most recent EI period, this meant that 10.1 percent of the comparison group population in the April to September time cohort (they were EI eligible at the mid-point of this time period, on June 1, 1997) was sampled from those with 14-26 weeks between the reference date (the mid-point of the time period) and the beginning of their most recent EI eligibility period.
To correct for the fact that the comparison time cohorts are not mutually exclusive, each time period cohort was sampled separately and a flag was computed to identify sampled cases. As such, it was possible to track these cases and not include them when sampling from subsequent time periods. Thus the final comparison group sample consisted of 2,637 cases in three mutually exclusive time period cohorts from a population of 41,549.
| Time (Weeks) into EI Eligibility that Program Ended | Participants in Population | Total Sample from Comparison Group Population | ||
| # | % of Subtotal | # | % of Subtotal | |
| April 26, 1997 to September 30, 1997 | ||||
| Less than 13 | 26 | 8.9 | 20 | 8.9 |
| 14-26 | 46 | 15.2 | 35 | 15.2 |
| 27-39 | 84 | 27.8 | 63 | 27.8 |
| 40-52 | 67 | 22.2 | 50 | 22.2 |
| 52 and over | 79 | 26.2 | 59 | 26.2 |
| Subtotal | 302 (8.6% of total population) | 227 (8.6% of total sample) | ||
| October 1, 1997 to April 30, 1998 | ||||
| Less than 13 | 239 | 15.2 | 179 | 15.2 |
| 14-26 | 437 | 27.8 | 327 | 27.8 |
| 27-39 | 214 | 13.6 | 160 | 13.6 |
| 40-52 | 202 | 12.8 | 150 | 12.8 |
| 52 and over | 482 | 30.6 | 360 | 30.6 |
| Subtotal | 1,574 (44.6% of total population) | 1,176 (44.6% of total sample) | ||
| May 1, 1998 to October 31, 1998 | ||||
| Less than 13 | 52 | 3.1 | 38 | 3.1 |
| 14-26 | 155 | 9.4 | 116 | 9.4 |
| 27-39 | 446 | 27 | 334 | 27 |
| 40-52 | 319 | 19.3 | 239 | 19.3 |
| 52 and over | 679 | 41.1 | 507 | 41.1 |
| Subtotal | 1,651 (46.8% of total population) | 1,234 (46.8% of total sample) | ||
| Overall total | 3,527* | 2,637 | ||
| * This total represents the sample frame total rather than the population total. The sample frame consists of those members of the population with complete data, including telephone numbers, to permit sampling for the survey. | ||||
A comparison group survey instrument was developed in the early winter of 1998, and reviewed by the Joint Evaluation Committee (JEC) in January 1999. Based on the JEC's review, a number of changes were made to both the participant and comparison group surveys, including wording changes; the addition of a number of questions related to respondents' work profile, Labour Market Development Agreement programs, attitudes, and use of income assistance; and modifications to response categories.
Throughout February and March, as pretests were being conducted with the participant survey, the comparison group instrument was modified to reflect ongoing changes being made to the corresponding participant survey. Because the instrument was virtually identical to the participant survey, with the exception of several questions that were not asked of non-participants and slightly different wording of some questions, all pretest information from the participant survey was equally applicable to the comparison group survey. Thus, modifications to the sequencing and clarity of questions, as well as checks on wording and flow, were made to the comparison group survey instrument on the basis of participant survey results. Furthermore, the length of the comparison group survey was easily deduced from results of the participant survey pretest because we were able to record the difference in the number and type of questions between the two survey instruments.
The response rate for the comparison group survey47 is presented in Exhibit C.4. The response rate is the proportion of individuals from the functional sample who responded to the survey. Conversely, the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey. The functional sample factors out the attrition in the survey, leaving only the sample which is comprised of completions, refusals, and those numbers attempted but not reached by the completion of fieldwork. (e.g., appointments for interviews that were not kept, retired phone numbers, respondents who were unavailable for the duration of the survey). Attrition includes numbers not in service, duplicate phone numbers, respondents who did not speak either French or English and respondents who indicated no knowledge of the topic or were ineligible to take part (e.g., LMDA participants).
The overall margin of error is 4.4 percent. That is, the overall survey results are accurate within 4.4 percentage points, 19 times out of 20. The response rate for the survey was 29.2 percent and the refusal rate was 11.9 percent. For the purpose of analysis, the comparison group survey data were weighted according to age, sex and time cohort.
| Total | |
| Initial sample | 2,637 |
| (less) Unused sample | 648 |
| (less) Attrition | |
| Number not in service/Invalid number | 222 |
| Duplicate number | 13 |
| No knowledge of topic/ineligible | 77 |
| Language barrier (did not speak English or French) | 17 |
| Functional sample | 1,660 |
| Other numbers retired (not due to attrition) | |
| No answer/busy | 710 |
| Unavailable for duration of survey | 214 |
| Other/illness | 54 |
| Non-response | |
| Refusal | 193 |
| Incomplete refusal | 4 |
| Total non-response | 197 |
| Total completed | 485 |
| Refusal rate | 11.9% |
| Response rate | 29.2% |
| Margin of error |
4.4%
|
Eleven dependent variables representing key employment, earnings, and income support use outcomes were tested in the multivariate models. These variables are based for the most part on survey data. The models were run for all participants and separately for certain socio-demographic, rural/urban and claimant status segments. The outcomes correspond to the key objectives of the Canada/PEI LMDA and the Employment Benefits and Support Measures, which are to lead to sustained employment and a reduction in dependency on income support. Logistic (logit) regression was used for binary dependent variables (yes/no) and Ordinary Least Squares (OLS) regression was used for continuous dependent variables (numeric values). These variables follow:
Along with the intervention (EBSM) binary variables (e.g., participant in Employment Assistance Services or not), a common set of explanatory (control) variables was introduced into the models for each dependent variable (the outcomes). The purpose was to assess (or control for) the influence of other factors on the intervention's impact on the outcomes. These other factors included the time since the intervention and antecedent socio-demographic and employment-history variables. These variables were selected because they were thought, a priori, to have an impact on outcomes and because participants and non-participants differed with respect to these variables.49 Noting that "intervention" here refers to the end of the intervention for participants in Canada/PEI LMDA EBSMs and to the reference date for comparison group members. The variables entered into the models follow:
|
|
Employment Rates
Exhibit E.1 presents various employment outcomes for participant and comparison group survey respondents. Overall, these results suggest that some advantage may exist for participant claimants compared to comparison group claimants.
The first panel of Exhibit E.1 shows that a large majority of both the participant and comparison groups indicated that they have had a job since the end of the program or reference date. The incidence of employment at some time in the post-program period was high for all program respondents and ranged from a low of 75 percent of Employment Assistance Services (EAS) participants to a high of 97 percent of SE respondents. For claimants, participants were more likely than comparison group respondents to indicate that they have been employed at some time in the post-program period (86 versus 81 percent respectively), although the opposite was true when we compared reachbacks for these two groups (81 versus 88 percent respectively).
A review of employment rates according to respondents' socio-demographic profiles (not shown) indicates that the younger the participant and the higher his or her education and household income, the more likely he or she was to have found a job following the intervention.
Employment Stability
Data on two measures of employment stability were collected in the participant and comparison group surveys: the proportion of respondents who have worked for 12 consecutive weeks following the end of the program or reference date, as well as the number of employers they have had since that time.
The second panel of Exhibit E.1 shows that the majority of participants in all programs had worked 12 consecutive weeks since the end of their intervention, with the highest incidence occurring among Self-Employment, Targeted Wage Subsidies (TWS) and Feepayer participants (95, 82 and 82 percent, respectively) and the lowest incidence occurring among Training and EAS participants (69 and 59 percent, respectively). Once again, while participant claimants were more likely than comparison group claimants to have been employed for 12 consecutive weeks (76 versus 64 percent, respectively) the opposite was true of reachbacks (71 versus 76 percent, respectively). The incidence of respondents having worked 12 consecutive weeks since the end of their program also rose with level of education (not shown).
Results for the second measure of employment stability, number of employers since the end of the program or reference date, is presented in the third panel of Exhibit E.1. Although there was little variation between participants in different programs on this measure, SE and TWS participants were somewhat more likely than participants overall to have had only one employer (91 and 85 percent, respectively) and to have had fewer employers on average (1.1 for each versus 1.4 overall). The employment pattern of comparison group claimants seems to have been slightly more stable than that of participant claimants. Comparison group claimants were more likely than their participant counterparts to have had only one employer (74 versus 60 percent, respectively) and to have had a lower mean number of employers (1.1 versus 1.4 respectively).
Interesting sub-group differences based on respondents' socio-demographic profiles (not shown) also revealed that number of employers in the post-program period declined with the respondents' age and the number of dependants, and rose with household income and for single respondents.
Employment Status Outcomes
Participant survey respondents were asked about their employment status at two points in time following their intervention: at one week following the end of the program and at the time of the survey. Overall, these results show a positive shift in employment between these two times for Employment Benefit and Support Measures (EBSM) participants, with the largest positive shifts in employment occurring for full-time year-round jobs. These findings suggest that some employment gains which may be attributed to the EBSMs tended to persist.
A comparison of the first and second panel of Exhibit E.2 reveals that, by the time of the survey, employment rates had risen somewhat among participants relative to their first post-program week (57 versus 50 percent, respectively). Although the incidence of full-time seasonal employment fell by the time of the survey (from 14 to 10 percent), this was more than offset by the incidence of full-time year round employment, which rose from 20 percent in the first post-program week to 26 percent at the time of the survey. Further, the incidence of respondents who were unemployed and looking for work fell between the first post-program week and the time of the survey (37 to 29 percent, respectively). This drop in the unemployment rate is attributable to both increased employment rates for most participant groups, as well as increases in the proportion of respondents in all groups, except SE, indicating that they were out of the labour force. The rates of full-time year-round employment rose with education level and fell with age, and the rates of official unemployment rose with age and fell with education level and with household income.
The only exception to the rise in employment rates among participants occurred for Job Creation Partnerships (JCP) and TWS participants. The decrease in employment among JCP and TWS respondents was due to the loss of full-time seasonal jobs between the first post-program week (25 and 21 percent respectively) and the time of the survey (11 and 14 percent respectively). Another notable shift in the proportion of respondents in different types of jobs occurred among Feepayers, where the proportion of respondents in full-time year-round employment rose considerably from 22 percent in the first post-program week to 40 percent at the time of the survey. This latter finding highlights the fact that, for some programs, a longer gestation period is required before an accurate assessment of program impacts can be made.
Exhibit E.2 also presents the employment status of comparison group respondents. The data show that the overall employment rate among participants was somewhat higher than that of the comparison group at the time of the survey, thus implying some advantage to EBSM participants over non-participants. Participants were also slightly less likely to indicate that they were out of the labour force at the time of the survey, and had slightly lower rates of unemployment (i.e., unemployed and looking for work). Little difference in the proportion of respondents holding different types of jobs at the time of the survey was observed for the participant and comparison groups, although comparison group claimants were much less likely than other groups to be employed full-time year-round at the time of the survey.
In this section, we examine the characteristics of respondents' current or most recent job for those respondents who have been employed at some time in the post-intervention period. On average across all programs, we see that only one in five program participants who had a job prior to the program returned to this same job in the post-program period (Exhibit E.3). Training and EAS participants were the most likely to have done so (31 and 29 percent, respectively). Comparison group respondents were much more likely than participants to return to the same job. The incidence of respondents returning to their pre-program job in the post-program period declined with education level.
Panel 2 of Exhibit E.3 shows that little variation existed in the number of hours per week worked by different participants in their post-program job, with the exception of SE participants who reported working an average of 51.4 hours per weeks compared to the 40-43 hour range for the other groups. The average number of hours worked per week was higher among men and declined with education level.
SE and Feepayer participants were most likely to report being employed year-round (75 and 61 percent, respectively) while JCP participants were more likely to report seasonal employment and had the highest rate of casual or contract employment (17 percent). Further, reachbacks were more likely than claimants to report being employed year-round (59 versus 47 percent, respectively for participant; 46 versus 35 percent, respectively for the comparison group). Overall, participants were more likely to be employed year-round than comparison group respondents.
The highest average weekly earnings were reported by Feepayer, JCP and Training participants ($440, $433, and $419 per week, respectively), and the lowest earnings were reported by TWS and EAS participants ($354 and $333 per week, respectively). Further, comparison group respondents reported higher average weekly earnings than participants. Sub-group differences based on client demographics (not shown) reveal that weekly earnings rose according to education level and household income and were higher among men.
In this section, we present survey results for three post-intervention outcomes: number of weeks jobless (duration), number of weeks looking for work while jobless (intensity), and job search activity. "Jobless" individuals are defined as people who are officially unemployed (i.e., unemployed and looking for work) plus those who are not in the labour force.
Duration of Jobless Spells
The first panel of Exhibit E.4 presents the participant and comparison group survey results for the duration of jobless spells following the intervention, scaled by the time since the intervention or program reference date. On average, participants were not working for 30.9 percent of the time since their intervention. The lowest rates of joblessness were observed for SE (12.6 percent) and Feepayer participants (22.4 percent), and the highest were observed for JCP and EAS participants (44.4 and 39.6 percent, respectively). Participants tended to be unemployed for a smaller proportion of the post-intervention period than the comparison group, and this was true for both claimants (27.2 versus 45.9 percent, respectively) and reachbacks (34.3 versus 38.2 percent, respectively). The contrast in the proportion of time that participant and comparison group respondents were jobless in the post-intervention period suggests that participation in EBSMs conferred an advantage, especially among EI claimants.
Another potential positive outcome of the EBSMs concerns the number of weeks clients searched for work in the post-program period while jobless. These results were scaled by weeks since the intervention, resulting in a measure of job-search intensity. This also may be considered a measure of interest in finding a job.
As shown in the second panel of Exhibit E.4, participants looked for work for an average of three-quarters (75.4 percent) of the post-intervention period while jobless. Fairly similar proportions (69-81 percent) were observed across EBSMs, apart from the decidedly lower percentage among SE participants (38.6 percent). Participants had higher job-search intensity than the comparison group. Among participants, there was little difference between active EI claimant and reachbacks, but among comparison group member, reachbacks had higher job search intensity than active claimants (67.8 versus 58.6 percent).
Another indicator of the extent to which EBSMs have positively impacted participants is the extent to which participation in the EBSMs has reduced reliance on income support. In this section, we present results regarding participants' post-intervention use of two forms of income support: Social Assistance (SA) and Employment Insurance (EI).
Social Assistance
Only a minority of participants used SA since the end of the intervention (13.1 percent) and on average, SA was used for 7.4 weeks (panels 1 and 2, Exhibit E.5). The lowest rate of post-program SA use was observed for SE respondents (4.1 percent) and the highest occurred among EAS participants (24.5 percent). The highest mean percentage of weeks since the intervention that SA was received was observed for both JCP and EAS participants (13.5 and 13.3 percent, respectively).
Despite only a small minority of participants making use of SA, participants were more likely than non-participants (in particular, comparison group claimants) to make use of SA in the post-program period. The mean proportion of post-program weeks in which SA was collected was also lower among comparison group respondents, and was lower among participant claimants than participant reachbacks (5.9 versus 12 percent, respectively).
Respondents to the surveys who had been on SA were also asked if their employment program had reduced their reliance on income support. For participants, one-third of those who had received SA benefits indicated that their employment program had helped to move them off social assistance and toward employment (panel 3, Exhibit E.5). The proportion of respondents who responded in this way did not vary considerably across different programs, with the exception of JCP participants where fully 52 percent indicated that their employment program had had this effect. Overall, participants were more likely than comparison group respondents to indicate that their program had helped them move to employment and off social assistance.
Employment Insurance
The last two panels of Exhibit E.5 present the incidence and rate of Employment Insurance (EI) use in the post-program/reference date period. Overall, 34 percent of participants indicated that they had collected EI in the post-program period. The highest incidence of EI use occurs among TWS participants (43.9 percent) and the least among SE participants (8.3 percent). The same pattern is observed with respect to the rate of EI use, with TWS participants collecting EI for the highest mean proportion of post-program weeks (19.5 percent) and SE participants collecting EI for the lowest mean proportion (3.7 percent).
Unlike the results pertaining to SA use and consistent with findings from the pan-Canadian study,50 EI use among participants and comparison group members implies an advantage for EBSM participants, who were much less likely to use EI in the post-program period. Further, participant reachbacks were somewhat less likely than participant claimants to have used EI in this time (25.4 versus 40.5 percent, respectively). The rate of EI use also shows that participants collected EI for a smaller mean proportion of post-program weeks than comparison group members. Rate of EI use also reveals that reachbacks in both the participant and comparison groups (12.7 and 18.6 percent, respectively) collected EI for a smaller proportion of post-program weeks than claimants in each of these groups (14.5 versus 23.6 percent, respectively).
The impacts for different employment outcome indicators can be summarized as follows:
|
| Post-Intervention Dependent Variables (Outcomes) | Participants | Comparison Group | Overall | |||
| n | Percent or Mean in Category | n | Percent or Mean in Category | n | Percent or Mean in Category | |
| Currently employed (proportion) | 1,164 | 59% | 470 | 69% | 1,634 | 61% |
| Currently full-time employed (proportion) | 1,164 | 29% | 470 | 18% | 1,634 | 26% |
| Currently seasonally employed (proportion) | 1,164 | 17% | 420 | 33% | 1,634 | 21% |
| Employed three consecutive months following intervention (proportion) | 1,104 | 74% | 481 | 75% | 1,585 | 74% |
| Percentage of weeks employed following intervention (mean) | 913 | 68% | 393 | 62% | 1,306 | 66% |
| Percentage of Weeks looking for work while jobless following intervention (mean) | 745 | 74% | 348 | 61% | 1,093 | 70% |
| Current weekly earnings(mean) | 997 | $ 328.70 | 416 | $ 425.40 | 1,413 | $ 357.18 |
| Absolute change in weekly earnings from one year prior to intervention (mean) | 927 | $ 14.80 | 401 | $ 47.48 | 1,328 | $ 24.67 |
| Percentage change in weekly earnings from one year prior to intervention (mean) | 789 | 234% | 366 | 307% | 1,155 | 257% |
| Percentage of Weeks receiving EI in a new spell following intervention (mean)** | 1,164 | 13.2% | 485 | 27.7% | 1,649 | 17.4% |
| Received SA benefits following intervention (proportion) | 1,123 | 13% | 485 | 8% | 1,614 | 11% |
| * Based on survey results (unless otherwise indicated) for the full sample of participants and non-participants, less observations with missing data or outliers. ** Based on administrative data. |
||||||
| Explanatory/Control Variables | Percent in Category or Mean | ||
| Participant Group | Comparison Group | Overall | |
| Intervention type (vs. non-participant in intervention) ( n=1,649) | |||
| Employment Assistance Services (EAS) | ** | ** | 13% |
| Self-Employment (SE) | ** | ** | 6% |
| Target Wage Subsidies (TWS) | ** | ** | 16% |
| Training/Feepayers (TFP)1 | ** | ** | 35% |
| Job Creation Partnerships (JCP) | ** | ** | 8% |
| Weeks since intervention ended (mean) (n=1,504) | 52.2 | 69.2 | 57.7 |
| Employment status one month before intervention (vs. not in labour force) (n=1,598) | |||
| Employed | 48% | 34% | 44% |
| Unemployed | 38% | 57% | 44% |
| Employed one year before intervention (vs. not) (n=1631) | 81% | 88% | 83% |
| Education level (vs. less than high school) (n=1,647) | |||
| High school certificate | 34% | 29% | 32% |
| At least some post-secondary | 41% | 39% | 41% |
| Age group (vs. <35 years)(n=1,631) | |||
| 35-44 years | 29% | 33% | 30% |
| 45-54 years | 16% | 21% | 17% |
| 55 years and over | 3% | 13% | 6% |
| Male (vs. female) (n=1649) | 48% | 47% | 48% |
| Mother tongue (vs. English) (n=1,647) | |||
| French | 5% | 4% | 5.0 |
| Other | 1.5% | 0.4% | 1.2 |
| Married (vs. non-married) (n=1,647) | 58% | 68% | 61% |
| Minority (vs. not) (n=1,647) | 9% | 6% | 9% |
| No dependents (vs. dependents) (n=1,637) | 44% | 9% | 34% |
| Pre-intervention interest in: | |||
| Being trained (mean 1-7) (n=627) | 5.7 | 4.0 | 5.2 |
| Starting own business (mean 1-7) (n=1,627) | 3.4 | 2.7 | 3.2 |
| Entering labour force (mean 1-7) (n=1,639) | 6.5 | 6.6 | 6.5 |
| Number of separations, 1992-1997 (vs. 2 or less) (n=1,649) | |||
| 3 to 5 separations | 27% | 25% | 26% |
| 6 or more separations | 52% | 61% | 55% |
| No. of weeks received EI since 1992 prior to intervention (vs. 0-24 weeks) (n=1,527) | |||
| 25-52 weeks | 19% | 19% | 19% |
| 53-104 weeks | 24% | 20% | 23% |
| 105 weeks and more | 44% | 38% | 35% |
| Earnings in year prior to intervention (vs.< $5,000) (n=1,523) | |||
| $5,000 — 9,999 | 32% | 35% | 33% |
| $10,000 — 19,999 | 19% | 33% | 23% |
| $20,000 — 29,999 | 7% | 13% | 9% |
| $30,000 and over | 8% | 6% | 7% |
| Received SA in year prior to intervention (vs. not) (n=1,523) | 10% | 7% | 9% |
| Use of other services | |||
| Used self-serve products (vs. not) (n=1,649) | 80% | 53% | 72% |
| Met with a counsellor (vs. not) (n=1,635) | 43% | 18% | 36% |
| Set up an action plan (vs. not) (n=626) | 19% | 6% | 15% |
| Used other services (vs. not) (n=1,643) | 7% | 15% | 9% |
| * Based on survey results for the full sample of participants and non-participants, less observations with missing data or outliers. ** Applicable to total population only. 1. The Training and Feepayers component of the Canada/PEI LMDA are now contained under the title Skill Development. |
|||
|
|
| Last Modified: 2003-03-28 | Important Notices |
![[To Top]](/web/20060127015416im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}