Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Formative Evaluation of the Canada/Manitoba Labour Market Development Agreement

Final Overview Report

Evaluation and Data Development
Strategic Policy
Human Resources Development Canada

March 2000

SP-AH-198-03-00E
(également disponible en français)

The formative evaluation of the Canada-Manitoba Labour Market Development Agreement has been completed and accepted by both orders of government. Evaluation findings have been, and will continue to be considered in the context of administering LMDA programs and services.

The evaluation report will be made available to partners and stakeholders.

The Canada-Manitoba LMDA Joint Evaluation and Accountability Committee would like to acknowledge the contribution of several individuals to the successful completion of this project. In particular, the Committee would like to express our appreciation to all those individuals who shared their views and feedback during the survey, case studies, focus groups and key informant interviews and without whose cooperation this research would not have been possible.

Introduction and Background

Under the terms of the Canada/Manitoba Labour Market Development Agreement (LMDA), the federal and provincial governments agreed that Manitoba would assume responsibility for the design and delivery of active labour market programs and services for the unemployed in the province. This report presents the results of the formative evaluation of Provincial Benefits and Provincial Measures (PBPMs) under the terms of the Canada/Manitoba LMDA. The Canada/Manitoba LMDA was signed on April 17, 1997, and came into force on November 27, 1997.

Evaluation Methodology

Multiple methods were used to collect data for this formative evaluation. Quantitative methods, specifically a survey of 1,393 program participants and a survey of 500 non-participants, were used to obtain information that could be applied to the full population of participants in the PBPMs. As well, qualitative methods consisting of 28 key informant interviews and 12 focus groups were employed to gather in-depth perspectives on the research issues.

The total survey sample included those clients who had participated in an LMDA program or service between November 27, 1997 and August 15, 1999. Qualitative data collection was undertaken in December 1999 and the survey was conducted in January 2000.

Key Findings

Implementation

The implementation of the Canada/Manitoba LMDA has gone well. However, there have been some "growing pains" and there are some outstanding issues to be addressed.

The PBPMs reflect the guidelines, principles and intent of the Employment Insurance (EI) Act and the LMDA. They are consistent with Human Resources Development Canada's (HRDC's) priorities but only in part with Manitoba's priorities. The eligibility criteria of the EI Act limiting access to EI clients does not permit the Province to provide training to the marginally-employed using LMDA funds. This is a major concern for Manitoba (and employers) as it is entering times of skills shortages, however, this is a broader issue that is outside of the realm of the delivery of the PBPMs.

The PBPMs were designed and implemented in partnership with stakeholders, in particular between Manitoba Education and Training and Human Resources Development Canada, but also to some extent with other provincial departments, municipal governments, industry associations, employers and community organizations. Many of these partnerships existed prior to the LMDA and have been strengthened with the implementation of the PBPMs.

The key strengths of implementation and delivery have been: an effective implementation structure and process with good cooperation between federal and provincial partners; the successful transfer of programs and highly experienced staff to the Province with minimal disruption to client service; partnerships with industry, employers, and educational and other organizations that have helped to ensure the relevance of programming to the regional labour market; and the delivery of programs at co-located Employment Centres featuring "one-stop shopping" for clients, with cost-effective support from third-party delivery agents.

PBPMs are harmonized with other provincial and federal initiatives. Although, there is some perceived overlap or lack of coordination between provincial and federal programs for youth, Aboriginal peoples, persons with disabilities and labour market information, the consensus is that programs are mostly complementary.

There are a number of perceived weaknesses identified including the need for more flexibility in the design and delivery of PBPMs so that they are better adapted to local and regional needs; more useful labour market information; better access to programs and services in rural and remote communities; more consultation with and better promotion of the PBPMs to community groups; and better internal communications.

In addition, some staff found program guidelines too general to facilitate consistent application among individual staff and between employment centres. As well, human resource issues were also noted, including the desire for a resolution of concerns related to the job classification of staff transferred from HRDC, clarification of staff roles and responsibilities and alleviation of staff anxiety over the future of the LMDA and their job security.

Clients

The survey evidence indicates that participants in the PBPMs are representative overall of EI clients in Manitoba (e.g., in terms of equity group status and other socio-demographic characteristics). PBPMS are largely relevant to the needs of clients in Manitoba. Most clients indicated that the PBPMs met their needs and expectations and impacted positively on their lives. These clients noted that the PBPMs helped them gain self-confidence, gave them a sense of direction and increased their ability to get a job.

Clients expressed high satisfaction with the timeliness and accessibility of services, self-serve resources, and the quality of programs and services.

The few areas about which clients expressed dissatisfaction involved aspects of service delivery rather than the PBPMs themselves and included the lack of up-to-date training courses as well as insufficient remuneration for Wage Subsidies and Apprenticeship participants. As well, 78% of Enhanced Fee Payers indicated that the requirement to contribute to the cost of training made it difficult (to a moderate or large extent) to access training.

With respect to accessibility to programs and services in the official language of their choice, evidence indicates that the PBPMs are easily accessible to most clients in their preferred official language.

Staff, clients, community partners and third-party agents all agreed that PBPMs have a positive impact on the lives of participants. Most staff also stated they are satisfied that the programs and services offered meet the needs of EI clients. Among the reported impacts observed were improved confidence and self-esteem, life skills, job search skills, employment, job satisfaction, earnings, employability and less reliance on income support.

Employers and Communities

Most employers felt that the programs are relevant to their needs. Respondents expressed satisfaction with the degree to which programs suited their organizational goals as well as with the employees who worked for them through the program. Some dissatisfaction, primarily with the Wage Subsidies program, was expressed with respect to the lack of background information on job candidates, the match of participants to the employers' businesses and the length of programs (i.e., period of funding).

Some provincial officials and community partners felt that the idea that training can only be done through Skills Loans and Grants/Enhanced Fee Payer is a limitation because it does not take into account the needs of industry. For example, when new companies are considering relocating in the province, some respondents believe they can no longer have an industry-based approach and develop programs at the local community college to meet the skills needs of such new companies. This was previously accomplished using Project-Based Training and Purchase of Training which were phased out on July 1, 1999.

The majority of training deliverers (colleges, universities, institutions, etc.) felt that the phase out of Project-Based Training and Purchase of Training negatively impacts on the relevance of PBPMs. Some respondents perceived that there is limited recourse to address the training needs of a particular sector, industry or employer, and that there is less incentive for training institutions to partner with local industry in addressing labour market issues.

Most employers and community partners felt that the programs are relevant to the needs of their communities. They indicated, however, a need for: more access to training for marginally-employed people, clients with special needs and those in rural and remote areas; longer periods of job placements/wage subsidies and longer-term follow-up with clients to increase the success rate of programs; training that is better targeted to the needs of the labour market; and, reduced paperwork associated with the PBPMs.

Regional Findings

There are some unique challenges in the northern region of Manitoba, including the difficulty of providing programs in small communities. There is also a lack of relevant training being offered in these communities. Furthermore, the individual approach to training through the Skills Loans and Grants/Enhanced Fee Payer was also viewed by some as a limitation because of the lack of available training programs.

Adequacy of Information and Monitoring Systems

Respondents indicated that the administrative systems for monitoring and reporting on PBPMs, program participants and third-party delivery agents are inadequate for proper planning, management and evaluation of the PBPMs. The validity and usefulness of the current accountability measures as well as the integrity of the data are perceived to be problems.

Weaknesses were also observed in the supporting information systems used to assess the accountability results targets. Comparisons between employment indicators from the survey and the return to work indicator in the administrative data indicated that a large number of actual returns to work were not captured in the information systems. This suggests that the accountability results for Manitoba may under represent actual success with respect to returns to work and unpaid EI benefits.

Labour Market Information/Labour Exchange

Labour Market Information (LMI) is readily available to staff, community partners, employers and clients via the Internet, although this medium is not accessible to those who are not computer literate. Available LMI is widely criticized, however, for being outdated and limited in relevance to client needs and to small regions and communities in Manitoba. In recognition of this problem a federal-provincial working committee on LMI has been formed and is planning to develop joint LMI research and products.

With respect to the Labour Exchange, the conclusions are similar to those for LMI, i.e., although job listings are easily accessible via the Internet and Job Bank, the information is often out of date and hence of limited use.

Service Delivery Model

With respect to the service delivery model, many respondents indicated that they feel there is a need for more flexibility in order to be able to address the needs of all the unemployed population of Manitoba. Although the PBPMs allow some degree of flexibility in program decision-making and responsiveness at the provincial, sub-provincial and local levels, some staff perceive that decision making (approval of expenditures) is more centralized than it had been prior to the LMDA and this can cause delays in the implementation of programs. The eligibility criteria stemming from the EI Act are also thought to limit flexibility in programming.

Third-party delivery agents expressed concern as well about the short-term contracts they are given for the delivery of employment services. They feel that these short contracts (of one year or less) limit their ability to do proper long-range planning and to develop and retain skilled employees.

Community partners stated a preference for service delivery models that rely on partnerships between the government and the community which included: sector-based models, where programs are designed and implemented in consultation with sector agencies made up of employers, branches of government, schools, and private training institutions; and third-party models, through which programs are designed and delivered through community and third-party organizations that possess greater expertise in delivering services to different client groups.

The evidence suggests that co-location has been beneficial for client service as well as for working relationships and information sharing between federal and provincial staff.

The major perceived strengths of the PBPM delivery structure include:

• single-window service for clients through the co-located Employment Centres;
• accessibility of the PBPMs in both official languages;
• the skill and commitment of front-line delivery staff at the Centres;
• the partnership approach to PBPM delivery that helps to ensure the relevance of programs; and,
• the cost-effective use of third-party delivery agents.

The major perceived weaknesses of delivery are:

• poor communications and a lack of program delivery guidelines for staff; inadequate promotion of the PBPMs to clients and community groups;
• some confusion about federal versus provincial roles and responsibilities for the reception function at co-located Centres and for programming for youth, Aboriginal peoples and persons with disabilities; and,
• a lack of training opportunities for clients in remote areas of Manitoba, for persons with disabilities and other special interest groups.

1.1 Background

Under the terms of the Canada/Manitoba Labour Market Development Agreement (LMDA), the federal and provincial governments agreed that Manitoba would assume responsibility for the design and delivery of active labour market measures for the unemployed in the province. A federal government commitment to devolve responsibility for labour market development to the provinces was made in response to the expressed desire by provincial governments to assume greater control over labour market development programs.

The federal government made formal offers to the provinces to assume these responsibilities in May 1996. On July 1, 1996, it enacted the Employment Insurance (EI) Act, which, as described in greater detail below, called for agreements with the provinces regarding the administration of active labour market measures for the unemployed and the monitoring and evaluation of these programs. The current document describes the methodology and findings from the formative evaluation of the Canada-Manitoba Labour Market Development Agreement.

As mentioned, a precursor to the signing of the LMDA was Bill C-12, the Employment Insurance (EI) Act. In addition to bringing about changes to the unemployment insurance regime, the objectives of the EI Act were (and are) as follows:

• to bring together several active labour-market provisions;
• to commit the federal government to work more closely with provinces in labour market activities;
• to authorize the federal government to enter into labour-market agreements with the provinces to provide employment benefits and support measures similar to those described in Part II of the Act and consistent with the purpose and guidelines of Part II (described below); and
• to encourage federal and provincial governments to work together in designing and developing a plan to implement and evaluate the benefits and measures.

The EI Act is in two parts. Part I provides for passive income support (EI benefits) to those temporarily out of work. Part II provides for active employment benefits to enable unemployed persons to return to work. Under the EI Act, the administration of Part I benefits is retained by the federal government, which is also responsible for delivering active labour market programs that are national in scope, such as those concerned with inter-provincial mobility or the operation of the labour exchange. Part II benefits and measures are expected to be delivered by the provincial governments through agreements with the federal government, provided that these benefits and measures are "similar" to those outlined in the EI Act and that they meet specified guidelines (see below).

Provinces that assume responsibility for these Part II benefits and measures will be provided appropriate funds from the EI account.

The EI Act outlines eight types of labour market programs for EI clients¹ (referred to as employment benefits and support measures²), which are to serve as the models for corresponding provincial programs.³ They are as follows:

• wage subsidies to encourage employers to hire EI clients;
• earnings supplements to encourage EI clients to accept jobs;
• financial assistance to encourage EI clients to become self-employed;
• financial assistance to provide work experience to EI clients;
• financial assistance to help EI clients acquire employment skills;
• assistance to organizations providing employment assistance to the unemployed;
• assistance to organizations dealing with adjustment; and
• assistance to undertake research and innovative projects in the area.⁴

An additional change that occurred under the EI Act involved ways in which training is funded. The EI Act specifies that three years following the date of its enactment training could no longer be funded through payments made directly to a public or private training facility.⁵ Rather, training is to be funded, in whole or in part:

• through payments flowing directly to the client in the form of grants or contributions, and / or loans or loan guarantees, for the payment of approved services; and / or,
• with vouchers given to the client to be exchanged for approved services.⁶

As a result of this change, Purchase of Training and Project-Based Training programs were phased out on June 30, 1999.

1.2 Canada/Manitoba Labour Market Development Agreement

The Canada/Manitoba LMDA was signed on April 17, 1997 and came into effect on November 27, 1997. Under the terms of the Agreement, Canada retained responsibility for the delivery of employment insurance (EI) benefits; for aspects of labour-market development that are national in scope such as interprovincial mobility, sector councils and national labour market information and exchange; and, for the financing of the provincial benefits and provincial measures (PBPMs) and the National Employment Service (NES) in Manitoba. Manitoba assumed the responsibility to design and manage the PBPMs.

Annex 1, Framework for Provincial Benefits and Provincial Measures, of the LMDA specifies the following;

"Manitoba will plan, design, deliver and manage provincial benefits and provincial measures which are similar to the employment benefits and support measures established by the Commission under Sections 59 and 60(4) of the Employment Insurance Act and are consistent with the purpose and guidelines of Part II of the Act." (Section 1.2) "Manitoba's objectives...are as follows:

• To provide a "seamless", fully integrated and cost effective continuum of employment programs and services for unemployed Manitobans, and in doing so, improve client service and reduce overlap and duplication.
• To provide access and linkages to employment programs and services which are appropriate to the needs of Manitobans and relevant and responsive to the needs of employers, communities and the changing economy.
• To foster partnerships with communities and employers and encourage local level participation in planning and delivery of labour market development programming and services which are consistent with provincial economic priorities and opportunities.
• To foster self-reliance and personal commitment by individuals to achieve self-sufficiency through sustainable employment." (Section 2.2)

"The objective of provincial benefits and provincial measures is to assist individuals to prepare for, obtain and maintain employment and to reduce their dependency on government forms of income support, including EI benefits and income assistance. In support of this objective and the objective of eliminating overlap and duplication, Manitoba intends to modify its existing employment programs and services to make them more responsive to client and community needs and to develop new approaches that are responsive to client and community needs.

In modifying existing employment programs and services and developing new ones, Manitoba intends to incorporate the following design features:

• increased flexibility to allow planning and delivery decisions to be made at the local level;
• strengthening of cooperative and partnership arrangements with other service providers and delivery agents;
• requirement of persons receiving assistance to develop and commit to personal action plans and also to share the costs of assistance as appropriate;
• the use of a case management approach to support clients, coordinate assistance and enable appropriate follow up to be done.

Manitoba will assume responsibility for selecting priority clients for provincial benefits and provincial measures but only [EI clients] will be given access to provincial benefits." (Section 3.1)

Another important aspect of the LMDA is accountability, described in Annex 4 to the Agreement. A results-based accountability framework was built into the LMDA to enable the setting of targets in accordance with the above objectives and expected impacts and to track and report results. Primary Result Indicators, as described in Annex 4 are:

1. active EI claimants as a percentage of EI clients who access PBPMs;
2. returns to employment of EI clients, with an emphasis on active EI claimants; and,
3. savings to the EI account (currently operationalized as unpaid benefits).

Based on data extracted from the management information system of the Human Resource Investment Branch (HRDC) this formative evaluation provides preliminary evidence on these three measures. In-depth assessment of the measures requires data gathered over a longer period of time and the application of higher level analysis than was possible during the formative evaluation.

The PBPMs are part of a range of programs and services administered through the Employment and Training Services Branch (ETS) of the Training and Continuing Education Division of Manitoba Education and Training (MET). The other programs and services administered by ETS include Management Services, Labour Market Support Services, Youth Programs, Workforce 2000 and Apprenticeship. Within ETS, the funding for different programs is determined and distributed under three distinct funding appropriations that are related to the client groups served. The three funding appropriations are for:

• ETS Programs which serve Provincial and Municipal Income Assistance (IA) eligible clients, private, public and community sector employers, communities, municipalities, and organizations requiring the development of employment and training projects and partnerships;
• Labour Market Development Agreement (LMDA) Programs which are designed to address the needs of EI clients, all unemployed Manitobans, employers and community partners⁷; and
• Adult Literacy and Continuing Education Programs (ALCE) which are designed to meet the needs of employed Manitobans, IA clients, EI clients and self-directed adult-learners, as well as literacy program delivery agents and referral agencies.

In the Canada/Manitoba LMDA, Annex 1, four benefits (Wage Subsidies, Employment Partnerships, Self-Employment Assistance, Loans and Grants/Enhanced Fee Payer) and three measures (Employment Assistance Services, Labour Market Partnerships, and Research and Innovation) are specified. Two of the measures, Labour Market Partnerships and Research and Innovation, do not have clients associated with them and therefore were not part of the client survey. A brief description of each follows:

• Wage Subsidies. The objective of Wage Subsidies is to help workers, who are at risk of long-term unemployment, lack experience or face other barriers to employment, to find a job and gain work experience. Under this benefit a wage subsidy is provided to eligible employers to encourage them to offer permanent jobs to such individuals. The wage subsidies are paid to the employer on a "claim back" basis and are based on actual wage costs for the eligible worker. Eligible participants include EI clients and Manitobans who are eligible for or receiving Income Assistance (IA).⁸
• Employment Partnerships. The objective of Employment Partnerships is to create meaningful work experience opportunities for eligible individuals or to provide short-term work experience to enable the acquisition of skills needed by the local community. This benefit is available to EI clients and Manitobans who are eligible for or receiving IA. For EI clients, programming is delivered through third-party service providers who are under contract with ETS. Participants may receive pre-employment preparation, skills enhancement, work experience, a job placement and/or job maintenance or support services.
• Self-Employment Assistance. This employment benefit aims to create self-employment for EI clients by assisting those with sound business plans to start their business. Under this benefit, Manitoba will contract with community and other service delivery partners to select clients and provide customized self-employment services, which may include: evaluating the individual's business idea, assessing whether the individual is a suitable candidate; job coaching in business plan development and implementation; the provision of advice and support; and referrals to other supports as needed. Participants must contribute money, work or equipment toward the business.
• Skills Loans and Grants/Enhanced Fee Payer. The objective of Skills Loans and Grants is to encourage individuals to acquire skills through education and training for purposes of ending dependence on EI benefits and entering employment. Programming is delivered by ETS staff in Employment Centres. Staff will determine whether financial support is advisable and negotiate the level and nature of support with clients according to their individual need. Each participant is expected to complete and present information to develop an achievable Return-to-Work plan. All participants are expected to contribute to the cost of their training. Individuals eligible for this benefit include EI clients and apprentices who qualify for EI.
• Employment Assistance Service (EAS). Under this measure, third-party partners such as community-based organizations are contracted by ETS to help unemployed and job-threatened individuals to prepare for, find, obtain or maintain employment that meets the needs of the community. Examples of the types of National Employment Service activities now covered under EAS include: the provision of labour market information, individualized counselling, job-search groups, referral services, general awareness/education activities, marketing of clients, encouraging volunteer work, and recommending various employment benefits. This measure tends to be short-term in nature.
• Labour Market Partnerships. This measure was designed to encourage communities and employers to address economic development and long-term employment needs by building the capacities and strengths of local residents. Under this measure, Manitoba will form partnerships with employer and employee groups, associations and other delivery agents to facilitate activities to assist the unemployed to return to work or those threatened with job loss to find other employment. Program activities include: labour force adjustment activities to facilitate the employment of job-threatened workers; workforce development planning to support local economic development; research and marketing of new programs and services; coordination and analysis of local employment opportunities; and tracking and assessment of labour market and community interventions.
• Research and Innovation. Under this measure, Manitoba provides financial assistance to partners to support research, planning and innovative activities that contribute to the economic future of the province and ensure the active participation of provincial residents seeking to work. Projects are funded through contracts with ETS.

In addition to the PBPMs described above, this evaluation assessed two other programs that were delivered in the province. These programs, which were phased out as of June 30, 1999, were:

• the Purchase of Training benefit which authorized the purchase of training on behalf of a select client group; and
• the Project-Based Training benefit, which provided group training contracted through a community coordinator, trainer or not-for-profit group.

1.3 Evaluation Objectives and Issues

According to the Evaluation Framework⁹, the overall objectives of the LMDA evaluation in Manitoba were:

1. To measure:

1. the extent to which the new arrangements under LMDA and the specific provincial benefits and measures are successful in achieving their objective;
2. whether Manitoba's objectives, as described in Section 4.2.2 of the Evaluation Framework, have been achieved; and
3. the validity and reliability of the primary indicators, as described in Annex 4 of the LMDA.

2. To provide useful and relevant information to managers, policy makers and program designers on a number of issues including implementation, design, delivery, client flows and experiences and data needs of the program so that optimum use of resources is possible within the Manitoba labour market and local labour markets within the three demographic areas: Winnipeg, Southern Manitoba and Northern Manitoba. In particular, the evaluation is to provide timely information on these impacts in the short, medium and longer terms.

3. To estimate the cost effectiveness (and/or cost benefit analysis) of the provincial benefits and measures under the LMDA.

4. To demonstrate "best practices" and what lessons can be learned.

Given that this is the formative stage of the evaluation process, the objectives of this evaluation were focused on the shorter-term outcomes of the programs/services as well as the implementation of the LMDA. In broad terms, the purpose of the formative evaluation is to supply information indicating what improvements, if any, are required to the LMDA design, delivery and infrastructure that would permit it to better meet its objectives. The formative evaluation also provides a foundation for the future summative evaluation¹⁰ by collecting baseline information (e.g., early outcomes, profile information) on participants. In that light, an important objective of the formative evaluation was to determine the extent to which current management information systems are sufficient for conducting the summative evaluation.

The LMDA Evaluation Framework identified four evaluation issues (relevance, design and delivery, success and cost-effectiveness) and 21 core evaluation questions. The formative evaluation addresses three of the issues; cost-effectiveness is a summative evaluation issue and is not addressed in this evaluation. As such, 19 of the 21 evaluation questions are addressed. The 19 evaluation questions are presented in Appendix A¹¹.

1.4 Manitoba Labour Market

A profile of the Manitoba labour market serves two purposes. The first is to establish a baseline picture of the provincial labour market just before and during the implementation of the Canada/Manitoba LMDA (1997 and 1998). Subsequent evaluations of the LMDA can include a comparison of the labour market to what it was at the start of the process. The second purpose is to give a snapshot of the current provincial labour market (1999).

Using selected labour market indicators, Table 1.1 presents an overall profile of Manitoba's labour force for 1997 to 1999. Comparative 1999 data for Canada is also presented. While the focus is on 1999, the other years are included for contextual reasons.

Industry Mix

Manitoba has a diverse economy. Table 1.2 presents figures on employment by industry in Manitoba — 1999, change in employment numbers in Manitoba, by industry, between 1990 and 1999, and percentage distribution of employment by industry, Manitoba and Canada — 1999.

Table 1.2 shows that the service sector represents close to three-quarters (73.4 per cent) of the employed labour force in Manitoba. Within the service sector, the largest components are health care and social assistance (63.7 thousand jobs) and retail trade (62.1 thousand jobs).

During the 1990s, the service sector experienced greater gains in employment than the goods sector (17.5 thousand versus 10.1 thousand). In the service sector the largest employment gains have been in management/administrative (5.4 thousand jobs), health care and social assistance (5.0 thousand jobs) and professional, scientific and technical services (4.8 thousand jobs).

Manufacturing not only has the greatest concentration of employment within the goods-producing sector, it employs the greatest number of individuals in all of the Manitoba economy. Agriculture is the second largest industry within the goods-producing sector (37.3 thousand). However, agriculture has experienced a decline in employment since the beginning of the decade (-2.7 thousand jobs). Construction also generates much employment (29.3 thousand), though not as much as most service sector components, many of which generated over 30,000 jobs in 1999.

A comparison of the percentage distribution of employment by industry for Manitoba and Canada (columns three and four of Table 1.2) shows that the proportion of employment attributable to agriculture is noticeably higher in Manitoba than the country at large. Other noteworthy observations are that the manufacturing and professional & technical services industries account for greater proportions of employment in Canada than in Manitoba.

Aboriginal Population

Aboriginal peoples are an important component of Manitoba's labour supply. Much of what is presented in this section is based on the 1996 Census of Canada data¹² and the 1998 unpublished document "Manitoba's Aboriginal Population: A Statistical Profile and Compendium of Aboriginal Labour Market Information" obtained from Manitoba Education and Training.

Census data indicate that in 1996, there were 128,685 Aboriginal persons in Manitoba. This represented 11.7 per cent of the total population of the province, in comparison Aboriginal persons represent 2.8 per cent of the total Canadian population. The 1996 Census data showed that Manitoba had the highest proportion of Aboriginal peoples to total population of all the provinces. Only neighbouring Saskatchewan had a similar proportion (11.4 per cent). Winnipeg, Manitoba's capital city, had one of the highest proportion of Aboriginal peoples to total population (6.9 per cent) among Census Metropolitan Areas (CMAs) in the country. Regina and Saskatoon were the only CMAs to have greater concentrations (over 7 per cent).

As youth represent the future labour supply¹³, it is important to observe their numbers within the Aboriginal population. In 1996, Aboriginal youth under 25 years of age represented 56 per cent of all Aboriginal persons in Manitoba (approximately 72,000). This is considerably higher than this age group's share in the non-Aboriginal population of the province (33 per cent).

Census 1996 data indicate a difference in educational attainment between Aboriginal and non-Aboriginal Manitobans. For example, over half (51 per cent) of Aboriginal peoples aged 25-29 had less than a high-school certificate, compared to 20 per cent of non-Aboriginal Manitobans in this age group.

Analysis of the 1996 Census data also indicates lower labour force attachment among Aboriginal peoples. Just over one half (53.9 per cent) of Aboriginal Manitobans participated in the labour force compared to two thirds (67.6 per cent) of non-Aboriginal Manitobans. In 1996 the unemployment rate for Aboriginal Manitobans was almost four times that of non-Aboriginal persons (25.5 versus 6.4 per cent).

Regions

A major issue for Manitoba is regional diversity. Labour Force Survey (LFS) data obtained by Manitoba Region of HRDC¹⁴, indicate that the unemployment rate in Northern Manitoba (22.1 per cent) is approximately 3.5 times the rate in Winnipeg and Southern Manitoba (5.7 and 5.5 per cent, respectively).¹⁵

1.5 Profile of LMDA Participants by Region

Demographic characteristics of LMDA participants by region, based on HRDC administrative data (for the period of November 27, 1997 to August 15, 1999), are presented in Table 1.3. Participants are compared to the population of all those eligible to participate in the PBPMs and significant differences are noted. The following patterns are noteworthy:

• The distribution of male and female participants was similar in the north and south of Manitoba, where there were slightly more men than women participating in PBPMs. In Winnipeg, however, there was a higher proportion of female participants (56.4 per cent) than of male participants (43.6 per cent).
• With respect to participants' age, the southern region is unique in that there was a relatively high proportion of young participants under the age of 30 (53.9 per cent) and a low proportion of participants 45 or older (13.1 per cent). Compared to the eligible population, LMDA participants tended to be much younger in every region of the province.
• The distribution by participants' first language was quite similar across regions, though a somewhat higher proportion of Winnipeg participants spoke a language other than English (6.9 per cent).
• In the northern region, there were comparatively more Aboriginal participants (30.9 per cent) — the fewest Aboriginal participants were located in Winnipeg (7.3 per cent).
• In the southern region there were relatively high proportions of persons with disabilities (12.1 per cent) and visible minorities (8.4 per cent).
• The distribution of participants on Income Assistance (in 1997) was very similar across the three regions.
• Participants in Winnipeg were more highly educated than those in either the north or south. For example, in Winnipeg there was the highest proportion of university graduates (15.3 per cent) and the lowest proportion of participants educated at only Grades 7 to 11 (24.8 per cent).

Exhibit 1.1 presents the rates with which participants in different PBPMs have made use of multiple interventions. Training participants tend to use multiple interventions more often than other participants. The distribution for each PBPM includes all participants who have ever participated in the program, regardless of whether a PBPM was their first, most recent or only intervention. Participants who have used multiple interventions are counted once for each intervention in which they have participated.

1.6 Participation of Target Groups

The degree to which EI clients participating in the PBPMs are representative of the EI client population in Manitoba can be assessed by comparing the socio-demographic characteristics of participants in the survey with those of respondents in the comparison group. Although the participants and comparison group are similar in most respects, there are some minor differences.

As shown in Exhibit 1.2 participants and comparison group respondents are very similar with respect to their educational backgrounds and mother tongue, although participants are slightly more educated and more likely to be anglophone.

Compared to respondents in the comparison group, a relatively small proportion of participants are currently married and more are single (i.e., never been married). The distribution of respondents according to equity group status — Aboriginal peoples, persons with disabilities and visible minorities — is very similar for participants and the comparison group (Exhibit 1.3).

On average, participants have a smaller household size and fewer dependants than comparison group respondents. Participants also tended to have a lower household income in the year prior to the survey (Exhibit 1.4).

In conclusion, the survey evidence indicates that participants in the PBPMs are representative of EI client population in Manitoba (e.g., in terms of equity group status and other socio-demographic characteristics). PBPMs participants are somewhat more likely to be single, educated at the post-secondary level and anglophone, and to have a smaller household size and lower household income, though the differences are small.

This section of the report presents the methodology for the Formative Evaluation of the Provincial Benefits and Provincial Measures under the Canada/Manitoba LMDA. For a more detailed description of the methodology, see the Technical Report under separate cover.

The final design of the evaluation consisted of 28 key informant interviews, 12 focus groups, a survey of 1,400 program participants (1,100 PBPM participants and 150 participants in each of the PBT and POT programs)¹⁶ and a survey of 500 non-participants.

The following is a description of the methodology used for each component of the data collection process.

2.1 Key Informant Interviews

A series of 28 key informant interviews was conducted for the formative evaluation. The list of potential key informants was prepared by the Joint Evaluation and Accountability Committee (JEAC). The interview respondents included a variety of program managers and policy makers who collectively were in a position to comment on the design, implementation and delivery of the PBPMs that were assessed in this evaluation.

The key informants included the following:

• four (4) federal officials with Human Resources Development Canada (HRDC), including a representative of the Canada Employment and Immigration Union (CEIU);
• six (6) provincial officials with Manitoba Education and Training, including one official from the Apprenticeship Branch;
• eleven (11) key community partners; and
• seven (7) third-party delivery agents.

2.2 Focus Groups

A total of 12 focus group discussions were conducted. Focus groups had an average of eight to 10 participants each and were conducted in each of the three service regions of the province: Winnipeg; Southern Manitoba (Brandon); and Northern Manitoba (Thompson). There were five types of participants: clients; staff; third-party delivery agents; employers; and community partners (including employee groups, employer groups, sectoral associations and others). The distribution of the 12 focus groups is presented in Table 2.1.

For recruiting clients, program participants were selected from the sample for the survey of participants (representing a variety of PBPMs). For employers, community partners and delivery agents, the focus group recruitment relied on lists provided by the JEAC.

2.3 Survey of Participants and Non-Participants

Survey of Participants: Population

The final participant data set included only those participants who started a PBPM intervention on or after November 27, 1997 and completed/terminated before August 15, 1999. The length of time since completing or terminating the intervention was a key variable for stratifying the sample in the final analyses as the length of time since completion is related to employability outcomes.

Survey of Participants: Data Sampling

The data files were originally developed to include participants who participated in LMDA employment programs and services at any time between November 27, 1997 and August 15, 1999. As mentioned, these files were aggregated, yielding a single data file containing information for 19,252 participant cases, with the individual client as the unit of analysis. This was not equal to the sum of all the cases from the administration and data files because clients that had taken part in more than one intervention could appear in more than one file. Following the removal of all cases without valid phone numbers, start and end dates for EI benefits, and start and end dates for most recent interventions, the final data file consisted of 14,784 individuals.

Three primary sampling variables were taken into account when developing the survey sample frame. These variables were program type (EAS¹⁷, Wage Subsidies, Self-Employment, Employment Partnerships, SLG/Enhanced Fee Payer, Purchase of Training, Apprenticeship and Project-Based Training), region (Winnipeg, Southern Manitoba and Northern Manitoba), and claimant status (active EI claimant, near reachback, mid reachback and far reachback).¹⁸ Table 2.2 presents the sampling plan for the participant survey (in the column labelled "quota"), as well as the number of interviews that were actually completed (in bold) with participants who differed according to these three variables. The targets for individual cells were adjusted based on the distribution of participants by geographical area, program and claim status, thus attempting to ensure that a minimal sample size (e.g., 40 cases) was available to permit segmented analysis based on any two of the variables simultaneously.

Given the size of the final sample (i.e., intended to be 1,400 interviews), segmenting the sample on the basis of all three variables simultaneously for the purpose of analysis would require that the sample be divided into 96 cells, reflecting the product of the number of categories in each of the sampling variables (i.e., eight programs multiplied by three regions and four claim status categories). For some cells, however, the available sample was not sufficient to support a quota of 40 interviews for the analysis of the data by two of the sampling variables (i.e., Employment Partnerships in Northern Manitoba). Thus for some programs, additional interviews were completed in other regions that had sufficient sample (i.e., Winnipeg) and, if no additional sample was available within the same region, attempts were made to complete additional interviews within the same program and claimant status category. A total of 1,393 interviews were completed. The distribution of sample drawn to obtain the desired number of survey completions is presented with response rate information in Table 2.2.

Survey of Participants: Response Rate

The response rates and refusal rates for participants in each type of program are presented in Table 2.3. The response rate is the proportion of cases from the functional sample who responded to the survey, while the refusal rate represents the proportion of cases from the functional sample¹⁹ who declined to participate in the survey.

Table 2.3 Response Rate for the Participant Survey

	EAS	WS	SE	EP	SLG/ Enhanced Fee Payer	App	POT	PBT	Total
Initial sample	3,565	581	864	487	1,677	728	381	1,158	9,441
(less) Unused sample	2,513	115	245	37	675	107	0	569	4,261
(less) Attrition
Number not in service	288	118	113	114	226	120	148	148	1,275
No knowledge of topic/ineligible	90	42	78	13	32	6	33	12	306
Language barrier (not English/French)	13	2	8	3	3	3	8	3	43
Functional sample	661	304	420	320	741	492	192	426	3,556
Other number retired (not due to attrition)
No answer/busy (<10 calls before end of fieldwork)	253	44	132	91	353	226	9	173	1,281
Unavailable for duration of survey	4	6	5	5	7	5	4	4	40
Retired/called 10+ times	20	15	19	7	17	14	28	14	134
Quota filled (Number called but no longer needed)	17	1	18	7	24	6	0	19	92
Other/illness	31	17	11	15	26	15	12	17	144
Total numbers retired	325	83	185	125	427	266	53	227	1,691
Non-response
Refusal	89	60	71	38	77	44	31	38	448
Incomplete refusal1	11	3	3	1	2	1	2	1	24
Total non-response	100	63	74	39	79	45	33	39	472
Total completed	236	158	161	156	235	181	106	160	1,393
Refusal rate	15.1%	20.7%	17.6%	12.2%	10.7%	9.1%	17.2%	9.2%	13.3%
Response rate	35.7%	52%	38.3%	48.8%	31.7%	36.8%	55.2%	37.6%	39.2%
Margin of error	±6.3%	±6.7%	±7.0%	±6.5%	±6.2%	±7.0%	±8.6%	±7.2%	±2.5%
1 Incomplete refusals refer to interviews during which the respondent refuses to continue part way through.

The response rate for the survey was lower than expected, ranging from 31.7 per cent among SLG/Enhanced Fee Payer participants to 55.2 per cent for Purchase of Training respondents, with an overall response rate of 39.2 per cent. The overall refusal rate was also higher than expected (13.3 per cent) and ranged from 9.1 per cent for Apprentices to 20.7 per cent for Wage Subsidies participants. The sampling error for the survey is ±2.5 per cent. That is, the overall survey results are accurate within ±2.5 percentage points, 19 times out of 20. The margin of error is higher for the sub-group analyses.

Survey of Participants: Weighting of Data

The grouping of weighting variables was dependent on the type of analysis. For the overall analyses, the participant data were weighted according to program type, region, claimant status and age. For analyses by program type, the data were weighted by region, claimant status and age. For analyses by claimant status (i.e., active EI claimant versus reachback clients), the data were weighted by program type, region and age.

Comparison Group Survey: Sample Frame Creation

The comparison group case file was drawn from a file of EI claims that were active at some time between 1994 and 1998. The administrative files included T1, Status Vector, ROE, NESS, and BNOP files. This dataset of comparison group members was then linked to the administrative data files, yielding a single file of 204,531 active EI claimants and reachbacks from which to draw the comparison group sample.

The comparison group active EI claimant and reachback samples were developed separately. To sample comparison group active EI claimants, the comparison group data file was matched to the participant data file based on the time periods for which members of the comparison group were receiving EI. To accomplish this, three time periods were defined according to observed values for program end dates in the population of active EI claimant participants.

The comparison group active EI claimant sample was drawn in the same proportions as were observed for active EI claimants in each of the three time period cohorts in the participant population.

To develop a comparison group sample for reachbacks, the proportion of different reachbacks (i.e., near, mid and far reachbacks) in each of the three time cohorts was determined for the participant population. Comparison group reachbacks (i.e., they were not on claim during any of the three theoretical reference dates) were then randomly assigned to one of the three time cohorts. Reachback status (i.e., near, mid, far) was determined based on the length of time since the end-date of their most recent EI claim period and the theoretical reference date (i.e., mid-point of the time cohort). Near, mid and far reachback comparison group cases were then sampled such that they reflected the proportion of these cases in the three time period cohorts that were observed for the participant population.

To correct for the fact that a comparison group case could possibly fall into any of the three time cohorts, each time period cohort was sampled separately. The final comparison group sample frame consisted of 2,633 cases in three mutually exclusive time period cohorts from a population of 204,531.

Comparison Group Survey: Response Rate

The response rate for the survey is presented in Table 2.4. The response rate is the proportion of cases from the functional sample who responded to the survey. Conversely, the refusal rate represents the proportion of cases from the functional sample who declined to participate in the survey.

The response rate for the survey was 34 per cent²⁰ and the refusal rate was 26.4 per cent. The sampling error is ±4.4 per cent. That is, the overall survey results are accurate within ±4.4 percentage points, 19 times out of 20. Comparison group survey results presented by claimant status (i.e., active EI claimants versus reachback) were weighted by region and gender.

Evaluation findings pertaining to the consistency of the Provincial Benefits and Provincial Measures (PBPMs) with the EI legislation, the LMDA and government priorities as well as the relevance of LMDA programs and services are presented in this chapter. The findings draw on qualitative evidence gathered from the key informant interviews and the focus groups.

3.1 Compatibility of PBPMs with EI Legislation, LMDA and Government Priorities

Federal and provincial officials agreed that the PBPMs reflect the guidelines, principles and intent of the EI Act and the LMDA. In addition, the officials agreed that given the PBPMs are virtually identical to the former HRDC programs, they continue to be relevant and consistent with HRDC's priorities. Some provincial key informants also indicated that the PBPMs reflect HRDC's priorities in terms of the returns to work and the savings to the EI account.

Provincial officials believe that the PBPMs do not respond to all of Manitoba's priorities given that the province is facing skills shortages and needs to provide training to a broader range of individuals than simply EI clients, including in particular, the marginally-employed (working in low skill jobs without other skills)²¹. Provincial officials indicated they also have the responsibility for providing training and employment support to people on income assistance and they are now trying to integrate the delivery of services and programs targeted to this group with the delivery of some PBPMs. The process is not completed but this integration has been made possible by the LMDA.

Federal officials observed that the province is putting a stronger emphasis than HRDC did on serving individual clients as opposed to employer or project-based interventions, particularly through Skills Loans and Grants/Enhanced Fee Payer, which is consistent with the LMDA.

The content of the PBPMs was determined within the context of those parameters set out by the LMDA and the EI Act. In consultation with other senior managers in the federal government, provincial officials divided the dollars between the various PBPMs based on historical data and current trends in the labour market.

3.2 Relevance to the Needs of Individuals, Employers and Communities

Most key informants and focus group participants agreed that the PBPMs are relevant to the needs of individuals, employers and communities, though there are some limitations and service gaps. There was, however, some difference in opinion among Employment Centre staff. Some staff stated they can meet clients' needs better now than before the LMDA because they are aware of additional resources and provincial programs, whereas other Employment Centre staff disagreed and indicated they were better able to serve the needs of clients before the implementation of the LMDA because less local decision-making is possible now.

Community partners felt the Province is committed to increasing the relevance of programming as they noted that the Province's organizational structure to deliver PBPMs has improved since the LMDA was first implemented. The Province's willingness to evaluate service delivery (e.g., evaluating the accessibility of Employment Centres) also increases the confidence of LMDA partners. Finally, the approval process for PBPMs has become faster due to increased specialization of project officers. Evidence of the relevance of the PBPMs include: the availability of funding for regionally and provincially relevant projects (e.g., long-distance learning, health care training, accessibility assessments and sensitivity training for staff); improved focus on basic education; more partnering, including more involvement from employers; and the use of the individual training approach for certain areas where there is a high demand for training (e.g., computers and IT).

Key informants and focus group participants commented on some facets of the administration and delivery of the PBPMs that reduce the relevance of the programs. Some Employment Centre staff perceive that decision making (approval of expenditures) is more centralized than it had been prior to the LMDA and that this can cause delays in the implementation of programs. Many provincial officials, however, believe that there is probably more flexibility within the LMDA than is currently being utilized because LMDA partners are fairly new to the process.

Concerns were expressed regarding the eligibility criteria limiting accessibility to LMDA funded programming to EI clients, which the vast majority of respondents see as creating limitations in terms of serving the needs of the Manitobans. Individuals noted that the limited access to LMDA funding affects a significant cross-section of Manitobans, including:

• the marginally-employed;
• persons with disabilities who have been unemployed for a long time;
• mothers who have been unemployed for a long period (while raising children);
• Aboriginal peoples;
• training for youth "at risk";
• immigrants entering the province;
• income assistance recipients; and,
• people in rural areas where greater reliance on part-time and seasonal employment means lower levels of EI eligibility.

The issue of eligibility for the PBPMs is complex and confusing to many respondents (e.g., employers, staff, community partners, third-party delivery agents)²². Some respondents perceived there are important needs for employment and training programming that are specifically not addressed through the LMDA PBPMs. Other respondents did not feel that these needs were being addressed at all, whether through the LMDA or any other existing provincial or federal initiatives. As well, some respondents do not seem to be able to clearly identify and distinguish between LMDA dollars and provincial dollars and hence, which activities (i.e., training and employment programs) are being funded by the different sources of money. In fact, some programs offered through ETS can address the needs of both EI clients and individuals receiving/eligible for income assistance, so this confusion may not be unexpected. For example, Employment Partnerships aim to facilitate the employment of "eligible" unemployed participants. If the participant is an EI client, programming is delivered through third-party service providers who are under contract with ETS using LMDA funding. If the participant is receiving income assistance, programming is funded with provincial dollars. Another example is the Wage Subsidies program, an employment benefit under the LMDA, which is being integrated with the provincial wage subsidies programming.

Provincial officials indicated that Manitoba is in the process of developing a training strategy to address provincial skill shortages. These officials feel constrained in that they can not access LMDA funding to re-skill an existing worker who is in an entry-level position who is not an EI client.

Another issue with respect to the development of a provincial training strategy to address provincial skill shortages is how the Province will address skills shortages in the immediate future if it is not able to support longer-term community college programs or university programs that actually respond to skill shortage areas. As noted previously, Section 1.1, Background, the EI Act specifies that three years following the date of its enactment training could no longer be funded through payments made directly to a public or private training facility. Rather, training is to be funded, in whole or in part:

• through payments flowing directly to the client in the form of grants or contributions, and / or loans or loan guarantees, for the payment of approved services; and / or,
• with vouchers given to the client to be exchanged for approved services.

As a result of this change, Purchase of Training and Project-Based Training programs were phased out on June 30, 1999. In light of this policy change many third-party delivery agents stated that unless the longer-term needs for higher skill training are addressed, as well as the broader economical and social issues, the governments will not achieve the expected savings to the EI account. For example, they feel that quick skill programs to get people into low-level entry positions which they will likely no longer occupy six months later are not a solution. They explained that clients who lack solid and useful labour market skills will not be able to relocate into better employment and will thus be back applying again for EI support.

Some respondents (third-party delivery agents and provincial officials) noted that many persons with disabilities do not qualify as EI clients and as such their employment training is not covered under LMDA. While this client group has access to EAS, key informants do not feel that current EAS are sufficient to meet the long-term needs of these clients.

Another group of individuals that normally falls outside the "EI client" requirement is recent immigrants.

Relevance to Clients

Most survey participants agreed that the PBPMs met their needs and expectations and impacted positively on their lives. They were asked how interested they were in pursuing each of three different labour market activities (i.e., education or training program, starting a business and working on a full-time or part-time basis) in the week prior to starting their program. Overall, the results suggest that PBPM participants were well matched to their interventions. Specifically, Self-Employment participants were most likely to rate themselves as very interested in starting their own business; participants involved in education or training programs were most likely to rate themselves as very interested in pursuing education or training; and Employment Partnerships or Wage Subsidies participants were most likely to have been very interested in entering the workforce on a full-time or part-time basis. This evidence demonstrates a good match between participants' interests and the interventions they received. However, it should be noted that this question was asked of participants after they had completed their programs and may be subject to retrospective bias.

Overall, clients participating in the focus groups stated that the PBPMs they had access to provided hope, helped them gain self-confidence, gave them a sense of direction and increased their ability to get a job. In certain cases, clients stated that the services received even exceeded their initial expectations. However, many clients complained about not being adequately informed of available programs and services, or of available jobs and related details (e.g., employment prospects, salary range) by Employment Centre staff.

Most staff agreed that the PBPMs were doing a good job of meeting the needs of some clients but also identified programming gaps, administrative problems and insufficient communication as possible threats to the level of relevance to clients' needs. The PBPM guidelines are found by many staff to be too restrictive to respond to the needs of multi-barriered clients.

Another major issue is the lack of relevant training available in rural and northern communities. Key informants and focus group participants indicated that the availability of relevant training has been hindered by the replacement of the former Purchase of Training approach with the new Skills Loans and Grants/Enhanced Fee Payer program. There is a perceived need for more funding for distance learning approaches for clients in rural and remote areas (e.g., correspondence courses, courses over the Internet/e-mail). In addition, some key informants indicated that the LMDA will not provide funding for clients to attend an out-of-province training program, even if it is thought to make sense and is the closest available location (e.g., those living close to the Saskatchewan border). Key informants stated that they believe relevance could be improved with more local, decentralized decision-making authority regarding the delivery of PBPMs.

Community partners identified the following regarding the relevance of PBPMs to clients:

• The lack of financial resources has contributed to a focus on shorter-term programs when longer-term interventions are often thought to be more appropriate (e.g., for multi-barriered clientele).
• Skills Loans and Grants is regarded as relevant for assisting students with their tuition and educational expenses but there was a lack of funding for program materials and supports, such as textbooks or day care.
• A student loan or LMDA money is now considered income, which works against students who would like access to subsidized housing.
• The amount of the government contribution has decreased over the last few years and student needs are not being met to the same degree as they used to be. This lack of funds has compromised program effectiveness and relevance because clients are unable to access programming.

Relevance to Employers and Communities

Most employers felt that the PBPMs are relevant to their needs, those of individuals and of communities. Some employers indicated that the PBPMs could be more responsive to local needs and should better address the cultural and economic peculiarities of specific regions of Manitoba. They suggested that the lack of opportunities, the isolation and the cost of living in northern communities should be taken into consideration when developing programs. Several employers who took part in the focus group held in Winnipeg also indicated that the devolution of responsibility for labour market programs to the Province has increased the extent to which the needs of clients, communities and employers are being met.

Employers who participated in the Winnipeg focus group were involved in Apprenticeship (a subset of SLG), Wage Subsidies or Employment Partnerships. Results from the focus group indicate the following:

• Employer expectations for impact on their organizations tended to involve improvements to their human resources, either through training for employees or a long-term job placement of a suitable candidate.
• Although the programs currently delivered were deemed to be useful to employers (e.g., Wage Subsidies), the lack of programming to train and upgrade current employees represented a programming gap (e.g., the employees need to be laid off before they can access training under LMDA PBPMs²³).
• Employers are generally pleased with the degree to which programs meet their organization's goals and with the employees working for them through the Wage Subsidies program. There was some dissatisfaction with regard to the lack of information they were given on a candidate's skills as well as on the length of programs (which they generally perceive to be too short).
• Some employers stated that there is sometimes a failure to match the interests and skills of program participants and employers, and that there is a lack of follow-up to ensure the success of a program. In particular, employers in specialized fields for which no training is available in the province of Manitoba stated they take a financial risk in investing upwards of two years providing hands-on training to employees and therefore cannot afford a high turnover rate. Thus, the quality of the candidates they accept under the program becomes of primary importance because they will need to continue training these employees beyond the end of the wage subsidy and must be able to rely on these employees to stay with the company following the end of their training.
• Feedback from employers involved in Wage Subsidies programs indicates that these programs enabled the employers to provide a "decent" wage (i.e., above minimum wage) for needed employees during the subsidy period.
• Employment Partnerships are regarded by Employment Centre staff as relevant to employers' needs so long as a comprehensive assessment of the needs of the individual and the employer is completed.
• One particular issue raised during the evaluation relates to potential problems of specialized training for an industry such as aerospace. Specifically, the demand for workers and/or skills can change quite abruptly in this field, so sometimes highly skilled workers who have just completed their training (initiated when demand was high) can find themselves with no work if the demand decreases.
• It was noted that the approval process for PBPMs has become faster due to increased specialization for project officers (one deals with Wage Subsidies, one with training vouchers, etc.), thus there has been greater efficiency and more willingness by employers to participate.
• Finally, it should also be noted that all employers in Winnipeg felt they would like to be better informed about the available PBPMs.

Employment Partnerships are regarded by staff as relevant to employers' needs so long as "all the homework" is done about these needs. Specifically, a needs assessment has to be completed to ensure an appropriate match between employers and employees.

The evaluation found that the issue of eliminating Project-Based Training (PBT) and Purchase of Training (POT), was viewed as hindering the relevance to employers and communities. Specifically, the idea that training can only be done under the LMDA through the Skills Loans and Grants or Enhanced Fee Payer (EFP) program was viewed by many provincial officials and community partners as a limitation because this approach doesn't take into account the needs of industry. For example, when new companies are considering relocating to the province, there can no longer be an industry-based approach using LMDA PBPMs to develop programs at the local community college to meet the skills needs of such new companies. The phase-out of PBT and POT was felt by the majority of training deliverers (colleges, universities, institutions, etc.) to have seriously compromised the relevance of the PBPMs. Some respondents perceived that there is limited recourse to address the training needs of a particular sector, industry or employer, and that there is less incentive for training institutions to partner with local industry in addressing labour market issues.

Suggestions for Improvement

Key informants and focus group participants had numerous suggestions to improve the relevance of the PBPMs to the needs of individuals, employers and communities. Some changes and adjustments have already been made to enhance the relevance of the PBPMs. Prior to the transfer of dollars for the programs from the federal government, the province of Manitoba had a number of programs for income assistance recipients. Provincial officials stated that they have tried to take a blended approach where they use some of their provincial resources and some federal resources to bring together a package that will help meet the needs of a range of people (e.g., income assistance recipients) other than only EI clients²⁴. However, some provincial officials specified that they are limited in terms of the number of circumstances where they can use such an approach. It was suggested that greater relevance could be achieved through partnerships with external stakeholders to provide assistance to multi-barriered client groups when those barriers are beyond the Province's mandate and ability to deal with them under LMDA PBPMs.

Other respondents suggested there is a need for more dollars to be invested in skills development and training and for a lot more flexibility in the interpretation of the guidelines (e.g., third-party delivery agents cannot provide work experience under EAS). Other respondents felt that the issue was less one of the lack of relevant programming but more one of insufficient information and communication between governments concerning what programs can be accessed.

Also identified was the need for programs to be more responsive to the regional and local needs of communities. Some respondents felt that this situation would be improved with the recent hiring by HRDC of labour market information analysts in the north. Better labour market information and communication would also improve the relevance of programming.

Views on the degree of success of LMDA implementation with respect to the design and delivery of the Provincial Benefits and Provincial Measures (PBPMs) are presented in this chapter. These evaluation findings are drawn primarily from the key informant interviews and focus groups and supplemented with survey results.

4.1 Complementarity/Overlap of PBPMs with Other Programs

Most federal and provincial key informants as well as community partners and delivery agents perceived that the PBPMs may overlap and that there was a lack of coordination between:

• provincial programming for youth (defined as those aged 16 to 24) and federal youth programs (defined as those aged 16 to 29);
• PBPMs and federal programs for Aboriginal peoples; and,
• the Opportunities Fund for Persons with Disabilities and the Employability Assistance for Persons with Disabilities (EAPD) program.

Some provincial officials observed that there is also a lack of coordination between federal and provincial Labour Market Information (LMI) services. In addition provincial officials noted some administrative overlap in the treatment of apprentices by Employment and Training Services, the Apprenticeship Branch, the colleges and the federal EI program (e.g., overlap in the documentation of apprentices, reporting of absenteeism and follow-up on apprentices' cheques).

Although many respondents acknowledged that there is room to improve the coordination of programs (e.g., through partnerships), the consensus was that programs are mostly complementary.

4.2 Implementation of the LMDA

Key informants and focus group participants offered their views on aspects of the LMDA implementation that were most successful and on aspects that did not go as well. There is a high degree of similarity in the opinions expressed by the different respondent groups. On balance, it appears that the implementation of the LMDA has gone reasonably well, considering the complexity of the task and the effort involved, although some "growing pains" have been experienced and some problems remain to be resolved. The major observations on the most and least successful aspects of LMDA implementation are summarized in the remainder of this section.

Most Successful Aspects of LMDA Implementation

• Effective structure and process for LMDA implementation. The implementation of the LMDA was overseen by joint federal-provincial committees, including union representatives. These transition teams had responsibility for different areas, including human resources, operations, systems and finances. The Province developed a workplan and set two priorities to guide the implementation process: minimum disruption to staff and minimum disruption to clients. This structure and process worked well.
• Cooperation and compromise between federal and provincial partners. During the implementation process, there was a lot of dialogue, cooperation and compromise between the federal and provincial partners.
• Successful transfer of programs and responsibilities. Considering the commitment and effort required to implement an initiative as complex as the LMDA, all within a tight time frame, the transfer of responsibilities from HRDC to the Province went smoothly. The fact that the PBPMs are virtually identical to the former HRDC programs facilitated the transfer.
• Transfer of highly experienced and committed staff. Federal staff who were highly trained and experienced in the delivery of employment programs were transferred to the Province to deliver the PBPMs. The staff being transferred were treated with respect, and they received a financial incentive (i.e., signing bonus and severance pay) and recognition (i.e., a certificate and brief-case) from HRDC. Co-location has facilitated networking between federal and provincial staff. The cooperation and commitment of provincial staff members also made the transition go smoothly.
• Maintenance of good client service. For the most part, the transfer of responsibilities to the Province has been invisible to clients, and there has been minimal disruption to client service.

Least Successful Aspects of LMDA Implementation

• Limited time frame for negotiation of Agreement. Federal officials observed that LMDA negotiations were done very quickly. Also, from the union's perspective, staff and management on both the federal and provincial sides had insufficient input into the process because only those on the temporary negotiation team were involved. In addition, Employment Centre staff noted that, because the LMDA was "fast tracked", the required resources, systems, policies and procedures were not in place for the transition. These shortcomings caused problems, confusion and frustration for staff.
• Lack of consultation on PBPM design with community partners. Many community partners felt that they should have been consulted on the initial planning and design of the PBPMs. They believed that this would have helped the programs to better focus on client needs and that problems with design and implementation would have been avoided.
• Reduced program flexibility at local and regional levels. Many key informants and focus group participants perceived that the PBPMs do not have enough flexibility because of the Province's decision-making process (approval of expenditures) and also because the program eligibility criteria pursuant to the EI Act exclude many individuals in need of training and employment programs from participating in all the PBPMs²⁵.
• Inadequate information and monitoring systems. The majority of respondents identified inadequacies in the federal information and monitoring systems. Available measures are of limited use for the LMDA, there are errors in the data, and the systems are inadequate for planning, financial management and evaluation of the PBPMs. Moreover, provincial officials felt that staff were not properly prepared or trained in the use of these systems and that insufficient staff years and resources were provided for administrative support. The Province is currently developing a new information system.
• Difficult adjustment to new organizational culture and co-located sites. Former HRDC staff have had a difficult time adjusting to the provincial government culture which, from their point of view, involves having less power and decision-making authority, fewer resources, and a more centralized, bureaucratic organization. In addition, staff at co-located Employment Centres have had to sort out responsibilities for the reception function and how to share resources, space and information.
• Internal communications within the provincial government. Staff perceived that internal communications within the provincial government have been insufficient with respect to guidelines for delivering the PBPMs, human resource issues (discussed below), and the future of the LMDA. The lack of communications has left many staff feeling confused and anxious, and many perceive that program delivery is inconsistent.
• Human resource issues. Staff who were transferred to the Province expressed a number of concerns about human resource issues. In particular, they felt unfairly treated because their seniority with HRDC was not recognized by the Province and they were dissatisfied with how they have been re-classified in the provincial government (which is exacerbated by the perception that staff who remained with HRDC have been re-classified in a far more favourable manner). Some staff felt the need for clarification of their new job roles and for training in the new programs. Moreover, as indicated above, many staff were anxious about their job security due to inadequate communications on the future of the LMDA.

4.3 Partnerships

HRDC officials observed that the planning and implementation of the PBPMs has been facilitated by a productive federal-provincial partnership. Similarly, provincial key informants observed that PBPM planning and implementation has been done in partnership, in particular with HRDC, but also to some degree with other provincial departments, municipal governments, industry associations, employers and community organizations. In their view, many of these partnerships had existed prior to the LMDA, and have been strengthened through the process of implementing the LMDA and associated programs. Staff also felt that implementation has been done in partnership, citing Employment Partnerships and Labour Market Partnerships as successful examples.

Provincial respondents did identify some weaknesses in the partnerships:

• There have been challenges in developing working relationships with First Nations and Métis organizations, which have their own agreements with the federal government, though the Province has been working on resolving issues and developing employment strategies in cooperation with Aboriginal groups.
• Community partners and delivery agents would have liked to have been consulted more regarding the planning and implementation of the PBPMs, but this has been difficult because these organizations would want to put pressure on the Province to loosen the EI eligibility criteria— criteria that stem from the federal EI Act and hence are not under provincial jurisdiction.
• The involvement of EI client groups²⁶ and special interest groups in the design and delivery of programs has been limited.
• Some staff have experienced barriers in the development of partnerships, such as a lack of organizations/delivery agents in the region with which to form partnerships.

Most community partners and delivery agents indicated that they were not adequately consulted regarding the initial planning and implementation of the PBPMs. Still, they have generally found provincial government staff, such as project officers, to be very helpful and cooperative (e.g., in sharing information with community organizations and providing guidance for the development of projects relevant to the labour market). Cooperation has been good in Labour Market Partnerships and Employment Assistance Services, though a lack of support services like daycare for clients taking training was identified as a weakness. In addition, partnerships among various community organizations and delivery agents have a long and successful history, as this is a core feature of their service delivery approach.

4.4 Flexibility in Implementing PBPMs

Mixed views were expressed regarding the degree to which the PBPMs are sufficiently flexible to allow program decision-making and responsiveness at the provincial, regional and local levels. Federal officials observed that, because the Province is more centralized in its decision-making than HRDC used to be, some flexibility has been lost at the regional and local levels. For instance, federal officials perceive that third-party delivery arrangements under Employment Assistance Services are more restrictive; that local programs are somewhat less timely and responsive to client needs; and, that there are delays in project approvals. It was suggested that, on balance, the PBPMs offer more flexibility than the former provincial approach, but less flexibility than the former HRDC approach.

Provincial managers explained that they attempt to strike a balance between decentralized delivery to allow some local and regional flexibility, and centralized control to ensure that provincial priorities are met and that a consistent quality of service is provided across the province. Staff generally felt that the Province's centralized decision-making limits the regional and local flexibility of programs. In addition, some staff identified unique barriers to the delivery of flexible, responsive programming: difficulties in providing programs for small communities; and, administrative hurdles such as limited latitude to use various sources of funding and an inability to contract on Aboriginal reserves.

While community partners held mixed views about program flexibility, the opinions of third-party delivery agents were more consistent. The majority of delivery agents perceived that local-level flexibility in the delivery of PBPMs is impeded by both the Province's centralized decision-making approach and the restrictive EI eligibility criteria²⁷ . Delivery agents complained that the short-term contracts with the Province do not allow them to plan their service delivery appropriately or to keep experienced staff. Furthermore, there is a perception that many clients in need who do not meet the EI eligibility criteria are not being adequately served by the LMDA benefits and measures²⁸. These clients include youth and people with disabilities who have a weak attachment to the labour market, chronically unemployed people and marginally-employed people (i.e., those working in low skill jobs) who need skills upgrading. In addition, community partners indicated that clients in remote communities are not receiving adequate educational and training services (due to the lack of infrastructure and programs) and clients with multiple employment barriers do not have access to long-term programming. For many partners and delivery agents, the services under the LMDA are perceived as being driven more by program eligibility guidelines²⁹ than by client needs. Still, some delivery agents noted that they are able to serve non-EI eligible clients through other program vehicles and sources of funding.

4.5 Adequacy of Information and Monitoring Systems

The information and monitoring systems for the LMDA and PBPMs are a source of concern. Provincial government officials expressed dissatisfaction with the available federal systems for monitoring the PBPMs, third-party delivery and program participants, explaining that there are errors in the data, the available measures are of limited use for the current application, and the systems are inadequate for planning, financial management and evaluation of the PBPMs. Similarly, staff had little confidence in the current monitoring of third-party delivery agents and participants, and argued that the current systems do not adequately capture the work they are doing with the PBPMs and the range of client outcomes/impacts. Third-party delivery agents echoed these concerns and added that the Contact IV system entails excessive monitoring requirements, that some data fields are unclear to them and of limited use (e.g., "complete"), and that they have not been provided with adequate measurement tools or a budget for monitoring clients.

Federal officials acknowledged that there are problems with data integrity, and noted that this is a national problem. In order to address the systems issues, the Province is currently developing a new monitoring system — tailor made for the LMDA — that will replace the existing system and will have the capacity to provide better accountability and management information.

4.6 Adequacy of Labour Market Information and Labour Exchange

Federal and provincial officials agreed that Labour Market Information (LMI) is accessible to clients, employers and service providers via the Internet, though this is not useful for those who are not computer literate. In addition, federal LMI units, which provide labour market analysis and information products, are located in Winnipeg, Brandon and other centres. These federal units have recently hired more LMI staff. The Province also produces LMI. Currently, there is a federal-provincial working committee on LMI, and there are plans for the development of joint LMI research and products.

Although LMI is accessible, many provincial managers, staff, community partners, delivery agents and employers were dissatisfied with the usefulness of the currently available information. The major complaints about HRDC's information were that it is not current, national-level information is not relevant and specific enough for small regions and centres in Manitoba, and accurate forecasts of supply and demand in specific industry sectors in Manitoba are unavailable. A need to improve the provincial LMI system was also noted by some key informants. In fact, many respondents suggested that the best way to obtain accurate labour market information is simply to ask local employers if they are hiring and what types of skills they need.

In the survey of participants, only 25 per cent of respondents indicated that they had reviewed written material on the labour market on their own. However, a large portion of LMI is also available in electronic format and on the Internet. The evaluation found that of the proportion of respondents who reported having accessed a computer, Internet job postings or written LMI, 54 per cent had used one of these three resources. Very few clients consulted in focus groups indicated having used LMI, and among those who had, the majority were dissatisfied with the usefulness and relevance of the information to their needs. Many clients wished that staff had provided them with useful guidance on the labour market.

Regarding the Labour Exchange, federal officials as well as third-party delivery agents observed that job listings in the Job Bank and on the Internet are frequently out of date, which frustrates clients. Very few clients in the survey (21 per cent) indicated viewing job listings on the Internet, though roughly half (52 per cent) had used the Job Bank kiosk.

4.7 Strengths and Weaknesses of the PBPM Delivery Structure

The service delivery structure for the PBPMs includes four Employment Centres and the Employment Partnership office in Winnipeg and 12 Centres in smaller communities throughout the province, all of which are co-located except the Employment Centre in Brandon. In addition, for some PBPMs (i.e., Employment Assistance Services, Employment Partnerships and Self-Employment), programming is delivered through third-party service providers who are under contract with ETS. In the key informant interviews and focus groups, a number of strengths and weaknesses of the delivery structure for the PBPMs were identified. The strengths included:

• The delivery of the programs through the co-located Employment Centres was regarded as a key strength because it offers single-window service to clients and facilitates coordination and networking between provincial and federal staff sharing the Centres.
• The skill and commitment of staff at Employment Centres facilitate program delivery and client service.
• The fact that PBPMs are being delivered in partnership with industry, employers, educational/training institutions and other community organizations was viewed as a strength because it helps to ensure the delivery of training and employment programming that is relevant to the labour market.
• Using the services of third-party delivery agents (who have significant expertise and sensitivity in their area of programming) was thought by many respondents to be a cost-effective delivery approach.

Respondents identified the following weaknesses and areas for improvement:

• A major issue raised by many key informants and focus group participants pertains to a perceived lack of communication, guidelines and policies for the delivery of the PBPMs. This is a multi-faceted issue incorporating:
  • lack of communication from the Province to community partners and delivery agents regarding future directions for the PBPMs; and,
  • a lack of clear staff guidelines for the delivery of PBPMs and for third-party contracting (including the lack of an expert "program consultant" on staff, who can answer questions on delivery guidelines, etc.). This lack of clear guidelines causes inconsistent delivery; and inadequate outreach and promotion of the PBPMs to clients and community groups.
• Linked to the above first point, many respondents suggested that there is a need for some clarification of roles and responsibilities. This would include the specification of federal and provincial responsibilities for the reception function at co-located Employment Centres, clarifying federal and provincial roles for programming for youth, persons with disabilities and Aboriginal peoples, and clarifying the role of the communities in the delivery of services.
• The short-term contractual basis for delivery of services by third-party delivery agents makes it difficult to administer the programs.
• In some cases, excessive administrative requirements (e.g., for agents/partners delivering training and for those dealing with apprentices) were identified as a barrier to efficient service delivery.
• Some community partners suggested that there is a need to develop more relevant guidelines for the administration of some programs and to increase flexibility when applying program guidelines.
• Limited access to programs and services in remote areas of the province and poor access for persons with disabilities (who have special needs) and for non-EI eligible clients were identified as problems.
• Some staff suggested that there is a need for more human resources and for private office space (for interviewing and counselling clients).

4.8 Impact of Co-Location on PBPM Delivery

Most federal and provincial officials as well as community partners believed that co-location will be beneficial for client service (e.g., the provision of seamless, "one-stop shopping") and that relations between federal and provincial staff have been generally good, despite some "growing pains". There does not appear to be any difference in the quality of federal-provincial relations at co-located sites as compared to the one site not co-located (Brandon), except that a little more effort needs to be devoted to communications at the latter.

In the focus group discussions, some employers and clients were able to provide some observations on a co-located employment centre. The key beneficial impacts of co-location were perceived to be improved access to services, quicker and more efficient service, and more convenient service because everything is located in a single office. Clients did appear to have some initial difficulties finding the appropriate staff person to serve them.

4.9 Accessibility of PBPMs in Both Official Languages

The PBPMs are quite accessible to clients in both official languages, though there appears to be a minor problem with the availability of program information in French at the Winnipeg Employment Centre. In the survey of participants, 79 per cent of respondents indicated being satisfied with the accessibility of programs and services in general, and no significant differences as a function of clients' first language were observed. In addition, only three per cent of respondents indicated that they were unable to obtain information about employment programs in their preferred language (which was French for over half of these clients).

One possible explanation of the problem with the availability of program information in French was mentioned in the focus groups. Employment Centre staff in Winnipeg noted that, due to a lack of office space at present, there is not always enough room to display the government program literature in French. In addition, francophone clients indicated that it can be difficult to obtain services in French at the Winnipeg Employment Centre, though French services are readily available at the St. Boniface Centre.

4.10 Satisfaction with the PBPMs

PBPM participant survey respondents were asked to rate their satisfaction with various aspects of the programs and services they received. As shown in Exhibit 4.1, respondents were satisfied (responded with a 5, 6 or 7 on a 7-point scale) with most aspects of the programs and services they received and were most likely to be satisfied with the accessibility of the programs and services they received (78 per cent) and the comprehensiveness of the information about the programs and services they required (78 per cent). Respondents were slightly less likely to be satisfied with outcomes they achieved as a result of the services or programs (70 per cent).

Analyses were conducted to determine whether there was a relationship between clients' rated satisfaction with various aspects of the programs and services they received and the actual employment outcomes that they experienced. Satisfaction ratings were compared for participants who did and who did not report positive outcomes on four different employment outcome measures: 12 consecutive weeks of employment, weekly earnings, current employment status and type of current employment (i.e., year-round, seasonal or contract/casual). These analyses revealed that participants' employment outcomes tended to be inversely related to their level of satisfaction with different aspects of the programs and services.

Most respondents indicated that they were pleased with the services they received at the Employment Centre and a few indicated that their understanding of their needs changed as a result of speaking with a counsellor or other staff. The majority, however, noted that their understanding of their needs had stayed the same. Clients are generally satisfied with the PBPMs being offered and the areas of particularly high satisfaction included: accessibility of PBPMs; the expertise of staff; timeliness of services and self-service resources (e.g., Internet and computer access); quality of programs and services, particularly EAS; and financial support received whether EI or funds for training. There were a few negative comments expressed but the criticism generally related to the delivery structure or the guidance received rather than to the actual benefits and measures.

Program-specific opinions included:

• Most Enhanced Fee Payer clients felt that the training they received prepared them for the job market and they expected a positive outcome in terms of employment. A number of clients felt that the programs were not up to date or too short and did not provide them with the skills necessary to be job ready or to access higher-paying jobs.
• Overall, most participants who had been involved in some kind of employment or work placement programs, whether through an apprenticeship or with the help of a wage subsidy, were satisfied with the experience they gained and felt it was a positive step towards upgrading themselves. Wages were an issue for some participants in a wage subsidy placement who indicated that the salary was insufficient and/or they were overqualified for their particular job.
• Many focus group participants who obtained Employment Assistance Services were very satisfied with the services offered by third-party delivery agents and felt these were most useful in providing them with direction and helping them find employment. Participants stated that the services received are different from and complementary to those offered at the Employment Centres. Participants indicated they received individual counselling, employability skills, basic computer skills, job finding skills and help writing resumes.

4.11 Use of Self-Serve Resources

LMDA program participants were asked to indicate what resources or services they used on their own at an employment resource centre. This section presents the findings for all PBPM participants. As shown in Exhibit 4.2, respondents were most likely to indicate using job bank kiosks or job board listings (52 per cent) or a computer (34 per cent) on their own. Respondents were least likely to have used Internet job listings (24 per cent) and to have indicated they did not use any resources on their own at an employment resource centre (20 per cent).

Sub-group differences by program type revealed that EAS, SLG/Enhanced Fee Payer and Employment Partnerships respondents were more likely to report having used different resources and services on their own, whereas Apprenticeship participants were less likely to have used any of the resources on their own (which would be expected given the nature of the Apprenticeship program).

Respondents were most likely to have first learned about the employment services they used on their own through word of mouth (29 per cent) followed by a third party (17 per cent) and an advertisement (15 per cent). Roughly one in five respondents (19 per cent) indicated they knew of these services all along (Exhibit 4.3). Wage Subsidies and SLG/Enhanced Fee Payer respondents were more likely to report having first heard of the employment programs through word of mouth, whereas EAS respondents were less likely to have first heard of the programs through word of mouth. Advertising seems to have reached EAS and PBT clients more often and was reported less often by Wage Subsidies clients and Apprentices. EAS participants reported a referral from a third party more often, and Employment Partnership participants were more likely to report having first heard of these services from employment resource centre staff.

Respondents (active EI claimants) reported that they found out about the employment program in which they participated an average of 4.7 weeks after they established their most recent EI claim. On average, college educated and married respondents reported that it had taken them longer to find out about the program after the establishment of their most recent claim (8.4 and 6.4 weeks, respectively). This period was shorter for single respondents (2.9 weeks) and those with a high school education or less (2.9 weeks).

4.12 Suggestions for Improvement

In the focus groups, participants were asked if they had any suggestions for improving the design and delivery of the PBPMs. Many of their concerns have already been raised in earlier sections of this chapter. Their major suggestions are summarized below.

• Employment Centre Staff: As already noted, many staff felt the need for improved internal communications/guidelines, training and clarification of their job responsibilities for the delivery of the PBPMs. They also suggested that better communications between federal and provincial program officials would help to provide a check on potential abuse of program funds (i.e., inappropriate use of similar programs offered by both the federal and provincial governments). In addition, in order to improve the local relevance of programming, staff indicated a need for local/regional strategic planning, for more staff input into program design, and for more flexibility to adapt program guidelines to local and regional priorities. Related to this, some staff suggested that more flexibility to extend the period of funding would improve service delivery in some cases. For instance, this would allow a longer period of subsidized job experience for clients who need this following training and would enable community delivery organizations to develop their service capability and clientele. Finally, some staff felt the need for more financial and human resources to support PBPM delivery.
• Third-Party Delivery Agents: Like the Employment Centre staff, delivery agents suggested that program delivery would be improved with more flexibility at the local level and with quicker, more decentralized decision-making regarding program funding. In addition, they indicated a need for: longer-term contracts (e.g., three years rather than one year) so they can plan properly and retain skilled staff; more government consultation with delivery agents and opportunities for agents to share best practices; and more funding for professional development, both for themselves and for Project Officers at Employment Centres. Finally, some delivery agents suggested that allowing some funds for "purchase of training" or "project-based training" would help to serve the needs of clients with special needs (e.g., people with disabilities) and those in rural and remote communities where there is limited access to educational/training programs. However, it should be noted that "purchase of training" and "project-based training" are no longer permitted under the EI Act. These activities were terminated nationally and replaced with SLG under the LMDA.
• Community Partners: Included in the suggestions of this group was the need for: more access to training for marginally-employed people, clients with special needs (e.g., persons with disabilities) and those in rural and remote areas; longer periods of job placements/wage subsidies; longer-term follow-up with clients to increase the success rate of programs; training that is better targeted to the needs of the labour market; and, reduced paperwork associated with the PBPMs.
• Employers: Similar to the community partners, some employers called for a better matching of educational and training programs with the labour market and for an extension of the duration of wage subsidies. Moreover, they indicated a need for better screening of clients (by counsellors) being considered for wage subsidies and for the release of more client background information to employers so that a better employer-client match can be achieved in the Wage Subsidies. They also noted that the PBPMs need to be better promoted to employers. Finally, some employers emphasized the importance of pre-employment programs and for more in-depth counselling and coaching for clients who need this to be job ready.
• Clients: PBPM participants indicated a need for better service from Employment Centre staff, including more coaching, guidance and counselling for clients who need the assistance to increase their self-esteem, improve their job search skills, learn how to use the Job Bank, and learn about the labour market. Some participants also called for longer-term financial assistance, better promotion of available programs and services, more and better training opportunities, better labour market information, and more services for persons with disabilities and older workers.

In this chapter, evaluation findings related to the success of the PBPMs to date are presented. It is important to note that this formative evaluation provides preliminary evidence on the success of the PBPMs. In-depth analysis and assessment of the success of the PBPMs requires data gathered over a longer period of time and the application of higher level analysis than was possible during the formative evaluation. A summative evaluation, planned for fiscal year 2002-03, will be undertaken to provide a comprehensive and thorough reporting of the success of the PBPMs.

Issues addressed in this chapter include promotion of client self-reliance, the attainment of results targets, survey results pertaining to the impact of the PBPMs on clients, the impact of PBPMs on employers and communities, features of PBPMs associated with positive outcomes, and unintended impacts of the PBPMs. The treatment of issues presented in this chapter draws primarily on evidence derived from the participant and comparison group surveys, as well as the key informant interviews and focus groups. In several areas, such as satisfaction with the PBPMs and perceived impacts, a number of sub-group differences by program type, as well as by respondent demographics, were observed. Subgroup differences by program type are reported in the text. Differences according to respondent demographics are not presented in the text, but appear separately in Appendix B³⁰, as no clear pattern of results was apparent.³¹

5.1 Promotion of Client Self-Reliance

More than half of the respondents indicated that they had spoken with an employment counsellor or facilitator to help them plan their strategy to return-to-work (56 per cent), and 23 per cent of respondents indicated having developed an action plan with an employment counsellor. Fully 78 per cent of respondents who developed an action plan indicated that they had completed the activities in their plan. By far the most common reason for non-completion of action plan activities among the remaining 20 per cent of these participants was that they had found a job (61 per cent) (Exhibit 5.1).

Focus group participants (Employment Centre staff and third-party delivery agents) agreed that the design and delivery of the PBPMs is compatible with a focus on client responsibility. Respondents stated that some clients are more self-reliant under the PBPMs and are assuming more responsibility for ensuring their own success. Others commented that the level of responsibility assumed by clients varies with each individual depending on a number of factors, such as their willingness to accept this responsibility, their life skills, and their social and economic conditions.

With respect to making decisions about training, preparing and following through on action plans, and job search, staff generally agreed that the clients' responsibility has increased. It was further mentioned that there are better resources in the Employment Centres now for clients to do their own research, but labour market information is limited and is often not up to date. Some Employment Centre staff also stated that the termination of Purchase of Training and the new Skills Loans and Grants/Enhanced Fee Payer approach helps promote client responsibility because clients are asked to make a commitment to their training.

Third-party delivery agents reported a number of factors associated with client involvement, including: providing a safe and secure environment where trust and confidence can develop between the client and the service provider; client's self-identification of their employment needs; appropriate referrals to services; and proper needs assessments. Opinions varied, however, as to the willingness and capacity of clients to assume responsibility for their employment situation. A barrier for some clients is the poor access to training in rural communities coupled with their inability to re-locate to a larger centre due to family responsibilities. Several third-party agents also observed that many clients lack job search skills and are not necessarily job ready, so there is a limit on how much responsibility they can assume. In addition to developing job search skills these clients would greatly benefit from receiving coaching and developing life skills.

Clients themselves indicated that they take on a lot of responsibility and many stated they took almost total responsibility for ensuring their own success. Clients noted they assumed responsibility in identifying their needs, conducting research, and selecting not only the type of program or training they should pursue but also the training provider. For the most part, clients felt they were asked to take on the right amount of responsibility. However, some clients felt they were asked to take on too much responsibility and interpreted staff's promotion of their self-reliance as poor service; they felt that because staff were expecting them to do the research and to find out what was available to them (e.g., training programs, college courses), the staff were not doing their job. Also, for clients who are not computer literate the job bank kiosk was difficult to use. Overall a majority of clients believed that an appropriate balance had been established in terms of client-counsellor responsibilities.

With respect to the government's role in ensuring the success of clients involved in various PBPMs, most clients agreed that the government assumed an important role with respect to financing their training. Many participants stated that they would not have been able to access training without this support.

In summary, staff and clients agree that clients are assuming more responsibility for their employment situations, although the degree to which they are able to do this differs as a function of skills and personal resources. Additional barriers to self-reliance were felt to be the lack of good labour market information and poor access to training in rural areas. Most clients felt they were asked to assume an appropriate level of responsibility, though some clients interpreted the emphasis on client self-reliance as poor service delivery.

5.2 Primary Indicators

The Canada/Manitoba LMDA, like all provincial and territorial labour market development agreements, specifies primary results indicators for Provincial Benefits and Provincial Measures (PBPMs). The targets are set annually in three areas: the percentage of PBPM participants who are active EI claimants; participants returning to work; and unpaid EI benefits resulting from active EI claimants returning to work before the EI claim has ended.

It was impossible to fully assess target attainment for the 1999/2000 fiscal year owing to the timing of the evaluation, administrative data being available only up to August 15, 1999. Table 5.1 below presents the results of the computations while Appendix C³² contains the complete analysis.

5.3 Comparison of Evaluation Findings and Accountability Measures

In this evaluation, employment outcomes from the participant survey were compared against the return-to-work results in the administrative data systems. Our analysis found that administrative data returns to work were 66 to 80 per cent of returns to work as reported in the survey. See Appendix C³³ for the complete results of this analysis.

5.4 Impacts on Participants: Descriptive Analysis

This section presents a summary of findings relating to the impacts of PBPMs on clients. The focus is on clients' ratings of the importance of the help they received in obtaining employment, as well as objective measures of labour-market outcomes as revealed by clients' labour market status. The client outcomes that are presented in this section include: employment, joblessness, job-search behaviour, attitudes, and utilization of income-support. All sub-groups were compared to the participant total to determine whether they were significantly different from the overall result. The interested reader is referred to Annex 1 of this report³⁴ for the relevant data tables.

It is important to bear in mind that a true assessment of program impacts can only be made through the modelling of survey results, whereby various potentially confounding factors can be taken into account to provide accurate estimates of PBPM impacts.

For example, the fact that one PBPM appears to be more successful than another may have more to do with the characteristics of the PBPM participant (e.g., greater education or motivation) than the PBPM itself. It is those kinds of factors that multivariate analyses can control for. Additional information is provided in Section 5.5 of this report. Thus, the presentation of bivariate results in this section should be used for descriptive purposes only.

Rated Importance of Assistance

Survey respondents were asked to rate how important their employment program was in helping them to get their current or most recent job. Participant ratings of the importance of their employment program in this respect were moderate (Exhibit 5.2).

Perceived Program Impacts

Survey evidence reveals that the PBPMs were perceived to have had at least a moderate impact on a number of employment characteristics for the majority of PBPM respondents (Exhibit 5.3). Qualitative findings suggest, however, that the ability to detect impacts may be limited by inadequate tracking and monitoring systems, and narrow definitions of success related to immediate, rather than long-term, outcomes. Some clients were less happy with the impacts of PBPMs on improving their opportunities for better-paying jobs and their quality of life, and third parties expressed some concern over the limited access to PBPMs in rural areas.

Interest in Entering the Labour Force

Participants' motivations to be employed were measured through their rated interest in entering the labour force in the next 12 months. Of those respondents who were jobless at the time of the survey, the vast majority of participants rated themselves as very interested (responded with a 6 or 7 on a 7-point scale) in entering the labour force in the next 12 months. As expected, given their proactive behaviour in accessing a PBPM in order to return to work, participants were more likely to be interested in entering the work force than comparison group respondents. As well, active EI claimants in both groups were more likely than reachbacks to be very interested in entering the work force.

Labour Market Outcomes

This section presents survey evidence on several employment measures, including employment rates, employment stability, employment status and retention, for participants and comparison group members by claimant status (i.e., active EI claimant versus reachback). It should be pointed out that a different pattern of results was observed for reachbacks and active EI claimants for many of the employment outcome measures discussed in this section. Specifically, comparison group reachbacks were found to have more positive outcomes than participant reachbacks and comparison group active EI claimants, and similar outcomes to participant active EI claimants. As a result, for many of the employment outcomes presented in this section, while participant active EI claimants were found to have more positive results relative to their comparison group counterparts, the opposite pattern was observed for reachbacks (i.e., participant reachbacks had less positive results relative to comparison group reachbacks). Comparison group reachbacks differ from the other groups in that they would have received no form of assistance, whereas all of the participants and the comparison group active EI claimants would have received some form of assistance, either through employment insurance (EI), employment programming (PBPMs), or both. While existing data do not indicate significant differences in profiles between comparison group reachbacks and to the other groups, it is possible that these data do not capture all differences between the groups that could account for the more positive employment outcomes observed for the reachback comparison group.

a) Employment Rates

Overall, the survey results suggest that an advantage may exist for participants compared to comparison group members in terms of employment but not in terms of employment stability. Participants were more likely than comparison group members to be employed in the post-program period (first panel of Table D.1 in Annex 1).

b) Employment Stability

Data on two measures of employment stability were collected in the participant and comparison group surveys: the proportion of respondents who have worked for 12 consecutive weeks following the end of the program or reference date, as well as the number of employers they have had since that time (second and third panel of Table D.1 in Annex 1). While participant active EI claimants were more likely than comparison group active EI claimants to have worked for 12 consecutive weeks, the opposite pattern was observed for reachbacks. As well, comparison group members showed more stable employment than participants in terms of the number of employers they had in the post-program period.

c) Employment Status Outcomes

Participant survey respondents were asked about their employment status at two points in time following their intervention: at one week following the end of the program and at the time of the survey. Overall, these results show a positive shift in employment between these two times for PBPM participants, with the largest positive shifts in employment occurring for full-time year-round jobs (Table D.2 in Annex 1). These findings may indicate a persistence of positive employment outcomes among PBPM participants.

d) Pre-Post Employment Status

Another way in which the employment outcomes of LMDA program participants were measured involved a comparison of employment status in the week prior to the intervention or reference date and employment status at the time of the survey. Overall, these findings show positive shifts in employment from the pre-to post-intervention periods for PBPM participants. Relative to the comparison group the magnitude of the positive shifts in employment for PBPM participants was even greater (Table D.3 and D.4 in Annex 1).

e) Retention

A more direct measure of the contribution of PBPMs to positive employment outcomes is the extent to which job placement program participants were hired on by their host employers following the completion of the program. Only participants who completed their job placement program were asked this question. A majority of job placement program participants were retained in their PBPM program jobs, although the rates of retention, of being hired into the same program job and of being hired into full-time year-round employment were higher among Wage Subsidies participants relative to Employment Partnerships respondents (Table D.5 in Annex 1).

Characteristics of Current/Most Recent Job

In this section, the characteristics of respondents' current or most recent job are presented (for those respondents who have been employed at some time in the post-intervention period). In the post-program/reference date period, roughly one in four participants and roughly one in two comparison group members had the same job they had prior to their employment program reference date (Table D.6 in Annex 1). Participant active EI claimants worked longer hours than respondents in the other groups, and were more likely than comparison group active EI claimants to be employed year-round, although the opposite pattern was observed for reachbacks. Comparison group members reported higher weekly earnings.

Joblessness and Job Search Outcomes

Survey results for three post-intervention outcomes: number of weeks jobless (duration); number of weeks looking for work while jobless; and job search activity are presented in this section. "Jobless" individuals are defined as people who are officially unemployed (i.e., unemployed and looking for work), plus those who are not in the labour force.

a) Duration of Jobless Spells

Survey data were collected on the duration of jobless spells following the intervention (or reference period), scaled by the time since the intervention or program reference date, for both participants and comparison group members. The duration of jobless spells in the post-program period was shorter for participant active EI claimants than comparison group active EI claimants, although the opposite pattern was observed for reachbacks (first panel of Table D.7 in Annex 1).

b) Duration of Job Search

Another potential positive outcome concerns the number of weeks clients searched for work in the post-program period. It should be noted that the analyses conducted on this measure include only those respondents who were unemployed at some time in the post-program period. Overall, participants were much more likely than comparison group members to look for work in the post-program period (second panel of Table D.7 in Annex 1).

c) Job Search Activity

Those who reported actively searching for work in the post-program period were asked to specify the job search methods they used while looking for work. The most common job search methods were distributing resumes or applications and checking job banks (Table D.8 in Annex 1). Participants were more likely than comparison group respondents to engage in most job search methods.

Utilization of Income Support

This section presents results regarding participants' post-intervention use of two forms of income support: Income Assistance (IA) and Employment Insurance (EI).

a) Income Assistance

Another indicator of the extent to which PBPMs have positively impacted participants is the extent to which participation in the PBPMs has reduced clients' reliance on income support (IA). Survey evidence suggests that participant active EI claimants may have benefited from the program in terms of a reduced rate of IA use in the post-program period, although they may also have been at a disadvantage concerning the duration of IA use (first and second panel of Table D.9 in Annex 1).

b) Employment Insurance

Evidence concerning the incidence and rate of Employment Insurance (EI) use in the post-program/reference date period revealed that, overall, both the rate of EI use and the duration of EI use was lower for participant active EI claimants than for comparison group active EI claimants. The opposite pattern was observed for reachbacks. Thus, any advantage observed for participants in this regard may be limited to active EI claimants only.

5.5 Impacts on Participants: Multivariate Modelling

The multivariate analysis conducted in this evaluation examines the preliminary incremental impacts of the PBPM interventions in terms of employment, job search³⁵, earnings and income support use. The multivariate analysis controlled for antecedent differences in socio-demographic and background characteristics as well as the use of services such as counselling or an action plan between participants and the comparison group. It also attempted to control for possible self-selection bias (Heckman Correction Factor) — the possibility that the same unobserved variables determining participation in the programs may contribute to the outcomes. Any remaining advantage for participants could then be attributed to participation in the PBPMs. See Appendix E (under separate cover) for the supporting analysis.

As noted previously, this formative evaluation provides preliminary evidence on the success of the PBPMs. In-depth analysis and assessment of the success of the PBPMs requires data gathered over a longer period of time and the application of higher level analysis than was possible during the formative evaluation.

5.6 Impacts on Employers and Communities

Focus group participants (employers and community partners) agreed that the PBPMs have had a positive impact on employers, individuals and communities. Some employers stated that Wage Subsidies have enabled them to hire employees. In some instances, the fact that employers were able to hire new employees permitted them to expand their businesses and serve outlying communities. Furthermore, employers commented that they have been able to improve the quality of services they deliver to their own clients as they are able to provide training to new employees, which results in a more skilled workforce and better job performance.

5.7 Program Features of PBPMs Associated with Positive Client Outcomes

Staff indicated a number of features associated with positive client outcomes. They included:

• initial assessment of client needs, good counselling and a well-prepared action plan; and
• the SLG/EFP requirement for a client contribution towards his or her training.

Finally, staff suggested that longer-term funding for contracts with third-party delivery agents to develop longer-term programs might be important to ensure the success of clients with multiple barriers to employment who may need more time to complete their training or acquire the necessary skills to find and maintain employment.

Third-party agents delivering PBPMs identified many characteristics or program components associated with positive outcomes. They included:

• a comprehensive and thorough initial assessment of a client's needs to make sure the individual will be receiving the appropriate services;
• a comprehensive and thorough program that helps clients develop the necessary employability skills;
• a holistic approach that promotes self-esteem and self-confidence and empowers individuals with the tools and know-how to make career choices;
• marketing people directly to employers and targeted wage subsidies;
• a comprehensive and thorough screening process for employers to increase the probability that they will continue employing the client once the subsidy is over;
• sufficient flexibility to be able to adapt programming to respond to the needs of the labour market;
• a comprehensive and thorough screening process for individuals who want to participate in a Self-Employment program to make sure they have the qualifications to succeed in business;
• continuous monitoring of a client's progress throughout a Self-Employment program and follow-up after completion with timely interventions if necessary to prevent the failure of emerging businesses; and
• overall quality of training being offered.

Community partners generally stated a preference for service delivery models that rely on partnerships between the government and the community. These include:

• sector-based models, where programs are designed and implemented in consultation with sector agencies made up of employers, branches of government, schools, and private training institutions; and,
• third-party models, through which programs are designed and delivered through community and third-party organizations that possess greater expertise in delivering services to different client groups.

Finally, some employers made suggestions to ensure positive client outcomes:

• providing clients with an opportunity to obtain labour market experience;
• a comprehensive initial client assessment;
• an appropriate client/employer match;
• good communication between employers and provincial staff responsible for the Wage Subsidies program; and,
• increased government responsibility, in the form of a monitoring system to provide follow-up assistance to clients and employers.

5.8 Unintended Impacts

Focus group participants (Employment Centre staff, third-party delivery agents, community partners and employers) were asked if they had observed any unintended impacts, positive or negative, related to the employment programs. Most respondents could not comment on this issue. Some staff, however, noted there was a negative impact on non-EI eligible unemployed Manitobans who they feel are being neglected due to their ineligibility for benefits under the LMDA (e.g., benefits such as Enhanced Fee Payer which are only accessible to EI clients as per the EI Act). Staff also felt they do not have the time to adequately address the needs of non-EI eligible unemployed Manitobans who often need more counselling. Another unintended impact mentioned was a consequence of providing training cheques directly to clients, which apparently resulted in some students spending the training money for other purposes than their tuition fees and then leaving their educational program. The educational institutions would thus prefer that the money be sent to the school directly.

Many third-party agents noted that the programs have had positive impacts on the lives of people in a way that cannot be reflected in the current accountability framework. Clients develop a different perspective about what work is all about and a new attitude toward roles and responsibilities and these are important changes that affect all aspects of their lives. They suggested that the outcome measures should be broadened to capture all the medium and longer-term impacts on client groups.

A summary of the key evaluation findings from the formative evaluation of the Canada/Manitoba Labour Market Development Agreement (LMDA), incorporating all lines of evidence used in the research, is reported here. Findings are presented pertaining to the major evaluation issues addressed in this research, including:

• the implementation of the LMDA and Provincial Benefits and Provincial Measures (PBPMs);
• the relevance of PBPMs to clients, employers and communities and their satisfaction with outcomes;
• regional findings;
• the adequacy of information and monitoring systems, program specific findings;
• the service delivery model; and,
• key strengths and weaknesses of the delivery structure of PBPMs.

6.1 Implementation

The implementation of the Canada/Manitoba LMDA and the delivery of the PBPMs have gone well, however, there have been some "growing pains" and there are some outstanding issues to be addressed.

a) Consistency of PBPMs with EI Act and LMDA

With respect to consistency, the PBPMs reflect the guidelines, principles and intent of the Employment Insurance (EI) Act and the LMDA. They are consistent with Human Resources Development Canada's (HRDC's) priorities but only in part with Manitoba's priorities. The eligibility criteria of the EI Act limiting access to EI clients does not permit the Province to provide training to the marginally-employed individuals using LMDA funds³⁶. This is a major concern as Manitoba is entering times of skills shortages, however, this is a broader issue that is outside of the realm of the delivery of the PBPMs.

b) Partnerships

The PBPMs were designed and implemented in partnership with stakeholders, in particular between Manitoba Education and Training (MET) and Human Resources Development Canada (HRDC), but also to some extent with other provincial departments, municipal governments, industry associations, employers and community organizations. Many of these partnerships existed prior to the LMDA and have been strengthened with the implementation of the PBPMs. Although community partners and delivery agents have found provincial staff to be very cooperative, they do feel that they should have had more input into the planning, design and implementation of the PBPMs. In addition, there have been challenges in developing working relationships with First Nations and Métis organizations, as they also have their own agreements with the federal government to fund employment and training programs. The Province has been working on resolving these issues and developing employment strategies in cooperation with Aboriginal groups.

c) Most Successful Aspects of LMDA Implementation

The key strengths of implementation and delivery have been: an effective implementation structure and process with good cooperation between federal and provincial partners; the successful transfer of programs and highly experienced staff to the Province with minimal disruption to client service; cooperation of provincial staff members that made the transition go smoothly; partnerships with industry, employers, and educational and other organizations that have helped to ensure the relevance of programming to the regional labour market; and the delivery of programs by highly committed staff at co-located Employment Centres featuring "one-stop shopping" for clients, with cost-effective support from third-party delivery agents.

PBPMs are harmonized with other provincial and federal initiatives. Although there is some perceived lack of coordination between provincial and federal programs for youth, Aboriginal peoples, persons with disabilities and labour market information, the consensus was that programs are mostly complementary.

d) Least Successful Aspects of LMDA Implementation

There are a number of perceived weaknesses identified including the need for more flexibility in the design and delivery of PBPMs so that they are better adapted to local and regional needs; more useful labour market information; better access to programs and services in rural and remote communities; more consultation with and better promotion of the PBPMs to community groups; and better internal communications.

In addition, clearer guidelines and staff training for the delivery of the PBPMs are required to help ensure consistency; attention to some pressing human resource issues, including the resolution of concerns related to the job classification of staff transferred from HRDC, clarification of staff roles and responsibilities for program delivery and for the reception function at co-located centres, and alleviation of staff anxiety over the future of the LMDA and their job security. Results measures and monitoring systems to support the management and evaluation of the PBPMs continue to be a problem for the PBPMs.

Finally, the limited timeframe for the initial LMDA negotiations created some problems at the outset (e.g., the required resources, systems, policies and procedures were not in place).

6.2 PBPM Clients

The evaluation found that the PBPMs are largely relevant to the needs of clients and that clients are satisfied with the outcomes of the interventions.

a) Profile of Unemployment in Manitoba and Clients of PBPMs

The survey evidence indicates that participants in the PBPMs are representative of EI clients in Manitoba (e.g., in terms of equity group status and other socio-demographic characteristics). Compared to EI clients in general, program participants are somewhat more likely to be single, educated at the post-secondary level and anglophone, and to have a smaller household size and lower household income, though the differences are small.

b) Relevance of PBPMs to Clients and their Satisfaction with Outcomes

PBPMs are largely relevant to the needs of clients in Manitoba. Most clients indicated that the PBPMs met their needs and expectations and impacted positively on their lives. These clients note that the PBPMs helped them gain self-confidence, gave them a sense of direction and increased their ability to get a job.

Clients expressed high satisfaction with the timeliness and accessibility of services, self-serve resources, the quality of programs and services, and the financial supports received. The few areas about which clients expressed dissatisfaction involved aspects of service delivery rather than the PBPMs themselves and included the lack of up-to-date training courses as well as insufficient remuneration for Wage Subsidies and Apprenticeship participants.

With respect to accessibility to programs and services in the official language of their choice, evidence indicates that the PBPMs are easily accessible to most clients in their preferred official language. Only a small minority of survey respondents (three per cent) reported that they were unable to obtain program information in their language of choice.

c) Preliminary Impacts of PBPMs on Clients

Staff, clients, community partners and third-party agents all agreed that PBPMs have a positive impact on the lives of participants. Most staff also stated that they are satisfied that the programs and services offered meet the needs of LMDA eligible clients. Among the reported impacts observed were improved confidence and self-esteem, life skills, job search skills, employment, job satisfaction, earnings, employability and less reliance on income support.

d) Client Characteristics Associated with Success

Generally, increased education and being greatly interested in entering the labour force were strong pre-intervention predictors of post-intervention success, whereas the reverse was true for being in an equity group (e.g., persons with disabilities or Aboriginal persons). Participants previously employed and in higher paying positions suffered earnings declines following the intervention. Prior users of IA or EI were quite likely to receive IA or EI, respectively, following the intervention, indicating income-support use is not, so far, being eliminated through PBPM participation.

6.3 Employers and Communities

Most employers felt that the programs are relevant to their needs. Respondents expressed satisfaction with the degree to which programs suited their organizational goals as well as with the employees who worked for them through the program. Some dissatisfaction was expressed with respect to the lack of background information on job candidates (although impossible to share given privacy issues), the match of participants to the employers' businesses and the length of programs (i.e., period of funding). Another area of employer concern is the lack of resources to support the training and upgrading of current employees.

Some provincial officials and community partners feel that the idea that training can only be done through Skills Loans and Grants/Enhanced Fee Payer is a limitation because it does not take into account the needs of industry. For example, when new companies are considering relocating in the province, some respondents believe they can no longer have an industry-based approach and develop programs at the local community college to meet the skills needs of such new companies. This was previously accomplished using Project-Based Training and Purchase of Training, which were phased out on July 1, 1999.

The phase-out of Project-Based Training and Purchase of Training was felt by the majority of training deliverers (colleges, universities, institutions, etc.) to have negatively impacted the relevance of the PBPMs. Some respondents perceive that there is no recourse to address the training needs of a particular sector, industry or employer and that there is less incentive for training institutions to partner with local industry in addressing labour market issues.

Most employers and community partners felt that the programs are relevant to the needs of their communities. However, they noted a need for:

• increased access to training for marginally-employed people, clients with special needs (e.g., persons with disabilities) and those in rural and remote areas;
• longer periods of job placements/wage subsidies and longer-term follow-up with clients to increase the success rate of programs;
• training that is better targeted at the needs of the labour market; and,
• reduced paperwork associated with the PBPMs.

6.4 Regional Findings

There are some unique challenges in the rural and northern regions of Manitoba, including the difficulty in providing programs in small communities. There is also a lack of relevant training being offered in these communities. Furthermore, the individual approach to training through the Skills Loans and Grants/Enhanced Fee Payer was also viewed by some as a limitation because of the lack of available training programs.

New service delivery models need to be considered in order to address the access issue in rural and remote communities with a small population.

6.5 Adequacy of Information and Monitoring Systems

Respondents indicated that the administrative systems for monitoring PBPMs, program participants and third-party deliverers are inadequate for proper planning, management and evaluation of the PBPMs. The validity and usefulness of the current measures as well as the integrity of the data are perceived to be problems.

Weaknesses were also observed in the supporting information systems used to assess the accountability results targets. Comparisons between employment indicators from the survey and the return to work indicator in the administrative data indicated that a large number of returns to work were not captured in the information systems. This suggests that the accountability results for Manitoba, as recorded in administrative data, may under represent actual success with respect to returns to work and unpaid EI benefits.

Recognizing the need for improvement, the Province is currently developing a new LMDA monitoring system to correct these problems.

6.6 Labour Market Information and Labour Exchange

Labour Market Information (LMI) is readily available to staff, community partners, employers and clients via the Internet, though this medium is not accessible to those who are not computer literate. Staff also has access to LMI through provincial and federal LMI units.

The information provided by LMI is widely criticized, however, for being outdated and limited in relevance to client needs and to small regions and communities in Manitoba. In recognition of this problem a federal-provincial working committee on LMI has been formed and is planning to develop joint LMI research and products.

With respect to the Labour Exchange, the conclusions are similar to those for LMI, i.e., although job listings are easily accessible via the Internet and Job Bank, the information is often out of date and hence of limited use.

6.7 Service Delivery Model

With respect to the service delivery model, many respondents indicated that they feel there is a need for more flexibility in order to be able to address the needs of all the unemployed population of Manitoba. Although the PBPMs allow some degree of flexibility in program decision-making and responsiveness at the provincial, sub-provincial and local levels, two factors are widely perceived to reduce program flexibility. First, staff perceive that decision- making (approval of expenditures) is more centralized than it had been prior to the LMDA and this can cause delays in the implementation of programs. Second, the program eligibility criteria stemming from the EI Act are thought to create some limitations in programming³⁷. Some changes and adjustments have already taken place to increase the access to services and programs but it is felt that more needs to be done for the Province to be able to respond to the skill shortages they are currently facing.

Third-party delivery agents expressed concern as well about the short-term contracts they are given for the delivery of employment services. They feel that these short contracts (of one year or less) limit their ability to do proper long-range planning and to develop and retain skilled employees.

Community partners stated a preference for service delivery models that rely on partnerships between the government and the community which included: sector-based models, where programs are designed and implemented in consultation with sector agencies made up of employers, branches of government, schools, and private training institutions; and third-party models, through which programs are designed and delivered through community and third-party organizations that possess greater expertise in delivering services to different client groups.

The evidence suggests that co-location has been beneficial for client service as well as for working relationships and information-sharing between federal and provincial staff. However, clients did appear to have some initial difficulties finding the appropriate staff person to serve them when visiting an Employment Centre. As well, many do not appear to be aware of any changes in the quality of service.

a) Key Strengths and Weaknesses of the PBPM Delivery Structure

The major perceived strengths of the PBPM delivery structure include:

• single-window service for clients through the co-located Employment Centres;
• the skill and commitment of front-line delivery staff at the Centres;
• the partnership approach to PBPM delivery that helps to ensure the relevance of programs; and
• the cost-effective use of third-party delivery agents.

The major perceived weaknesses of delivery are:

• a lack of communication and a lack of program delivery guidelines for staff;
• inadequate promotion of the PBPMs to clients and community groups;
• some confusion about federal and provincial roles and responsibilities for the reception function at co-located Centres and for programming for youth, Aboriginal peoples and persons with disabilities; and
• limited access to programs for clients in remote areas of Manitoba, for persons with disabilities and other special interest groups.

6.8 Program Specific Findings

Program specific findings from all lines of evidence and issue areas pertaining to the PBPMs that were evaluated are presented here. These programs are: Wage Subsidies; Employment Partnerships; Skills Loans and Grants/Enhanced Fee Payer; Self-Employment; Employment Assistance Services; Project-Based Training; and Purchase of Training.

a) Wage Subsidies (WS)

Overall, most participants who had been involved in some kind of employment or work placement programs (i.e., with the help of a wage subsidy) were satisfied with the experience they gained and felt it was a positive step towards upgrading themselves. Wages were an issue for some focus group participants in a Wage Subsidies program who indicated that the salary was insufficient and/or they were overqualified for the particular job in question. Survey results indicate the majority of job placement participants who completed the full period of their subsidy were hired on by their host employer (60 per cent).

Some employers commented that candidates rarely have the necessary skills, so a period of on-the-job training is required, which is a significant investment for them. Feedback from employers indicated that one of the strengths of the Wage Subsidies program was that it enabled them to provide a wage for needed employees while the employees were being trained. Without the Wage subsidies program many of these employers would not have taken on and hired new staff.

Respondents stated that they are generally pleased with the degree to which programs met their organization's goals and with the employees working for them through the Wage Subsidies program. Respondents also noted that the approval process for PBPMs has become faster due to increased specialization for Project Officers, thus there has been greater efficiency and more willingness for employers to participate.

Other respondents expressed dissatisfaction with the lack of information they were given on a candidate's skills as well as with the length of programs (which they generally perceive to be too short). Some employers stated that there is sometimes a failure to match the interests and skills of program participants and employers, and that there is a lack of follow-up to ensure the success of a program.

Simple comparisons of employment outcomes indicate that 84 per cent of WS participants were employed at the time of survey and 95 per cent had worked for at least three consecutive months. These percentages are greater than the average overall percentages across all programs (70 per cent and 89 per cent, respectively).

b) Employment Partnerships (EP)

Employment Partnerships are regarded by Employment Centre staff as relevant to employers' needs so long as a comprehensive assessment of the needs of the individual and the employer is completed.

One particular issue raised in this evaluation relates to potential problems of specialized training for an industry such as aerospace. Specifically, the demand for workers and/or skills can change quite abruptly in this field, so sometimes highly skilled workers who have just completed their training (initiated when demand was high) can find themselves with no work if the demand decreases.

Simple comparisons of employment outcomes indicate that 74 per cent of EP participants were employed at the time of survey, which is above the average for all programs. However, 87 per cent had worked for at least three consecutive months since the intervention, which is less than the average overall percentage across all interventions.

c) Skills Loans and Grants/Enhanced Fee Payer (SLG/EFP)

Overall, participants in SLG/EFP programs were satisfied with the services received and the outcomes. SLG/EFP participants were likely to rate themselves as satisfied with how well programs and services suited their needs; with the outcomes they achieved as a result of the programs or services; and reported that their ability to find employment had improved to a large extent as a result of participating in their program. However, 78% of Enhanced Fee Payers indicated that the requirement to contribute to the cost of training made it difficult (to a moderate or large extent) to access training.

Most Enhanced Fee Payer clients who participated in focus groups felt that the training they received prepared them for the job market and they expected a positive outcome in terms of employment, whereas a small number of clients felt that the programs were not up to date or too short and did not provide them with the skills necessary to be job ready or to access higher paying jobs.

Simple comparisons of employment outcomes indicate that 76 per cent of SLG/EFP participants were employed at the time of survey, which is above the average for all programs. As well, 91 per cent had worked for at least three consecutive months since the intervention, which is somewhat above the average overall percentage across all interventions.

d) Self-Employment (SE)

Survey results revealed that Self-Employment participants were more likely than participants in other programs to report being satisfied with the accessibility of programs and services. Along with Employment Assistance Services (EAS) participants, SE participants were the least likely to feel that the employment program was very important in helping them find a job and were less likely to report that their program had increased their education or job skills to a large extent. This is not surprising given these two outcomes are not the intent of the program.

Simple comparisons of employment outcomes indicate that just under one-half (48 per cent) of SE participants were self-employed at the time of survey. A total of 90 per cent were either employed or self-employed, which is above the average for all programs. Moreover, 94 per cent had worked (as an employee or were self-employed) for at least three consecutive months since the intervention, which is also above the overall average.

e) Employment Assistance Services (EAS)

Survey results indicated that the likelihood of respondents reporting that they were satisfied with how well programs and services suited their needs was lower among Employment Assistance Services (EAS) respondents than all other types of respondents. EAS participants were also less likely to rate themselves as satisfied with the outcomes they achieved as a result of the programs or services.

As noted above, along with Self-Employment (SE) participants, EAS participants were the least likely to feel that the employment program was very important in helping them find a job and were less likely to report that their program had increased their education or job skills to a large extent. This is not surprising given these two outcomes are not the intent of the program.

In contrast, many focus group participants who participated in EAS were, however, very satisfied with the services offered by third-party delivery agents and felt these were most useful in redirecting them and helping them find employment. They stated that the services received are different from and complementary to those offered at the Employment Centres. Participants indicated they received individual counselling, employability skills, basic computer skills, job finding skills and help writing résumés. Most third-party delivery agents believed that these services make a huge difference in people's lives. They noted that clients have been able to develop life skills and job search skills. Many delivery agents would like to see a broader definition of outcome measures and indicators that would not be limited to statistics related to immediate employment outcomes.

Simple comparisons of employment outcomes indicate that only 61 per cent of EAS participants were employed at the time of survey, and 85 per cent had worked for at least three consecutive months since the intervention. Both these percentages were below the overall average for all interventions.

Impacts on Participants by Program Type Descriptive Result³⁸

This appendix contains data tables pertaining to the outcomes of PBPMs on clients. Sub-group results that are significantly different from the participant total at the five per cent level or better are labelled with an asterisk. It is important to bear in mind that a true assessment of program impacts can only be made through the modelling of survey results, whereby various potentially confounding factors can be taken into account to provide accurate estimates of PBPM impacts. For example, the fact that one PBPM appears to be more successful than another may have more to do with the characteristics of the PBPM participant (e.g., greater education or motivation) than the PBPM itself. It is those kinds of factors that multivariate analyses can potentially control for, the results of which appear in Section 5.5 of this report. Thus, the presentation of bivariate results in this appendix should be used for descriptive purposes only.

Table D.9 Utilization of Income Support: Weighted Percentage Distribution by Use and Weeks of Income Assistance (IA) Among PBPM Participants and Comparison Group by Intervention and EI/Reachback Status¹

	PBPM Participants by Program Type									PBPM Participants by Claim Status		Comparison Group by Claim Status
	Total	Employment Partnerships	Self- Employment	Wage Subsidies (WS)	Employment Assistance Services (EAS)	SLG/ Fee Payer	POT	App	PBT	EI Claimant	Reach- back	EI Claimant	Reach-back
1. Ever Collected Income Assistance Since End of Program/Reference Date**
Yes	8.6	7	2.6*	6.7	15.3*	5.6	9.7	3.1*	13.6	9	9	14.2*	2.7*
No	91.4	93	97.4	93.9	84.7	94.4	90.3	96.9	86.4	91	91	85.8	97.3
Unweighted n***	1,373	165	164	158	261	267	131	178	171	743	521	129	329
2. Proportion of Weeks Since Intervention in which IA Collected****
Mean (%)	44	59.2	57.6	25.1	63.7*	53.4	54	15.1	17.2*	39.4	49.9	21.1	50.1
Median (%)	31.2	17.2	50.2	16	50.9	54.9	45.5	15.1	11	24.5	50.9	11.1	49.9
3. Ever Collected Employment Insurance Since End of Program/Reference Date*****
Yes	28.9	40*	9.7*	34.6	30.2	19.6*	40.4	30.1	44.3*	27*	39*	76*	12*
No	71.1	60	90.3	65.4	69.8	80.4	59.6	69.9	55.7	73	61	24	88
Unweighted n***	1,393	167	166	160	262	271	133	185	172	754	529	146	351
4. Number of Weeks in which Employment Insurance Collected**** as a Percentage of Weeks Since Intervention
Mean (%)	15.9	12.6	12.8	17.4	21.7*	11.3	20*	10*	15	13.5*	16.6*	25.3*	9*
Median (%)	12.9	7.6	7.3	14.3	15.8	7.7	17.7	8.7	15.3	10.2	15.6	23.9	5.3
Source: HRDC administrative data and Canada/Manitoba LMDA Participant and Comparison Group Surveys. 1 Overall participant results are weighted by program type, region, claimant status and age; Participant results by program type are weighted by region, claimant status and age; Participant results by claimant status are weighted by program type, region and age; Comparison group results by claimant status are weighted by region and gender. * Differences statistically significantly at the 5 per cent level or better. Significant differences are based on comparisons with similarly weighted totals and not the overall total presented here. ** Income Assistance figures computed from survey data. *** Respondents who have participated in multiple interventions are counted for each program in which they have participated. Thus, the sum of n's for each program will be higher than the total number of cases presented in the leftmost column. Furthermore, analyses by claimant status exclude respondents whose claimant status is unknown, thus the sum of claimant and reachback cases will be lower than the total number of cases presented in the leftmost column. **** Based only on those who received EI or IA. ***** Employment Insurance figures computed from HRDC Administrative data.

Footnotes

1	Under the EI Act, an EI client is defined as an unemployed person requesting assistance who (i) is an active EI claimant, or (ii) had a benefit period that ended within the previous three years, or (iii) had a benefit period within the last five years and was paid maternity/parental benefits, subsequently withdrew from the labour force and would like now to re-enter the labour force. The latter two groups are collectively referred to as "reachback" clients. In the Canada-Manitoba LMDA "EI client" is replaced by "insured participant" — however, for the purpose of this document, only the term "EI client " will be used.
2	In the Canada-Manitoba LMDA these employment benefits and support measures are called "provincial benefits and provincial measures" (PBPMs).
3	The EI Act also identifies certain guidelines for PBPMs to follow. These guidelines are: harmonization, reduced dependence on EI benefits, cooperation and partnerships, flexibility, official language, individual commitment, and evaluation (EI Act, Section 57, Sub-section 1).
4	EI Act, Section 59 and Section 60, Sub-section 4.
5	EI Act, Section 61, Sub-section 3.
6	EI Act, Section 61, Sub-section 1.
7	As noted above, under the LMDA, Annex 1, Section 3.1, "Manitoba will assume responsibility for selecting priority clients for provincial benefits and provincial measures but only EI clients will be given access to provincial benefits."
8	Please note that some provincial benefits and provincial measures have been integrated with other provincial programs for better coordination of service delivery to all client groups including Income Assistance recipients. It is important to note that the use of funds from the EI Account for the provision of training and/or employment programs under the LMDA is clear — pursuant to the EI Act, funds from the EI Account are to be used to reimburse the Province for PBPMs accessed by "EI clients".
9	The Canada-Manitoba LMDA Joint Evaluation and Accountability Committee drafted the Evaluation Framework (January 1999). This planning document outlines evaluation issues and questions of interest to both federal and provincial partners. It also identifies broad indicators and evaluation strategies that encompass a mix of formative and summative evaluation elements.
10	Summative evaluations are designed to measure outcomes, impacts and cost-effectiveness.
11	All the appendices (A through E) are presented under separate cover; Appendix A: Evaluation Issues Appendix B: Subgroup Findings by Respondent Demographics Appendix C: Attainment of Results Targets: Canada/Manitoba LMDA Appendix D: Impacts on Participants by Program Type — Descriptive Results [this appendix also appears at the end of this report as Annex 1]. Appendix E: Multivariate Analysis of Impacts on Participants — Canada/Manitoba LMDA
12	As a result of changes to the Census questionnaire between the 1991 and 1996, comparable data are not available.
13	Paradoxically, the overall youth share of employment and the population has been declining over the last decade and will decline over the next 15 years.
14	The data were adjusted for under-sampling in remote regions and exclusion of on-reserve Aboriginal peoples in the LFS.
15	Data obtained from the Region in the form of 12 consecutive three-month moving averages covering a period from December 6, 1998 to December 4, 1999. These figures were annualized by EKOS by summing and dividing by 12 to get a close approximation of rates for 1999.
16	It was decided that participants in the Project-Based Training (PBT) and Purchase of Training (POT) programs, both of which were phased out in 1999, should also be surveyed.
17	For EAS, only clients from the EI system were used in the sample and therefore the results/conclusions are only applicable to that client subgroup.
18	As described previously reachback clients are defined as clients that were not active EI claimants (i.e., collecting employment insurance benefits at the time of their participation in a PBPM) but who had had an active claim at some time in the three years leading up to their program(s) participation — or in the case of a maternal/adoption claim, in the five years leading up to their program(s) participation. Near reachbacks are participants whose most recent EI claim ended between one week and six months prior to program participation, mid reachbacks' most recent EI claim ended between 6.1 months and 36 months prior to program participation and far reachbacks' most recent EI claim ended between 36 months and 60 months prior to program participation. The decision to divide the sample according to these three groups was more pragmatic than theoretical in nature. First, far reachbacks were already operationally defined as a distinct group in the LMDA (i.e., far reachbacks comprise only clients who were on maternity or parental leave) thus it made sense to treat them as such. Secondly, the use of the near and mid reachbacks groups represented a relatively even division of the remaining reachback clients based on the distribution of these clients in the population.
19	The functional sample factors out the attrition in the survey, leaving only the sample which resulted in completions, refusals, and those numbers attempted but not reached by the completion of fieldwork (e.g., retired phone numbers, respondents who were unavailable for the duration of the survey, respondents who were unable to participate due to illness or some other factor, etc.). Attrition includes numbers not in service, respondents who do not speak either English or French, respondents who indicated no knowledge of the topic and respondents who were still available to be contacted once a quota had been reached (thus rendering them not needed).
20	Although the response rate for the comparison group survey (34 per cent) is similar to that obtained for the participant survey (39.2 per cent), this response rate is generally considered to be quite adequate for comparison group surveys. Considering that comparison group respondents have little direct connection to the topic of interest (employment programs and services), it is more appropriate to compare this response rate to rates obtained from surveys of the general public, where a response rate of 30 per cent is considered satisfactory.
21	At issue is the ability to access the EI Account to cover the provision of PBPMs to Manitobans who are not EI clients. The use of funds from the EI Account for the provision of training and/or employment programs under the LMDA is clear — pursuant to the EI Act, funds from the EI Account are to be used to reimburse the Province for PBPMs accessed by "EI clients". Under the EI Act, an EI client is defined as an unemployed person requesting assistance who; (i) is an active EI claimant; or (ii) had a benefit period that ended within the previous three years; or (iii) had a benefit period within the last five years and was paid maternity/parental benefits, subsequently withdrew from the labour force and would like now to re-enter the labour force. However, it is important to note that the EI Act does not prevent the Province of Manitoba from using its own resources to fund PBPMs accessed by Manitobans in need of training and/or employment programs, but who are not EI clients.
22	See footnote # 21.
23	See footnote # 21.
24	This is acceptable as long as the funds from the EI Account are only used to reimburse the Province for PBPMs accessed by "EI clients".
25	See footnote # 21.
26	See footnote # 1.
27	See footnote # 21.
28	See footnote # 21.
29	See footnote # 21.
30	As noted in footnote # 11, all the appendices (A through E) are presented under separate cover.
31	Demographics include sex, age, education, equity group status, income and region.
32	As noted footnote # 11, all the appendices (A through E) are presented under separate cover.
33	As noted footnote # 11, all the appendices (A through E) are presented under separate cover.
34	As noted footnote # 11, all the appendices (A through E) are presented under separate cover (Formative Evaluation of the Canada/Manitoba Labour Market Development Agreement, Appendices, Final Overview Report), in which Annex 1 appears as Appendix D.
35	The variable used to measure job search intensity is the percentage of weeks looking for work in the post-intervention period while jobless. While job search is not an explicit expected outcome of PBPM participation, the extent to which participants are looking for work while jobless would be indicative of desire to enter the work force.
36	As noted in footnote # 21, the use of funds from the EI Account for the provision of training and/or employment programs under the LMDA is clear — pursuant to the EI Act, funds from the EI Account are to be used to reimburse the Province for PBPMs accessed by "EI clients". Under the EI Act, an EI client is defined as an unemployed person requesting assistance who; (iv) is an active EI claimant; or (v) had a benefit period that ended within the previous three years; or (vi) had a benefit period within the last five years and was paid maternity/parental benefits, subsequently withdrew from the labour force and would like now to re-enter the labour force. However, it is important to note that the EI Act does not prevent the Province of Manitoba from using its own resources to fund PBPMs accessed by Manitobans in need of training and/or employment programs, but who are not EI clients.
37	See footnotes # 21 and 36.
38	Annex 1 also appears under separate cover — Formative Evaluation of the Canada/Manitoba Labour Market Development Agreement, Appendices, Final Overview Report — as Appendix D, Impacts on Participants by Program Type — Descriptive Results.