Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Introduction and Background

Under the terms of the Canada/Manitoba Labour Market Development Agreement (LMDA), the federal and provincial governments agreed that Manitoba would assume responsibility for the design and delivery of active labour market programs and services for the unemployed in the province. This report presents the results of the formative evaluation of Provincial Benefits and Provincial Measures (PBPMs) under the terms of the Canada/Manitoba LMDA. The Canada/Manitoba LMDA was signed on April 17, 1997, and came into force on November 27, 1997.

Evaluation Methodology

Multiple methods were used to collect data for this formative evaluation. Quantitative methods, specifically a survey of 1,393 program participants and a survey of 500 non-participants, were used to obtain information that could be applied to the full population of participants in the PBPMs. As well, qualitative methods consisting of 28 key informant interviews and 12 focus groups were employed to gather in-depth perspectives on the research issues.

The total survey sample included those clients who had participated in an LMDA program or service between November 27, 1997 and August 15, 1999. Qualitative data collection was undertaken in December 1999 and the survey was conducted in January 2000.

Key Findings

Implementation

The implementation of the Canada/Manitoba LMDA has gone well. However, there have been some "growing pains" and there are some outstanding issues to be addressed.

The PBPMs reflect the guidelines, principles and intent of the Employment Insurance (EI) Act and the LMDA. They are consistent with Human Resources Development Canada's (HRDC's) priorities but only in part with Manitoba's priorities. The eligibility criteria of the EI Act limiting access to EI clients does not permit the Province to provide training to the marginally-employed using LMDA funds. This is a major concern for Manitoba (and employers) as it is entering times of skills shortages, however, this is a broader issue that is outside of the realm of the delivery of the PBPMs.

The PBPMs were designed and implemented in partnership with stakeholders, in particular between Manitoba Education and Training and Human Resources Development Canada, but also to some extent with other provincial departments, municipal governments, industry associations, employers and community organizations. Many of these partnerships existed prior to the LMDA and have been strengthened with the implementation of the PBPMs.

The key strengths of implementation and delivery have been: an effective implementation structure and process with good cooperation between federal and provincial partners; the successful transfer of programs and highly experienced staff to the Province with minimal disruption to client service; partnerships with industry, employers, and educational and other organizations that have helped to ensure the relevance of programming to the regional labour market; and the delivery of programs at co-located Employment Centres featuring "one-stop shopping" for clients, with cost-effective support from third-party delivery agents.

PBPMs are harmonized with other provincial and federal initiatives. Although, there is some perceived overlap or lack of coordination between provincial and federal programs for youth, Aboriginal peoples, persons with disabilities and labour market information, the consensus is that programs are mostly complementary.

There are a number of perceived weaknesses identified including the need for more flexibility in the design and delivery of PBPMs so that they are better adapted to local and regional needs; more useful labour market information; better access to programs and services in rural and remote communities; more consultation with and better promotion of the PBPMs to community groups; and better internal communications.

In addition, some staff found program guidelines too general to facilitate consistent application among individual staff and between employment centres. As well, human resource issues were also noted, including the desire for a resolution of concerns related to the job classification of staff transferred from HRDC, clarification of staff roles and responsibilities and alleviation of staff anxiety over the future of the LMDA and their job security.

Clients

The survey evidence indicates that participants in the PBPMs are representative overall of EI clients in Manitoba (e.g., in terms of equity group status and other socio-demographic characteristics). PBPMS are largely relevant to the needs of clients in Manitoba. Most clients indicated that the PBPMs met their needs and expectations and impacted positively on their lives. These clients noted that the PBPMs helped them gain self-confidence, gave them a sense of direction and increased their ability to get a job.

Clients expressed high satisfaction with the timeliness and accessibility of services, self-serve resources, and the quality of programs and services.

The few areas about which clients expressed dissatisfaction involved aspects of service delivery rather than the PBPMs themselves and included the lack of up-to-date training courses as well as insufficient remuneration for Wage Subsidies and Apprenticeship participants. As well, 78% of Enhanced Fee Payers indicated that the requirement to contribute to the cost of training made it difficult (to a moderate or large extent) to access training.

With respect to accessibility to programs and services in the official language of their choice, evidence indicates that the PBPMs are easily accessible to most clients in their preferred official language.

Staff, clients, community partners and third-party agents all agreed that PBPMs have a positive impact on the lives of participants. Most staff also stated they are satisfied that the programs and services offered meet the needs of EI clients. Among the reported impacts observed were improved confidence and self-esteem, life skills, job search skills, employment, job satisfaction, earnings, employability and less reliance on income support.

Employers and Communities

Most employers felt that the programs are relevant to their needs. Respondents expressed satisfaction with the degree to which programs suited their organizational goals as well as with the employees who worked for them through the program. Some dissatisfaction, primarily with the Wage Subsidies program, was expressed with respect to the lack of background information on job candidates, the match of participants to the employers' businesses and the length of programs (i.e., period of funding).

Some provincial officials and community partners felt that the idea that training can only be done through Skills Loans and Grants/Enhanced Fee Payer is a limitation because it does not take into account the needs of industry. For example, when new companies are considering relocating in the province, some respondents believe they can no longer have an industry-based approach and develop programs at the local community college to meet the skills needs of such new companies. This was previously accomplished using Project-Based Training and Purchase of Training which were phased out on July 1, 1999.

The majority of training deliverers (colleges, universities, institutions, etc.) felt that the phase out of Project-Based Training and Purchase of Training negatively impacts on the relevance of PBPMs. Some respondents perceived that there is limited recourse to address the training needs of a particular sector, industry or employer, and that there is less incentive for training institutions to partner with local industry in addressing labour market issues.

Most employers and community partners felt that the programs are relevant to the needs of their communities. They indicated, however, a need for: more access to training for marginally-employed people, clients with special needs and those in rural and remote areas; longer periods of job placements/wage subsidies and longer-term follow-up with clients to increase the success rate of programs; training that is better targeted to the needs of the labour market; and, reduced paperwork associated with the PBPMs.

Regional Findings

There are some unique challenges in the northern region of Manitoba, including the difficulty of providing programs in small communities. There is also a lack of relevant training being offered in these communities. Furthermore, the individual approach to training through the Skills Loans and Grants/Enhanced Fee Payer was also viewed by some as a limitation because of the lack of available training programs.

Adequacy of Information and Monitoring Systems

Respondents indicated that the administrative systems for monitoring and reporting on PBPMs, program participants and third-party delivery agents are inadequate for proper planning, management and evaluation of the PBPMs. The validity and usefulness of the current accountability measures as well as the integrity of the data are perceived to be problems.

Weaknesses were also observed in the supporting information systems used to assess the accountability results targets. Comparisons between employment indicators from the survey and the return to work indicator in the administrative data indicated that a large number of actual returns to work were not captured in the information systems. This suggests that the accountability results for Manitoba may under represent actual success with respect to returns to work and unpaid EI benefits.

Labour Market Information/Labour Exchange

Labour Market Information (LMI) is readily available to staff, community partners, employers and clients via the Internet, although this medium is not accessible to those who are not computer literate. Available LMI is widely criticized, however, for being outdated and limited in relevance to client needs and to small regions and communities in Manitoba. In recognition of this problem a federal-provincial working committee on LMI has been formed and is planning to develop joint LMI research and products.

With respect to the Labour Exchange, the conclusions are similar to those for LMI, i.e., although job listings are easily accessible via the Internet and Job Bank, the information is often out of date and hence of limited use.

Service Delivery Model

With respect to the service delivery model, many respondents indicated that they feel there is a need for more flexibility in order to be able to address the needs of all the unemployed population of Manitoba. Although the PBPMs allow some degree of flexibility in program decision-making and responsiveness at the provincial, sub-provincial and local levels, some staff perceive that decision making (approval of expenditures) is more centralized than it had been prior to the LMDA and this can cause delays in the implementation of programs. The eligibility criteria stemming from the EI Act are also thought to limit flexibility in programming.

Third-party delivery agents expressed concern as well about the short-term contracts they are given for the delivery of employment services. They feel that these short contracts (of one year or less) limit their ability to do proper long-range planning and to develop and retain skilled employees.

Community partners stated a preference for service delivery models that rely on partnerships between the government and the community which included: sector-based models, where programs are designed and implemented in consultation with sector agencies made up of employers, branches of government, schools, and private training institutions; and third-party models, through which programs are designed and delivered through community and third-party organizations that possess greater expertise in delivering services to different client groups.

The evidence suggests that co-location has been beneficial for client service as well as for working relationships and information sharing between federal and provincial staff.

The major perceived strengths of the PBPM delivery structure include:

• single-window service for clients through the co-located Employment Centres;
• accessibility of the PBPMs in both official languages;
• the skill and commitment of front-line delivery staff at the Centres;
• the partnership approach to PBPM delivery that helps to ensure the relevance of programs; and,
• the cost-effective use of third-party delivery agents.

The major perceived weaknesses of delivery are:

• poor communications and a lack of program delivery guidelines for staff; inadequate promotion of the PBPMs to clients and community groups;
• some confusion about federal versus provincial roles and responsibilities for the reception function at co-located Centres and for programming for youth, Aboriginal peoples and persons with disabilities; and,
• a lack of training opportunities for clients in remote areas of Manitoba, for persons with disabilities and other special interest groups.