Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Views on the degree of success of LMDA implementation with respect to the design and delivery of the Provincial Benefits and Provincial Measures (PBPMs) are presented in this chapter. These evaluation findings are drawn primarily from the key informant interviews and focus groups and supplemented with survey results.

4.1 Complementarity/Overlap of PBPMs with Other Programs

Most federal and provincial key informants as well as community partners and delivery agents perceived that the PBPMs may overlap and that there was a lack of coordination between:

• provincial programming for youth (defined as those aged 16 to 24) and federal youth programs (defined as those aged 16 to 29);
• PBPMs and federal programs for Aboriginal peoples; and,
• the Opportunities Fund for Persons with Disabilities and the Employability Assistance for Persons with Disabilities (EAPD) program.

Some provincial officials observed that there is also a lack of coordination between federal and provincial Labour Market Information (LMI) services. In addition provincial officials noted some administrative overlap in the treatment of apprentices by Employment and Training Services, the Apprenticeship Branch, the colleges and the federal EI program (e.g., overlap in the documentation of apprentices, reporting of absenteeism and follow-up on apprentices' cheques).

Although many respondents acknowledged that there is room to improve the coordination of programs (e.g., through partnerships), the consensus was that programs are mostly complementary.

4.2 Implementation of the LMDA

Key informants and focus group participants offered their views on aspects of the LMDA implementation that were most successful and on aspects that did not go as well. There is a high degree of similarity in the opinions expressed by the different respondent groups. On balance, it appears that the implementation of the LMDA has gone reasonably well, considering the complexity of the task and the effort involved, although some "growing pains" have been experienced and some problems remain to be resolved. The major observations on the most and least successful aspects of LMDA implementation are summarized in the remainder of this section.

Most Successful Aspects of LMDA Implementation

• Effective structure and process for LMDA implementation. The implementation of the LMDA was overseen by joint federal-provincial committees, including union representatives. These transition teams had responsibility for different areas, including human resources, operations, systems and finances. The Province developed a workplan and set two priorities to guide the implementation process: minimum disruption to staff and minimum disruption to clients. This structure and process worked well.
• Cooperation and compromise between federal and provincial partners. During the implementation process, there was a lot of dialogue, cooperation and compromise between the federal and provincial partners.
• Successful transfer of programs and responsibilities. Considering the commitment and effort required to implement an initiative as complex as the LMDA, all within a tight time frame, the transfer of responsibilities from HRDC to the Province went smoothly. The fact that the PBPMs are virtually identical to the former HRDC programs facilitated the transfer.
• Transfer of highly experienced and committed staff. Federal staff who were highly trained and experienced in the delivery of employment programs were transferred to the Province to deliver the PBPMs. The staff being transferred were treated with respect, and they received a financial incentive (i.e., signing bonus and severance pay) and recognition (i.e., a certificate and brief-case) from HRDC. Co-location has facilitated networking between federal and provincial staff. The cooperation and commitment of provincial staff members also made the transition go smoothly.
• Maintenance of good client service. For the most part, the transfer of responsibilities to the Province has been invisible to clients, and there has been minimal disruption to client service.

Least Successful Aspects of LMDA Implementation

• Limited time frame for negotiation of Agreement. Federal officials observed that LMDA negotiations were done very quickly. Also, from the union's perspective, staff and management on both the federal and provincial sides had insufficient input into the process because only those on the temporary negotiation team were involved. In addition, Employment Centre staff noted that, because the LMDA was "fast tracked", the required resources, systems, policies and procedures were not in place for the transition. These shortcomings caused problems, confusion and frustration for staff.
• Lack of consultation on PBPM design with community partners. Many community partners felt that they should have been consulted on the initial planning and design of the PBPMs. They believed that this would have helped the programs to better focus on client needs and that problems with design and implementation would have been avoided.
• Reduced program flexibility at local and regional levels. Many key informants and focus group participants perceived that the PBPMs do not have enough flexibility because of the Province's decision-making process (approval of expenditures) and also because the program eligibility criteria pursuant to the EI Act exclude many individuals in need of training and employment programs from participating in all the PBPMs²⁵.
• Inadequate information and monitoring systems. The majority of respondents identified inadequacies in the federal information and monitoring systems. Available measures are of limited use for the LMDA, there are errors in the data, and the systems are inadequate for planning, financial management and evaluation of the PBPMs. Moreover, provincial officials felt that staff were not properly prepared or trained in the use of these systems and that insufficient staff years and resources were provided for administrative support. The Province is currently developing a new information system.
• Difficult adjustment to new organizational culture and co-located sites. Former HRDC staff have had a difficult time adjusting to the provincial government culture which, from their point of view, involves having less power and decision-making authority, fewer resources, and a more centralized, bureaucratic organization. In addition, staff at co-located Employment Centres have had to sort out responsibilities for the reception function and how to share resources, space and information.
• Internal communications within the provincial government. Staff perceived that internal communications within the provincial government have been insufficient with respect to guidelines for delivering the PBPMs, human resource issues (discussed below), and the future of the LMDA. The lack of communications has left many staff feeling confused and anxious, and many perceive that program delivery is inconsistent.
• Human resource issues. Staff who were transferred to the Province expressed a number of concerns about human resource issues. In particular, they felt unfairly treated because their seniority with HRDC was not recognized by the Province and they were dissatisfied with how they have been re-classified in the provincial government (which is exacerbated by the perception that staff who remained with HRDC have been re-classified in a far more favourable manner). Some staff felt the need for clarification of their new job roles and for training in the new programs. Moreover, as indicated above, many staff were anxious about their job security due to inadequate communications on the future of the LMDA.

4.3 Partnerships

HRDC officials observed that the planning and implementation of the PBPMs has been facilitated by a productive federal-provincial partnership. Similarly, provincial key informants observed that PBPM planning and implementation has been done in partnership, in particular with HRDC, but also to some degree with other provincial departments, municipal governments, industry associations, employers and community organizations. In their view, many of these partnerships had existed prior to the LMDA, and have been strengthened through the process of implementing the LMDA and associated programs. Staff also felt that implementation has been done in partnership, citing Employment Partnerships and Labour Market Partnerships as successful examples.

Provincial respondents did identify some weaknesses in the partnerships:

• There have been challenges in developing working relationships with First Nations and Métis organizations, which have their own agreements with the federal government, though the Province has been working on resolving issues and developing employment strategies in cooperation with Aboriginal groups.
• Community partners and delivery agents would have liked to have been consulted more regarding the planning and implementation of the PBPMs, but this has been difficult because these organizations would want to put pressure on the Province to loosen the EI eligibility criteria— criteria that stem from the federal EI Act and hence are not under provincial jurisdiction.
• The involvement of EI client groups²⁶ and special interest groups in the design and delivery of programs has been limited.
• Some staff have experienced barriers in the development of partnerships, such as a lack of organizations/delivery agents in the region with which to form partnerships.

Most community partners and delivery agents indicated that they were not adequately consulted regarding the initial planning and implementation of the PBPMs. Still, they have generally found provincial government staff, such as project officers, to be very helpful and cooperative (e.g., in sharing information with community organizations and providing guidance for the development of projects relevant to the labour market). Cooperation has been good in Labour Market Partnerships and Employment Assistance Services, though a lack of support services like daycare for clients taking training was identified as a weakness. In addition, partnerships among various community organizations and delivery agents have a long and successful history, as this is a core feature of their service delivery approach.

4.4 Flexibility in Implementing PBPMs

Mixed views were expressed regarding the degree to which the PBPMs are sufficiently flexible to allow program decision-making and responsiveness at the provincial, regional and local levels. Federal officials observed that, because the Province is more centralized in its decision-making than HRDC used to be, some flexibility has been lost at the regional and local levels. For instance, federal officials perceive that third-party delivery arrangements under Employment Assistance Services are more restrictive; that local programs are somewhat less timely and responsive to client needs; and, that there are delays in project approvals. It was suggested that, on balance, the PBPMs offer more flexibility than the former provincial approach, but less flexibility than the former HRDC approach.

Provincial managers explained that they attempt to strike a balance between decentralized delivery to allow some local and regional flexibility, and centralized control to ensure that provincial priorities are met and that a consistent quality of service is provided across the province. Staff generally felt that the Province's centralized decision-making limits the regional and local flexibility of programs. In addition, some staff identified unique barriers to the delivery of flexible, responsive programming: difficulties in providing programs for small communities; and, administrative hurdles such as limited latitude to use various sources of funding and an inability to contract on Aboriginal reserves.

While community partners held mixed views about program flexibility, the opinions of third-party delivery agents were more consistent. The majority of delivery agents perceived that local-level flexibility in the delivery of PBPMs is impeded by both the Province's centralized decision-making approach and the restrictive EI eligibility criteria²⁷ . Delivery agents complained that the short-term contracts with the Province do not allow them to plan their service delivery appropriately or to keep experienced staff. Furthermore, there is a perception that many clients in need who do not meet the EI eligibility criteria are not being adequately served by the LMDA benefits and measures²⁸. These clients include youth and people with disabilities who have a weak attachment to the labour market, chronically unemployed people and marginally-employed people (i.e., those working in low skill jobs) who need skills upgrading. In addition, community partners indicated that clients in remote communities are not receiving adequate educational and training services (due to the lack of infrastructure and programs) and clients with multiple employment barriers do not have access to long-term programming. For many partners and delivery agents, the services under the LMDA are perceived as being driven more by program eligibility guidelines²⁹ than by client needs. Still, some delivery agents noted that they are able to serve non-EI eligible clients through other program vehicles and sources of funding.

4.5 Adequacy of Information and Monitoring Systems

The information and monitoring systems for the LMDA and PBPMs are a source of concern. Provincial government officials expressed dissatisfaction with the available federal systems for monitoring the PBPMs, third-party delivery and program participants, explaining that there are errors in the data, the available measures are of limited use for the current application, and the systems are inadequate for planning, financial management and evaluation of the PBPMs. Similarly, staff had little confidence in the current monitoring of third-party delivery agents and participants, and argued that the current systems do not adequately capture the work they are doing with the PBPMs and the range of client outcomes/impacts. Third-party delivery agents echoed these concerns and added that the Contact IV system entails excessive monitoring requirements, that some data fields are unclear to them and of limited use (e.g., "complete"), and that they have not been provided with adequate measurement tools or a budget for monitoring clients.

Federal officials acknowledged that there are problems with data integrity, and noted that this is a national problem. In order to address the systems issues, the Province is currently developing a new monitoring system — tailor made for the LMDA — that will replace the existing system and will have the capacity to provide better accountability and management information.

4.6 Adequacy of Labour Market Information and Labour Exchange

Federal and provincial officials agreed that Labour Market Information (LMI) is accessible to clients, employers and service providers via the Internet, though this is not useful for those who are not computer literate. In addition, federal LMI units, which provide labour market analysis and information products, are located in Winnipeg, Brandon and other centres. These federal units have recently hired more LMI staff. The Province also produces LMI. Currently, there is a federal-provincial working committee on LMI, and there are plans for the development of joint LMI research and products.

Although LMI is accessible, many provincial managers, staff, community partners, delivery agents and employers were dissatisfied with the usefulness of the currently available information. The major complaints about HRDC's information were that it is not current, national-level information is not relevant and specific enough for small regions and centres in Manitoba, and accurate forecasts of supply and demand in specific industry sectors in Manitoba are unavailable. A need to improve the provincial LMI system was also noted by some key informants. In fact, many respondents suggested that the best way to obtain accurate labour market information is simply to ask local employers if they are hiring and what types of skills they need.

In the survey of participants, only 25 per cent of respondents indicated that they had reviewed written material on the labour market on their own. However, a large portion of LMI is also available in electronic format and on the Internet. The evaluation found that of the proportion of respondents who reported having accessed a computer, Internet job postings or written LMI, 54 per cent had used one of these three resources. Very few clients consulted in focus groups indicated having used LMI, and among those who had, the majority were dissatisfied with the usefulness and relevance of the information to their needs. Many clients wished that staff had provided them with useful guidance on the labour market.

Regarding the Labour Exchange, federal officials as well as third-party delivery agents observed that job listings in the Job Bank and on the Internet are frequently out of date, which frustrates clients. Very few clients in the survey (21 per cent) indicated viewing job listings on the Internet, though roughly half (52 per cent) had used the Job Bank kiosk.

4.7 Strengths and Weaknesses of the PBPM Delivery Structure

The service delivery structure for the PBPMs includes four Employment Centres and the Employment Partnership office in Winnipeg and 12 Centres in smaller communities throughout the province, all of which are co-located except the Employment Centre in Brandon. In addition, for some PBPMs (i.e., Employment Assistance Services, Employment Partnerships and Self-Employment), programming is delivered through third-party service providers who are under contract with ETS. In the key informant interviews and focus groups, a number of strengths and weaknesses of the delivery structure for the PBPMs were identified. The strengths included:

• The delivery of the programs through the co-located Employment Centres was regarded as a key strength because it offers single-window service to clients and facilitates coordination and networking between provincial and federal staff sharing the Centres.
• The skill and commitment of staff at Employment Centres facilitate program delivery and client service.
• The fact that PBPMs are being delivered in partnership with industry, employers, educational/training institutions and other community organizations was viewed as a strength because it helps to ensure the delivery of training and employment programming that is relevant to the labour market.
• Using the services of third-party delivery agents (who have significant expertise and sensitivity in their area of programming) was thought by many respondents to be a cost-effective delivery approach.

Respondents identified the following weaknesses and areas for improvement:

• A major issue raised by many key informants and focus group participants pertains to a perceived lack of communication, guidelines and policies for the delivery of the PBPMs. This is a multi-faceted issue incorporating:
  • lack of communication from the Province to community partners and delivery agents regarding future directions for the PBPMs; and,
  • a lack of clear staff guidelines for the delivery of PBPMs and for third-party contracting (including the lack of an expert "program consultant" on staff, who can answer questions on delivery guidelines, etc.). This lack of clear guidelines causes inconsistent delivery; and inadequate outreach and promotion of the PBPMs to clients and community groups.
• Linked to the above first point, many respondents suggested that there is a need for some clarification of roles and responsibilities. This would include the specification of federal and provincial responsibilities for the reception function at co-located Employment Centres, clarifying federal and provincial roles for programming for youth, persons with disabilities and Aboriginal peoples, and clarifying the role of the communities in the delivery of services.
• The short-term contractual basis for delivery of services by third-party delivery agents makes it difficult to administer the programs.
• In some cases, excessive administrative requirements (e.g., for agents/partners delivering training and for those dealing with apprentices) were identified as a barrier to efficient service delivery.
• Some community partners suggested that there is a need to develop more relevant guidelines for the administration of some programs and to increase flexibility when applying program guidelines.
• Limited access to programs and services in remote areas of the province and poor access for persons with disabilities (who have special needs) and for non-EI eligible clients were identified as problems.
• Some staff suggested that there is a need for more human resources and for private office space (for interviewing and counselling clients).

4.8 Impact of Co-Location on PBPM Delivery

Most federal and provincial officials as well as community partners believed that co-location will be beneficial for client service (e.g., the provision of seamless, "one-stop shopping") and that relations between federal and provincial staff have been generally good, despite some "growing pains". There does not appear to be any difference in the quality of federal-provincial relations at co-located sites as compared to the one site not co-located (Brandon), except that a little more effort needs to be devoted to communications at the latter.

In the focus group discussions, some employers and clients were able to provide some observations on a co-located employment centre. The key beneficial impacts of co-location were perceived to be improved access to services, quicker and more efficient service, and more convenient service because everything is located in a single office. Clients did appear to have some initial difficulties finding the appropriate staff person to serve them.

4.9 Accessibility of PBPMs in Both Official Languages

The PBPMs are quite accessible to clients in both official languages, though there appears to be a minor problem with the availability of program information in French at the Winnipeg Employment Centre. In the survey of participants, 79 per cent of respondents indicated being satisfied with the accessibility of programs and services in general, and no significant differences as a function of clients' first language were observed. In addition, only three per cent of respondents indicated that they were unable to obtain information about employment programs in their preferred language (which was French for over half of these clients).

One possible explanation of the problem with the availability of program information in French was mentioned in the focus groups. Employment Centre staff in Winnipeg noted that, due to a lack of office space at present, there is not always enough room to display the government program literature in French. In addition, francophone clients indicated that it can be difficult to obtain services in French at the Winnipeg Employment Centre, though French services are readily available at the St. Boniface Centre.

4.10 Satisfaction with the PBPMs

PBPM participant survey respondents were asked to rate their satisfaction with various aspects of the programs and services they received. As shown in Exhibit 4.1, respondents were satisfied (responded with a 5, 6 or 7 on a 7-point scale) with most aspects of the programs and services they received and were most likely to be satisfied with the accessibility of the programs and services they received (78 per cent) and the comprehensiveness of the information about the programs and services they required (78 per cent). Respondents were slightly less likely to be satisfied with outcomes they achieved as a result of the services or programs (70 per cent).

Analyses were conducted to determine whether there was a relationship between clients' rated satisfaction with various aspects of the programs and services they received and the actual employment outcomes that they experienced. Satisfaction ratings were compared for participants who did and who did not report positive outcomes on four different employment outcome measures: 12 consecutive weeks of employment, weekly earnings, current employment status and type of current employment (i.e., year-round, seasonal or contract/casual). These analyses revealed that participants' employment outcomes tended to be inversely related to their level of satisfaction with different aspects of the programs and services.

Most respondents indicated that they were pleased with the services they received at the Employment Centre and a few indicated that their understanding of their needs changed as a result of speaking with a counsellor or other staff. The majority, however, noted that their understanding of their needs had stayed the same. Clients are generally satisfied with the PBPMs being offered and the areas of particularly high satisfaction included: accessibility of PBPMs; the expertise of staff; timeliness of services and self-service resources (e.g., Internet and computer access); quality of programs and services, particularly EAS; and financial support received whether EI or funds for training. There were a few negative comments expressed but the criticism generally related to the delivery structure or the guidance received rather than to the actual benefits and measures.

Program-specific opinions included:

• Most Enhanced Fee Payer clients felt that the training they received prepared them for the job market and they expected a positive outcome in terms of employment. A number of clients felt that the programs were not up to date or too short and did not provide them with the skills necessary to be job ready or to access higher-paying jobs.
• Overall, most participants who had been involved in some kind of employment or work placement programs, whether through an apprenticeship or with the help of a wage subsidy, were satisfied with the experience they gained and felt it was a positive step towards upgrading themselves. Wages were an issue for some participants in a wage subsidy placement who indicated that the salary was insufficient and/or they were overqualified for their particular job.
• Many focus group participants who obtained Employment Assistance Services were very satisfied with the services offered by third-party delivery agents and felt these were most useful in providing them with direction and helping them find employment. Participants stated that the services received are different from and complementary to those offered at the Employment Centres. Participants indicated they received individual counselling, employability skills, basic computer skills, job finding skills and help writing resumes.

4.11 Use of Self-Serve Resources

LMDA program participants were asked to indicate what resources or services they used on their own at an employment resource centre. This section presents the findings for all PBPM participants. As shown in Exhibit 4.2, respondents were most likely to indicate using job bank kiosks or job board listings (52 per cent) or a computer (34 per cent) on their own. Respondents were least likely to have used Internet job listings (24 per cent) and to have indicated they did not use any resources on their own at an employment resource centre (20 per cent).

Sub-group differences by program type revealed that EAS, SLG/Enhanced Fee Payer and Employment Partnerships respondents were more likely to report having used different resources and services on their own, whereas Apprenticeship participants were less likely to have used any of the resources on their own (which would be expected given the nature of the Apprenticeship program).

Respondents were most likely to have first learned about the employment services they used on their own through word of mouth (29 per cent) followed by a third party (17 per cent) and an advertisement (15 per cent). Roughly one in five respondents (19 per cent) indicated they knew of these services all along (Exhibit 4.3). Wage Subsidies and SLG/Enhanced Fee Payer respondents were more likely to report having first heard of the employment programs through word of mouth, whereas EAS respondents were less likely to have first heard of the programs through word of mouth. Advertising seems to have reached EAS and PBT clients more often and was reported less often by Wage Subsidies clients and Apprentices. EAS participants reported a referral from a third party more often, and Employment Partnership participants were more likely to report having first heard of these services from employment resource centre staff.

Respondents (active EI claimants) reported that they found out about the employment program in which they participated an average of 4.7 weeks after they established their most recent EI claim. On average, college educated and married respondents reported that it had taken them longer to find out about the program after the establishment of their most recent claim (8.4 and 6.4 weeks, respectively). This period was shorter for single respondents (2.9 weeks) and those with a high school education or less (2.9 weeks).

4.12 Suggestions for Improvement

In the focus groups, participants were asked if they had any suggestions for improving the design and delivery of the PBPMs. Many of their concerns have already been raised in earlier sections of this chapter. Their major suggestions are summarized below.

• Employment Centre Staff: As already noted, many staff felt the need for improved internal communications/guidelines, training and clarification of their job responsibilities for the delivery of the PBPMs. They also suggested that better communications between federal and provincial program officials would help to provide a check on potential abuse of program funds (i.e., inappropriate use of similar programs offered by both the federal and provincial governments). In addition, in order to improve the local relevance of programming, staff indicated a need for local/regional strategic planning, for more staff input into program design, and for more flexibility to adapt program guidelines to local and regional priorities. Related to this, some staff suggested that more flexibility to extend the period of funding would improve service delivery in some cases. For instance, this would allow a longer period of subsidized job experience for clients who need this following training and would enable community delivery organizations to develop their service capability and clientele. Finally, some staff felt the need for more financial and human resources to support PBPM delivery.
• Third-Party Delivery Agents: Like the Employment Centre staff, delivery agents suggested that program delivery would be improved with more flexibility at the local level and with quicker, more decentralized decision-making regarding program funding. In addition, they indicated a need for: longer-term contracts (e.g., three years rather than one year) so they can plan properly and retain skilled staff; more government consultation with delivery agents and opportunities for agents to share best practices; and more funding for professional development, both for themselves and for Project Officers at Employment Centres. Finally, some delivery agents suggested that allowing some funds for "purchase of training" or "project-based training" would help to serve the needs of clients with special needs (e.g., people with disabilities) and those in rural and remote communities where there is limited access to educational/training programs. However, it should be noted that "purchase of training" and "project-based training" are no longer permitted under the EI Act. These activities were terminated nationally and replaced with SLG under the LMDA.
• Community Partners: Included in the suggestions of this group was the need for: more access to training for marginally-employed people, clients with special needs (e.g., persons with disabilities) and those in rural and remote areas; longer periods of job placements/wage subsidies; longer-term follow-up with clients to increase the success rate of programs; training that is better targeted to the needs of the labour market; and, reduced paperwork associated with the PBPMs.
• Employers: Similar to the community partners, some employers called for a better matching of educational and training programs with the labour market and for an extension of the duration of wage subsidies. Moreover, they indicated a need for better screening of clients (by counsellors) being considered for wage subsidies and for the release of more client background information to employers so that a better employer-client match can be achieved in the Wage Subsidies. They also noted that the PBPMs need to be better promoted to employers. Finally, some employers emphasized the importance of pre-employment programs and for more in-depth counselling and coaching for clients who need this to be job ready.
• Clients: PBPM participants indicated a need for better service from Employment Centre staff, including more coaching, guidance and counselling for clients who need the assistance to increase their self-esteem, improve their job search skills, learn how to use the Job Bank, and learn about the labour market. Some participants also called for longer-term financial assistance, better promotion of available programs and services, more and better training opportunities, better labour market information, and more services for persons with disabilities and older workers.

Footnotes

25	See footnote # 21.
26	See footnote # 1.
27	See footnote # 21.
28	See footnote # 21.
29	See footnote # 21.