Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The formative evaluation concentrates on process and implementation. Any comments on results and outcomes are preliminary and require confirmation by the next phase, where client follow-up surveys and non-participant data will allow us to infer program incrementality.

These observations are organized by evaluation theme.

6.1 Rationale

Fundamental to an understanding of the Canada-Saskatchewan Labour Market Development Agreement is appreciating that Employment Insurance (EI) Part 2 programming needed to be integrated into a pre-existing provincial labour market development program. The CS-LMDA is an example of a devolution process, as opposed to Human Resources Development Canada (HRDC) programs and services that are co-managed by the federal and provincial government.

Coinciding with the LMDA process, the province created the Saskatchewan Training Strategy (STS). In many ways, this strategy aligns with current thinking on optimal processes for training unemployed and economically disadvantaged persons to support their growth to self-sufficiency.⁶⁴

Central to the STS and its successor, the Sector Strategic Plan, are the following concepts:

• The publicly funded post-secondary education and training sector is pivotal to providing accessible, high quality training programs in Saskatchewan. An array of community-based organizations, private vocational schools, and private trainers augment the programs and services available to Saskatchewan adults through the publicly funded post-secondary institutions. All parts of the system should be supported to help ensure adequate capacity within it as well as quality and access to relevant labour market related training programs and services.
• The post-secondary education system must provide programs and services that are capable of addressing the needs of clients who may receive income support from a variety of sources including Employment Insurance, social assistance, student loans as well as their own resources. Saskatchewan is committed to designing programs around labour market and community need rather than income source.
• To support its move to an approach that links clients and funding sources with programs and services aimed at supporting labour market development and attachment, Saskatchewan embarked on a comprehensive information and tracking system known as the One Client Service Model (OCSM). This system is intended to offer an integrated capacity to record and share information about individual and organizational program and service provision.
• The final concept that guides the Saskatchewan approach to an integrated post-secondary education and training sector is an emphasis on regional planning underpinned by a sectoral approach to labour market analysis. Throughout the province there are venues that bring together business/industry, government, educational institutions, and community-based organizations to identify emerging labour market issues, assess the current array of programs and services available to meet these, and help plan actions that will enhance the programs and supports available to Saskatchewan adults.

The essential implication of the Saskatchewan approach for EI clients is that, in principle, they are to access the same portfolio of courses and supports open to any unemployed person in Saskatchewan. As the CS-LMDA evolved, some important differences were evident, and special programming has been implemented to serve EI clients. Most notable is the Skills Training Benefit, which offers educational financial support and in some cases an income supplement for EI clients.

The first year-and-a-half of the CS-LMDA has seen the two orders of government each endeavouring to fulfill the objectives of the EI Act and the Saskatchewan Sector Strategies Plan, which succeeded the STS. The process of negotiation and discussion is a process of searching for the common ground to allow the federal government to meet the requirements of the EI Act, and the province to integrate EI clients into a comprehensive labour market training initiative designed to serve all unemployed and underemployed residents.

An important goal for the evaluation of the CS-LMDA, particularly the summative evaluation, is to assess whether EI clients have been effectively trained, and whether they are returning to work. By comparing the experience of those who use programming under the province's approach and those who do not, the incremental benefit to the client can be measured.

6.2 Design, delivery, planning, and implementation

In general, despite the extended process of negotiating the Agreement, the design, delivery, planning and implementation has resulted in an effective system.

• Provincial programs and services reflect the intent of the CS-LMDA. A key issue in the early stages of the CS-LMDA was to ensure that provincial programs and services could be assessed as comparable to employment benefits and support measures (EBSM) offered in the pre-LMDA era. Federal and provincial officials are satisfied that this match now exists, especially with the creation of the Skills Training Benefit.
• Overlap and duplication of PPSs are minimal. Three areas exist where federal and provincial programming purposely overlap: programming for youth, persons with disabilities, and Aboriginal persons. Federal and provincial officials believe that this apparent overlap increases service to these clients.
• Partnerships with industry have continued to be strengthened under the CS-LMDA. Some provincial managers attribute this to the pre-existing sector partnerships program, which was part of the STS.
• Eleven clients reported requested services in French, and nine stated they were satisfied with services received. Insufficient recorded use of these services exists to support a conclusion on their adequacy.
• Federal managers reported that the province was slow to formally recognize the federal government's contribution to the CS-LMDA. Officials from both orders of government acknowledge that progress is evident and both federal and provincial contributions are recognized in program literature, and in the name given to Canada-Saskatchewan Career and Employment Services (CSCES) offices.
• The provincial information system needs further development. OCSM needs to be developed in order to support program monitoring and performance measurement. Development appears to be behind schedule, and information from various program sources has not been fully integrated. Reconciling existing databases and adopting standards across programs (e.g., type of program, outcome codes, dates for program start and end) are urgently needed to improve the system.
• Information sharing protocols between federal and provincial partners need to be enhanced. The inability to share information properly was criticized by both federal and provincial officials we interviewed. The process that provincial employees use to identify EI eligible clients is inefficient, and compromises effective program delivery. Without access to historical records (EI eligibility and training history), those responsible for counselling EI clients will be unable to prepare timely and complete training plans. As institutional memory fades, CSCES staff will have a more difficult time going through informal channels to access information about clients directly from HRDC staff.

6.3 Success/impacts/effects (Client satisfaction and preliminary outcomes

Some impacts can be discerned based on the client and employer surveys:

• Clients are satisfied with services provided at CSCES offices. Most who have used them have found them to be helpful - particularly career counselling, multimedia products for resume writing, and computer services for searching the Internet. Based on the client survey, there appears to be minimal variation in the use of CSCES services across regions. Those clients who have used services in both the pre-LMDA period and since the LMDA was signed report that services are the same or better. However, for the most part, clients are still using the same types of programs under the LMDA as under the federal system. They also report similar levels of income support; many remain on Part 1 EI benefits while in training.
• We also asked employers whether they believe that the CS-LMDA has improved training outcomes. However, employers find it difficult to distinguish between the former federal and current provincial training programs. Some feel that more programs are now available, and that wage and training subsidies have encouraged them to hire EI clients. Many employers who participate in a wage subsidy program or use the job order process to find employees are unaware that they have hired a former EI client. As such, they are unable to evaluate the current provincial programs with respect to serving EI clients. Due to the difficulty of "flagging" an EI client, the analysis in the formative evaluation could not "separate" these trainees and compare their experience with other non-EI clients.
• Service providers are aware of the changes brought on by the LMDA in terms of the use of community-based organizations. Some service providers report that they now receive some core funding, in addition to project-based funding. This has enabled some to plan for the long term, rather than providing services on a project-by-project basis. Others are concerned that the emphasis on using contract training has lessened. Some find it difficult to train Social Assistance (SA), EI, and non-income support students in a mixed class. Differences in income support regulations raise questions about equity among students.
• An intended outcome of the CS-LMDA is improved planning. Regional needs assessments and Sector Partnerships are easily identified processes, but these pre-date the agreement. Therefore, it is difficult to conclude on the extent to which the CS-LMDA has supported planning processes. Although these planning processes can have an impact on EI clients, they are not typically identified as a target group that needs further scrutiny. An important change resulting from the LMDA has been the shift in authority for planning to the recently created Regional Services Branch, which now controls the budget and has the decision-making authority.

6.4 Program impacts / effects are preliminary

Many questions posed in the evaluation framework regarding success, impacts, and effects require information from HRDC on all EI clients, which will be available for the summative evaluation. Program impacts are addressed in a preliminary and tentative way through client survey, focus groups, and dyads.

• Based on the client survey (n=1,250), about one-third of participants said they were required to prepare a case/action plan prior to beginning training. A similar proportion of clients completed case/action plans in each region. Of those who talked to a counsellor about a back-to-work plan (n=429), 28 percent had completed such a plan, 53 percent were part way through, while 18 percent stopped part way through or never started the steps. Based on the client and focus group responses, back-to-work plans (case/action plans) appear to be an effective and useful part of the training process.
• Clients that have used provincial programs and services express high levels of satisfaction. Although room for improvement certainly exists, clients rated CSCES centres highly. Clients expressed less support for the web sites (federal and provincial), largely because they saw them as often being out of date. Word-of-mouth and networking, particularly in rural areas, appear to be the most common way that clients reported getting work. This leads many clients to deny that the provincial programs and services helped them find employment.
• Employers are generally satisfied with programs targeted to EI clients. Programs have helped a majority of employers to "fill job vacancies" (63 percent), "fill skill shortages" (50 percent), and "meet human resource needs" (54 percent).
• Dyads revealed the sacrifices made by five families that had used provincial programs and services. The separation of the trainee from his or her family was the most important hardship. The role of a supportive spouse in completing successful training also emerged as a common theme in the dyads. All participants and their families strongly endorsed the value of the programming.

6.5 Other issues: the critical role of a comparison group and improved databases

The most important deficiency in the CS-LMDA is the weak information system to support both the provision of services to EI clients and the conduct of evaluations. The province is keenly aware of the need to improve OCSM.

Improving information in OCSM also supports the creation of intervention profiles for EI clients of provincial programs and services. This includes removing duplicate entries, ensuring that STB program interactions are recorded, and ensuring that information from other sources such as SIAST is recorded. For example, with complete client information, the survey can reference those provincial services that are part of the CS-LMDA and not other training interventions not funded by EI (Part 2). Further, the more detail available on the types of interventions used by participants, the more precise the estimate of incremental benefit arising from PPSs offered under the LMDA.

Footnotes

64	See Lessons Learned on Welfare Reform: Labour Market Perspectives (Prairie Research Associates (PRA) Inc., March 1999) and Lessons Learned on Aboriginal Social and Economic Development (Prairie Research Associates (PRA) Inc., December 1998) for reviews of these recent approaches.