Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Formative Evaluation of the Canada-Saskatchewan Labour Market Development Agreement

Evaluation and Data Development
Strategic Policy
Human Resources Development Canada

Prepared for:
Human Resources Development Canada and
Post Secondary Education and Skills Training (Saskatchewan)

December 2001

SP-AH187-12-01-E
(également disponible en français)

List of Acronyms

Full name	Acronym
Training Institutions
Saskatchewan Institute of Applied Science and Technology	SIAST
Regional colleges	RC
Community-based organization	CBO
Saskatchewan Indian Institute of Technologies	SIIT
Government Training Programs/Income Support
Skills Training Benefit	STB
Provincial Training Allowance	PTA
Social Assistance	SA
Employment Insurance	EI
Government Departments/Organizations
Human Resources Development Canada	HRDC
Post-Secondary Education and Skills Training	PSEST
Canada-Saskatchewan Career and Employment Services	CSCES
Database
One Client Service Model	OCSM
Career and Employment Information System	CEIS
Other Terms
Canada-Saskatchewan Labour Market Development Agreement	CS-LMDA
Employment Benefits and Support Measures (see PPS)	EBSM
Labour Market Information	LMI
Labour Market Exchange	LEX
On-line information system	OLIS
Provincial Programs and Services (see EBSM)	PPSs

Overview of the Canada-Saskatchewan Labour Market Development Agreement

The Government of Canada and Saskatchewan signed the Canada-Saskatchewan Labour Market Development Agreement (CS-LMDA) in February 1998. This Agreement gave responsibility to the Saskatchewan government for designing and delivering employment programs and services through the Employment Insurance Account. Within the province, the Department of Post-Secondary Education and Skills Training (PSEST) has assumed this additional responsibility.

Under the CS-LMDA, the following provincial programs and services (PPSs) will be delivered:

Provincial programs

• Work Placement, Community Works, Work-based Training: Work Placement programs subsidize private employers to provide work experience. Community Works operates similarly, but places clients with community-based organizations (CBOs), municipal and/or local government, Indian Bands, Tribal Councils, and Métis Nation. Work-based Training programs are delivered as part of the Job Start/Future Skills program, which provides unemployed people with work experience placements that include skill development leading to permanent employment, as well as recognition that the trainee has attained specific skills.
• Self-employment Programs: Clients receive assistance to become self-employed. The program includes business plan development and mentoring by people experienced in local business development.
• Unsubsidized work placements are alternatives within the scope of the Employment Programs provided by the province.
• Income support programs: The Skills Training Benefit (STB), implemented on January 1, 1999, supports Employment Insurance (EI) clients in training. It is used in conjunction with Saskatchewan and Canada Student Loans programs, and to a lesser extent, the Provincial Training Allowance (PTA).

Provincial services

• Bridging to Employment: A range of career and employment programs, services, and supports are available from alternate delivery partners/providers, to assist individuals to become "job ready."
• Career and Employment Services Development: Canada-Saskatchewan Career and Employment Services (CSCES) provides access to career and employment services at 20 locations across the province. As well, staff contact employers and agencies to identify job opportunities for clients and work with training institutions, community-based organisations, municipalities, and the private sector to prepare clients for employment. Alternate partners/providers may be used for career and employment services development.
• Sector Partnerships: As a component of the Job Start/Future Skills program, Sector Partnerships support industry sectors to work with training institutions, employers, and communities to design and implement regional needs assessments and human resource strategies.
• Regional Planning Partnerships (RPP): The RPP program is designed to support communities, employers, employees, and other groups to expand their local employment base and develop initiatives to respond to the employment needs of their community.
• Research and Innovation: The province may provide funding for research and innovation projects and activities, which identify better ways of helping persons prepare for or keep employment and be productive participants in the labour force.

Formative evaluation

The formative evaluation provides information on the operational effectiveness of the program, including an assessment of client satisfaction. The formative evaluation will assist both Human Resources Development Canada (HRDC) and Post-Secondary Education and Skills Training (PSEST) to create cost effective, longer-term policies.

The evaluation was intended to address the following issues:

• Rationale - Are provincial programs and services consistent with federal-provincial priorities, and do they meet a need?
• Design and delivery/planning and implementation - Is the program delivered in a manner that is appropriate to achieving the goals under the CS-LMDA?
• Success/impacts/effects - Do provincial programs and services meet the objectives of the CS-LMDA? Due to delays in obtaining a non-participant sample, this work will not proceed as part of the formative evaluation. The formative evaluation presents information on client satisfaction and measures of outcome based on self-report information provided by participants. The non-participant survey and the comparison group analysis will be undertaken as part of the summative evaluation scheduled for 2002.

All methodological components of the formative evaluation have been undertaken except for a non-participant survey and the comparison group analysis. This work will proceed in the course of a later summative evaluation that will measure the longer-term impacts of programs on the clients.

Main findings

Rationale

Prior to the initiation of the CS-LMDA, the province created the Saskatchewan Training Strategy (STS). It laid the foundation for the CS-LMDA and allowed for the integration of federal-provincial programs into the comprehensive labour market development initiative. As such, this agreement is similar to other LMDAs that feature significant devolution of federal labour market programming.

A basic theme of the STS is that labour market training should be undertaken by the current secondary and post-secondary educational system. The STS discouraged the streaming of unemployed persons into programs tailored for EI clients, Social Assistance recipients, and others. The idea is that many clients need common training, and that it makes little sense to tailor training based on client attributes.

To manage the new programs and funds, Saskatchewan has embarked on a comprehensive information strategy designed to record all client activity. Known as the One Client Service Module (OCSM), this system is intended to offer an integrated information solution to record all activity by department staff as well as public and community-based organizations that use or deliver programs and services funded by PSEST.

The final element of the STS was to ground the training design and delivery in a regional and sectoral framework, where business/industry, government, educational institutions, and community organizations could jointly plan courses and increase the efficiency of the labour market. To that end, a network of regional service centres and Internet services was created.

The essential implication of the STS for EI clients is that, in principle, they would access the same portfolio of courses and supports open to any unemployed person in Saskatchewan. However, as the CS-LMDA evolved, EI clients presented special needs, and the province has responded with special programming. Most notable is the Skills Training Benefit, which offers educational financial support (and in some cases income support) for EI clients.

The first year-and-a-half of the CS-LMDA has seen the two orders of government each endeavouring to fulfill the objectives of the EI Act and the Saskatchewan Sector Strategic Plan, which succeeded the STS. Most important has been a protracted negotiation period for the CS-LMDA. Even the evaluation framework (issues and questions) required an extended period of discussion. The process of negotiation and discussion is a process of searching for the common ground to allow the federal government to meet the requirements of the EI Act, and the province to integrate EI clients into its array of program and services.

Provincial programs and services align well with the EI Act, and meet the requirements of the CS-LMDA.

Design, delivery, planning and implementation

The following are the main findings related to the implementation of the Agreement.

• Based on interviews with senior management from the provincial and federal governments, an important goal of the CS-LMDA was to ensure that the programs and services in Saskatchewan matched the employment benefits and support measures (EBSM) offered in the pre-LMDA era. The STB is an example of a program not included in the Saskatchewan array of programs and services that the province added to meet the special needs of EI clients.
• For the most part, overlap and duplication of PPSs are minimal. Three areas exist where federal and provincial programming purposely overlap: programming for youth, persons with disabilities, and Aboriginal persons. Federal and provincial representatives stated that this apparent overlap increases service.
• Several federal and provincial officials reported that partnerships with industry have been strengthened under the CS-LMDA. Some provincial managers reported this may also reflect the sectoral planning process that the province has supported for almost a decade.
• It is not possible to establish whether services in French are adequate, because the total demand for service in French is very low. Most of those who requested services in French reported being satisfied (nine out of eleven).
• Federal officials reported that the province was slow to identify the contribution of the federal government. Both federal and provincial representatives reported that the situation has improved.
• The provincial information system needs further development to support program monitoring and performance measurement. Development appears to be behind schedule, and information from various program sources is not fully integrated. Reconciling existing databases and adopting standards across programs (e.g., type of program, outcome codes, dates for program start and end) are urgently needed to improve the system.
• Information sharing protocols between federal and provincial partners need improvement. Both federal and provincial representatives criticized the inability to share information properly. Key informants from both federal and provincial governments report that the process that allows Canada-Saskatchewan Career and Employment Services staff to identify EI eligible clients is inefficient, and compromises effective program delivery. The inability to obtain the non-participant sample shows that the information sharing process requires further development.

Client satisfaction and preliminary outcomes

Some impacts can be discerned based on the client and employer surveys:

• Clients are satisfied with services provided at CSCES offices. Most who have used them have found them helpful — particularly career counselling, multimedia products for resume writing, and computer services for searching the Internet.
• Employers find it difficult to distinguish between the former federal and current provincial training programs. Many employers who participate in a wage subsidy program or use the job order process to find employees are unaware that they have hired a former EI client, and as such, they are unable to evaluate the current provincial training programs with respect to serving EI clients.
• Service providers are aware of the changes brought on by the LMDA in terms of the use of community-based organizations. Some service providers report that they now receive some core funding, enabling long-term planning. A difficulty was noted in training SA, EI, and non-income support students in a mixed class.
• An intended outcome of the CS-LMDA is improved planning. An important change resulting from the LMDA has been the shift in authority for planning to the recently created Regional Services Branch (CSCES), which now controls the budget and has the decision-making authority.

Program impacts/effects are preliminary

Questions posed in the evaluation framework regarding success, impacts, and effects are addressed in a preliminary and tentative way through client surveys, focus groups, and dyads.

• Clients using provincial programs and services express high levels of satisfaction.
• Based on the client survey (n=1,250), about one-third of participants said they were required to prepare a case/action plan2¹ before beginning training. A similar proportion of clients completed case/action plans in each region. Based on the client and focus group responses, back-to-work plans (case/action plans) appear to be an effective and useful part of the training process.
• Employers are generally satisfied with programs targeted to EI clients. Programs have helped a majority of employers to fill job vacancies and skill shortages, and meet human resource needs.

The formative evaluation of the Canada-Saskatchewan Labour Market Development Agreement has now been completed and accepted by both orders of government. Recommendations will be considered in the context of ongoing programs and services and where possible, the findings will be used in the continuous improvement of services to clients.

The evaluation report will be shared with partners and stakeholders.

The Labour Market Development Agreement (LMDA) are the outcome of policy evolution during the 1990s.² An initial step in this evolutionary process is the revision to the Unemployment Insurance Act of 1970, which saw the implementation of Part II measures designed to support the re-entry of selected Unemployment Insurance (UI) claimants back into the workforce through various programs such as the payment of living allowances and training costs.

The LMDA initiative supported the transfer of financial resources, policy, and programming from the federal to provincial/territorial governments. These agreements also included the transfer of personnel from the federal to the provincial/territorial governments in some cases, and the identification of specific targets for performance.

Two general "styles" exist within the LMDA. "Devolution" agreements accommodated provinces/territories that wish to design and manage the delivery of Part II benefits. "Co-management" agreements maintain the federal programming, but include provision for extensive provincial/territorial input.

1.1 Overview of the Canada-Saskatchewan Labour Market Development Agreement and its context

The Government of Canada and Saskatchewan signed the Canada-Saskatchewan Labour Market Development Agreement (CS-LMDA) in February 1998. This Agreement gave responsibility to the Saskatchewan government for designing and delivering employment programs and services for Employment Insurance (EI) eligible clients. Funding for these programs and services comes through the Employment Insurance Account. The CS-LMDA came into effect a year after Saskatchewan had launched a major training initiative in April 1997, namely the Saskatchewan Training Strategy (STS). In effect, the CS-LMDA needed to be integrated into a substantial redesign of provincial labour market development activities. As such, the CS-LMDA rests firmly in the "camp" of LMDAs that support the devolution of federal programming for EI clients eligible for Part II benefits to provinces and territories.

This context is important, since the STS provides the framework for all labour market development programs in the province. It is essential to place the CS-LMDA in the general labour market development context for Saskatchewan.

The STS had three goals: to develop a skilled workforce relevant to Saskatchewan's labour market; to enhance access and support opportunities for all learners; and to create a coherent, effective, and sustainable delivery system. The results of an evaluation in 2000 serve as input to the Sector Strategic Plan for the department. An important objective of both the STS and the Sector Strategic Plan is to reduce the use of programs targeting selected clients. A goal of the STS was the development of programs and services for all learners.

At the same time, Saskatchewan has not created interventions designed primarily for a specific target group. The strategy focuses on eliminating the "categorical" definition of clients based on ethnicity or program participation (e.g., reliance on Social Assistance), and instead bases support on income levels and employment status. Services, including career counselling, job search assistance, and Internet-based resources are available to all Saskatchewan residents. The training strategy also attempts to foster a training culture among Saskatchewan employers and included skill upgrading for those currently employed.³ EI clients fit within this "holistic" training environment, and receive the same programs and services as other clients.⁴

The provincial training strategy minimizes the creation of specific services for EI clients (or for any target group), and attempts to harmonize and simplify the overall training environment. Funding for EI clients flows from the EI Account via the CS-LMDA.⁵

1.1.1 Features of the CS-LMDA

Several features of the CS-LMDA are worth noting:

• Saskatchewan has used CS-LMDA funding to further develop a single window career and employment services strategy, coordinated through the 20 Canada-Saskatchewan Career and Employment Services (CSCES) centres located throughout the province. These centres enable the province to deliver certain functions of the National Employment Service, including employment counselling, needs assessment, and job placement services. Both governments have agreed to participate in providing and distributing labour market information (LMI).
• The Government of Canada, through Human Resources Development Canada, remains responsible for determining eligibility and distributing Part I EI benefits.⁶ HRDC also provides national labour market services, such as labour market information.
• Like other LMDAs, Saskatchewan residents who are collecting EI benefits or have done so in the past three years ("reachback clients") are eligible for programs and services funded under the Agreement.⁷

From 1998—99 to 2001—02, more than $149 million (approximately $38 million per year) will flow from the EI Account to support provincial programs and services (PPSs).

The LMDA commits the province to provide programs and services to EI eligible individuals; 65 percent must be active EI claimants. During 1998—99 targets set were 4,332 insured participants returned to employment, and $19,952 million will be saved from the EI Account.

The formative evaluation is one step in determining the achievement of these targets, and complements other accountability processes that the two orders of government will take under the Agreement.

1.2 Description of provincial programs and services

Under the CS-LMDA, the following provincial programs and services⁸ will be delivered:

Provincial programs

• Work Placement, Community Works, Work-based Training: Work Placement programs subsidize private employers to provide work experience. Community Works operates similarly, but places clients with community-based organizations (CBOs),⁹ municipal and/or local government, Indian Bands, Tribal Councils, and Métis Nation. Work-based training programs are delivered as part of the JobStart/Future Skills program, which provides unemployed people with work experience placements that include skill development leading to permanent employment, as well as recognition that the trainee has attained specific skills.
• Self-employment Programs: Clients receive training to become self-employed. The program includes business plan development and mentoring by people experienced in local business development. A flexible range of financial supports is provided during the business development phase, depending on the needs of the client.
• Unsubsidized work placements are alternatives within the scope of the Employment Programs provided by the province.
• Income support programs: The Skills Training Benefit (STB), implemented January 1, 1999, supports EI clients in training. It is used in conjunction with Saskatchewan and Canada Student Loans programs, and to a lesser extent the Provincial Training Allowance (PTA). The aim of the STB is to help clients gain skills needed for employment, by providing support to clients for accessing/participating in programs offering short-term skills training, including employability skills and work experience.

Provincial services

• Bridging to Employment: A range of career and employment programs, services, and supports are available from alternate delivery and training partners/providers, to assist individuals to become "job ready." This includes (but is not limited to) assessments, employment counselling, assisted job placement, job search assistance, job readiness training, and the provision of labour market information.
• Career and Employment Services Development: Canada-Saskatchewan Career and Employment Services (CSCES) centres provide access to career and employment services at 20 locations across the province. As well, staff contact employers and agencies to identify job opportunities for clients and work with training institutions, community-based organizations, municipalities, and the private sector to prepare clients for employment. Alternate partners/providers may be used for career and employment services development.
• Sector Partnerships: As a component of the JobStart/Future Skills program, Sector Partnerships support industry sectors to work with training institutions, employers, and communities to identify sector-wide human resource needs and develop human resource strategies. Partnerships are used to decrease skill mismatches in the labour market.
• Regional Planning Partnerships (RPP): The RPP program is designed to support communities, employers, employees, and other groups to expand their local employment base and develop initiatives to respond to the employment needs of their community.
• Research and Innovation: The province may provide funding for research and innovation projects and activities, which identify better ways of helping persons prepare for or keep employment and be productive participants in the labour force.

1.3 Purpose of the formative evaluation

In general terms, the formative evaluation assesses the operational effectiveness of the program, including measurement of client satisfaction, staff perspectives, and a preliminary review of outcomes. The formative evaluation will assist both Human Resources Development Canada (HRDC) and Post-Secondary Education and Skills Training (PSEST) to create cost effective, long-term programs and services. The formative evaluation will also offer management information to support program improvement.

Specifically, the evaluation addresses:

• Rationale - Are provincial programs and services consistent with federal-provincial priorities, and do they meet a need?
• Design and delivery/planning and implementation - Is the program delivered in a manner that is appropriate to achieving the goals under the CS-LMDA?
• Success/impacts/effects - Do provincial programs and services meet the objectives of the CS-LMDA? (Due to unforeseen delays in assembling lists of EI clients that have not participated in provincial programs and services, it was not possible to create a comparison group. The non-participant survey and comparison group analysis will be completed as part of the summative evaluation scheduled for 2002.)

1.4 Outline of the report

This report has five sections, responding directly to the evaluation framework:¹⁰

• Section 2: Review of evaluation framework and methodology;
• Section 3: Issues in the rationale of the program;
• Section 4: Design, delivery, planning and implementation;
• Section 5: Success/impacts/effects (confined to client satisfaction and preliminary outcomes);
• Section 6: Concluding observations.

2.1 Main issues for the Phase I evaluation

The Canada-Saskatchewan Labour Market Development Agreement (CS-LMDA) evaluation has a formative phase and a summative phase, with the following objectives.

Formative evaluation (Phase I):

• to provide quantitative and qualitative data, which may be used by program managers and others to improve and enhance the design and delivery of the program before processes become entrenched;
• to indicate what is required with respect to information gaps;
• to demonstrate how well the program is working;
• to provide early warnings if expectations are not being met.

Summative evaluation (Phase II):

• to focus on the longer-term impacts of interventions, and to measure the outcomes, impacts, and cost-effectiveness of the interventions;
• to provide an assessment of the program's impacts on clients, by answering the following key questions:
  • Have the programs and services produced impacts on individuals and communities that are attributable to the intervention itself (incremental impacts)?
  • Have the desired outcomes/objectives of programs and services for clients been met, as defined in the Agreement?

2.2 Methodology

The formative evaluation relied on the methodologies summarized below:

The first issue addresses the rationale for the program, which breaks into two sub-questions. The first asks how provincial programs and services align with the Employment Insurance (EI) Act and the Canada-Saskatchewan Labour Market Development Agreement (CS-LMDA), and the second addresses the relevance of Provincial Programs and Services (PPSs) to clients, employers, and the community.

3.1 How do provincial programs and services align with the EI Act and the CS-LMDA?

The Canada-Saskatchewan Labour Market Development Agreement transferred to the Province of Saskatchewan the responsibility to design and deliver employment programs and services through the Employment Insurance Account. As part of this Agreement, provincial programs and services (PPSs) had to be similar to those previously offered, and consistent with the principles, guidelines, and intent of the EI Act and CS-LMDA.

Table 2 below outlines the relationship between federal benefits (programs) and measures (services), and provincial programs and services. This table shows that provincial programs and services appear to align with the EI Act and the CS-LMDA.

The labour market development environment in Saskatchewan presents challenges for fully appreciating the extent to which provincial programs and services reflect all requirements of the EI Act and CS-LMDA. In comparison with the LMDAs that feature co-management, it is clear that the federal government has devolved considerable responsibility for designing and delivering labour market development services to EI clients. For those LMDAs that feature devolution, it is likely that Saskatchewan represents a significant "evolution of devolution", but this evaluation does not offer a comparison among LMDAs.¹²

Guided by the Saskatchewan Training Strategy and the Sector Strategic Plan for Post-Secondary (PSEST), the province is attempting to streamline provincial income supports and training programs and services available to unemployed, low-income, and disadvantaged people in the province. PSEST used the STS to broaden and simplify the process of accessing training and income supports by offering support services to all Saskatchewan residents, and basing eligibility for training on income rather than other characteristics (e.g., participation in Social Assistance, ethnicity, etc.). An example of broadening and simplifying services under the STS is the creation of the Provincial Training Allowance. This program creates a single income support measure that enables low-income people to participate in basic education or short (<8 weeks) skills training. It is conceivable that the Provincial Training Allowance (PTA) could support an EI reachback client who no longer qualifies for EI Part 1 (income support). The PTA allows income support for anyone whose income is low — EI reachback, active EI claimant, or Social Assistance recipient. Prior to the CS-LMDA, EI clients had separate income support programs and were more separated in their training.¹³

Integrating EI clients within training environments presents both opportunities and challenges aligning provincial programs and services with the EI Act and the CS-LMDA.

• An important advantage of adopting a unified training approach such as Saskatchewan has done is the ability to deliver a menu of programs and services to all clientele. The array of provincial training programs is relatively unchanged by the CS-LMDA. The Employment Programs (Work Placement, Community Works) and JobStart/Future Skills all existed prior to it. They serve a broad client base, including Social Assistance recipients, youth, the disabled, and the unemployed. The CS-LMDA has had little effect on program delivery for these interventions, although, the province is now an active participant in the delivery of labour market specific programming, given the addition of EI clients to its specific clientele.

  In Saskatchewan, the Saskatchewan Institute of Applied Sciences and Technology (SIAST) or regional colleges deliver skills training and basic education programs. A self-employment program and the Skills Training Benefit are the only significant changes introduced by the CS-LMDA. The self-employment program appears closely modeled on the federal self-employment program that previously existed. The Skills Training Benefit is only available to EI clients. It is the program that Saskatchewan developed to meet the federal Skills, Loans, and Grant Benefits of the CS-LMDA—(Annex 1).

• There are more significant differences relating to support services and income support programs for EI clients.
  • In conjunction with the CS-LMDA, PSEST has developed a Regional Services Branch structure and created 20 Canada-Saskatchewan Career and Employment Services (CSCES) offices. These centres provide services such as counselling, job order taking, resume writing, job searches, and workshops to all Saskatchewan residents seeking work or wishing to access training. EI clients can use the same services as others using the centres. In addition to providing services to clients, CSCES offices act as a hub for regional and human resources planning, and for coordination among service providers/partners (SIAST, regional colleges, CBOs) and employers.

  • The CS-LMDA also led to the creation of the Skills Training Benefit (STB), a income support program that represents Saskatchewan's response to the federal "Skills, Loans, and Grants" component of the federal transfer. The STB provides support to EI clients for tuition and books for training and income support only to those clients with a demonstrated need. The amount of support is negotiated between the client and a counsellor at one of the Canada-Saskatchewan Career and Employment Services. The negotiated process between client and counsellor is used to facilitate a better match between funding and need. According to several provincial key informants, the creation of the STB went against the Saskatchewan Training Strategy's goal of streamlining income support in the province by blending existing income support measures into a single, non-categorical program (i.e., the Provincial Training Allowance). The PTA is the main provincial income support program available to provincial clients in basic education and short (< 8 weeks) skills training. It supplemented the student loan program that is available to qualifying clients for courses of more than 12 weeks in duration at a recognized post-secondary institution.

    Some federal key informants reported concerns that the PTA may offer less financial support than was previously available to EI clients during the pre-LMDA period. The province created the STB to address the needs of EI clients, adding to the array of income support programs available to adults undertaking post-secondary training.

    An important question for the evolution of the CS-LMDA is whether the STB should remain as is, and continue to use a negotiated process. An alternative is to consider moving the STB to a flat rate, such as the PTA. A third option would be to combine the PTA and STB to form a single income support program. Since the PTA is a provincial program, it is within its mandate to determine how its income support programs and services will be operated.

Table 2 above shows that it is possible to list Employment Benefits and Support Measures with provincial programs and services, and therefore, alignment exists. However, it is important to stress that EI clients co-mingle with all other "learners" in the province. The discussion above illustrates how the EBSMs available to EI clients before the CS-LMDA have been integrated into a broader labour market development system, and not merely devolved to the province. Therefore, the alignment is complex. The essence of this issue is whether the same EI clients can access the same programs and services under the CS-LMDA as before, and whether these programs and services meet the requirements of the EI Act, to which the response is unequivocally "yes."

3.2 How do PPSs meet the needs of EI clients, employers, and the community?

Within the training environment, the needs/interests of clients, employers, service providers, and community-based organizations intersect with PPSs in different ways. Even within these groups, diversity of need exists. For example, an EI client who takes an apprenticeship program and is unemployed only while in training has different requirements than someone with obsolete work skills. Some employers require skilled workers to train for long-term careers, while others seek temporary help from minimally trained workers. Service providers differ considerably, from SIAST and regional colleges to community-based organizations.¹⁴ The formative evaluation provides a sense of whether needs/interests have been met, as well as the remaining challenges that these groups face in using or delivering programs and services.

3.2.1 How PPSs meet the needs of EI clients

As mentioned above, EI clients have diverse needs. Some participate in longer-term programs such as academic upgrading, while others use short-term training for specific trade/job skills. For still other EI clients, the loss of employment can represent the beginning of a difficult cycle in and out of training and work. Finally, some report minimal interaction with the EI "system," as they appear to move relatively seamlessly between employment and training. Often these clients are in unionized trades, where hiring processes are highly formalized and unemployed workers are accustomed to waiting for the call back to work.¹⁵

Key findings regarding service to clients under the CS-LMDA are as follows:

• On balance, clients (n=216) claim service is the same or better under the CS-LMDA. Based on the client survey, 38 percent reported better service after January 1999 than before, while 39 percent said the level of service was the same. Nine percent (9 percent) said the service was worse, while 13 percent did not know. Only those who had experience both before and after January 1, 1999 responded to this question.¹⁶
• Key informants¹⁷ (n=41) disagreed on whether individuals received better service than before the CS-LMDA. Almost half of respondents¹⁸ (16 of 41) believe individuals are now better served. Reasons given by respondents were the introduction of "one-stop shopping," the assisted self-serve resources, and the fact that the public can now access services. However, a similar number was unsure, did not know, or gave no response, mainly because there are no benchmarks for comparison. The remaining respondents said individuals do not get better service. These key informants said disruption caused by changes have negatively affected or confused clients.¹⁹
• Many clients welcome greater opportunities to plan their careers. Some clients said they approached CSCES with a clear idea in mind regarding the kind of training they wished to pursue, and were able to train in that area. Others said they were uncertain about the direction they should take, and either selected a training course with the help of CSCES counsellors, or enrolled in programs specifically designed to expose them to a variety of training and employment possibilities. Most appreciate having the responsibility of making their own decisions, and planning a path that best suits them.²⁰
• Participants expressed satisfaction with the training they received, and believe it met their needs. Seventy-eight percent of clients (n=1,017) on the client survey reported that the programs and services they received during their most recent time on EI met "all" or "some" of their needs, while only 16 percent said that it "did not meet any of their needs."²¹ However, several focus group participants reported being placed in programs for which they were unsuited. These participants said CSCES is not adequately matching individuals' skills and interests with training and employment options.
• Most training meets the regional needs of clients. Although some participants believe training is not tied closely enough to local labour market opportunities (especially in small or rural markets where there are few jobs), several participants noted that identifying suitable areas in which to train is one of the main purposes of the case/action plan.²²
• Career counselling meets the needs of clients, though some were dissatisfied with the guidance they received. Eighty-two percent (82 percent) of clients (n=576) on the client survey reported that counselling met "all" or "some" of their needs, while 18 percent said it "did not meet any of their needs."²³ Focus groups support this finding, though the groups also amplified the negative experiences of some clients. Several focus group participants said they succeeded in spite of the guidance from their counsellor, who would not listen or guided them into training for which they were not suited. Overall, most EI clients report the counselling as a positive experience.
• Some participants affirmed that employers should direct the training process to ensure relevance to local labour market opportunities. A few participants commended regional colleges for taking direction from the private sector in providing training in relevant areas.²⁴
• Key informants agreed that the type of programs is similar to what was previously available, with the exception of the new CSCES offices. Many respondents said availability has increased, because programs and services are now available to a wider range of clients. However, a few believe that program criteria and policies have become more stringent under the province. There is a perception that the province is more "rules-oriented" than the federal government. Some key informants also noted that the public is confused over the difference between HRDC and Canada-Saskatchewan Career and Employment Services, or is unaware of changes since the CS-LMDA.

3.2.2 How PPSs meet the needs of employers

Employers are both directly and indirectly involved in provincial programs and services for EI clients. Most employers become directly involved with government training programs because they need some type of support (e.g., training subsidy, wage subsidy) to hire an employee. They become indirectly involved through sectoral planning, where industry associations often represent employers within a specific industry, and identify training needs and supports. In both cases, employers are not specifically concerned with training and supports for EI clients; they seek support for hiring new employees, whomever they may be. Employers' perspectives typically reflect the broader training environment, rather than EI programs specifically.

Key findings are as follows:

• Most of the 146 employers surveyed expressed satisfaction with the most recent EI client they hired. Seventy-four percent (74 percent) of respondents reported that the EI client they hired fit into their workplace.²⁵ Fifty-nine percent (59 percent) of employers said that programs/services met or exceeded their expectations. Of those who said their expectations "were not met" or were "partially met," 88 percent (35 of 40) cited a problem with the employee.
• Wage and training subsidies encourage some employers to specifically hire EI clients. According to the survey of employers, more than half of employers said they would not have hired EI clients without a government subsidy. Several participants in the focus groups stated they hired someone whose skills were not well suited to the job, but hired them because government support was available. Forty-seven percent (47 percent) of employers said that having programs specifically targeted to EI clients helped them locate employees.²⁶
• Employers indicate that the same, if not more programs are now available to them. Thirty-two percent (32 percent) indicate that there are now more programs available, while 33 percent remained neutral. Fifteen percent (15 percent) thought there were fewer programs, while a similar proportion did not know.²⁷
• Of the 146 employers surveyed, one-third stated that it is as easy or easier to get government support for employment/training programs since the CS-LMDA was implemented. Thirty-four percent (34 percent) reported that it is now easier to get government support for employment/training programs, while 33 percent remained neutral. Sixteen percent (16 percent) thought there were fewer programs, while 17 percent did not know or did not respond.²⁸
• PPSs have helped employers meet labour shortages. Over half of the respondents to the employer survey agreed that in the last two years, PPSs have helped them fill job vacancies (63 percent), fill skills shortages (50 percent), and meet human resource needs (54 percent).²⁹

• During the focus groups, some employers indicated they want training programs tailored to meet their needs. The main complaint from employers was that those hired for work placements/on the job training typically require a lot of training, often beyond the scope of support provided by the program. Further, employers often have to supervise or train these employees using experienced workers, which decreases their productivity. In the focus groups, some employers expressed the belief that government does not always recognize or compensate their contribution to the training process. Employers with less demanding training needs reported that three to six months was a sufficient period to decide whether an employee would fit into the company.³⁰

  The employer survey confirms this finding: 44 percent of respondents suggested increasing the length of the wage/training subsidy, while a similar proportion wanted the amount of the subsidy increased (43 percent). Many employers would not have provided any on the job training without a subsidy (55 percent).³¹

• Some employers in the focus groups desire greater program flexibility. According to several focus group participants, programs should be flexible in meeting employer needs. If the employee has higher needs (skill development, workplace support, language), the system should be able to adapt and help the employer either financially or by providing additional supports. The employer survey did not test this issue, and it should be revisited in the summative evaluation.³²
• Employers report program staff to be helpful. Based on the survey of employers, 75 percent agreed with the statement, "Program staff who provided assistance are knowledgeable and helpful."³³ Focus group participants said they are satisfied with the information they received from government upon receiving support to hire an employee. Those who had used the system more than once now understand how to apply for funding more effectively. Several employers reported that the CSCES counsellor provided excellent follow-up and support over the duration of the program.³⁴
• Employers have mixed views regarding program paperwork. Approximately the same proportion of respondents either agreed, disagreed, or remained neutral when asked whether "too much paperwork is involved when applying for programs," and whether "too much paperwork is involved when reporting on program outcomes." Twenty-three percent (23 percent) of employers suggested reducing paperwork/improving administration as a way to improve PPSs for EI clients.³⁵
• While most employers are satisfied with proposal "turnaround" time, a few indicate that this process should be faster. When asked to suggest any changes to the programs, 24 percent suggest reducing the time required by administration to "make decisions."³⁶ Participants in focus groups echoed this opinion.
• Employers are generally aware of CSCES, but some appeared unclear about the range of programs and supports available to them. Based on the employer survey (n=146), 27 percent of employers had visited a centre, 54 percent had phoned or faxed a centre, while 10 percent reported contact via the Internet. One-quarter reported no contact.
• Several employers who participated in the focus groups said that government should do more direct marketing to employers, and explain the range of options available. Employers claimed that government should help employers access support that would benefit them and potential employees. Passive marketing strategies (such as brochures) are not always effective, as they do not always capture an employer's attention. Furthermore, focus group participants reported that they have trouble distinguishing between programs (who they serve, who funds them). Most could not distinguish between federal and provincial employment programs.
• Employers are satisfied with CSCES services, but suggest a need to improve Internet-based resources. As indicated in Table 3, CSCES rated much higher with respect to "posting job vacancies" (3.8 out of 5), "applying for a training program" (3.8), "providing information about education/training programs" (3.7), and "locating potential employees" (3.5) than the HRDC web site and Saskatchewan web site.³⁷ This could indicate that employers prefer face-to-face contact, or it may indicate that Internet services need to be improved.

• Some key informants believe that PPSs have not yet fully addressed employer needs. Some of the ways that government is trying to address employer needs are: on-line job order/job matching system (n=8); employer participation in needs assessment/planning (n=9); program interventions linked to regional labour market realities (n=5); and increased funding to subsidize employment (n=5).

The scope of the CS-LMDA and PPSs targeted to EI clients is broad. While most employers and key informants (federal and provincial) believe that PPSs are successful, some suggested areas for improvement. For example, employers suggested additional program marketing, and tailoring of training subsidies and amounts to meet individual needs. During focus groups, employers supported the development of a training culture in Saskatchewan.

It is important to note that an employer will, by economic necessity, focus first on the needs of the firm, followed by the needs of EI clients. This is a critical issue, as the needs of employers and the needs of clients (or government as a representative of those clients) may not coincide. Employers want to hire the person with the most appropriate skills, but government may specify that a training subsidy can only be used for an EI client. This divergence of requirement is not easily resolved, but it lies at the root of why some employers have a muted interest in supporting training programs for economically disadvantaged persons. Further, some employers stigmatize the "EI" label as identifying someone unable to find work on their own.

3.2.3 PPSs and the interests of the community (service providers)

The evaluation framework includes a question about the extent to which the PPSs are relevant to the needs of the community. The concept of community is very broad, and in consultation with the Joint Evaluation Committee, we elected to interpret community as the various service providers. Under the CS-LMDA, this includes SIAST, regional colleges, and various non-governmental and community-based organizations that play an important role within the Saskatchewan training environment.

• SIAST and regional colleges (RCs) deliver most provincial training. They also collaborate with government, consult on program design and delivery, and often participate in regional needs assessments.
• Medium-sized service providers, such as Dumont Technical Institute and Saskatchewan Indian Institute of Technologies, provide training and support predominantly for Aboriginal persons.
• Community-based organizations tailor services to specific groups (e.g., people with disabilities, recent immigrants, Aboriginal persons, women), or find niches that other providers do not fill.
• Private trainers deliver "niche" training services or compete with SIAST and regional colleges to provide technical training.

Based on several focus groups, service providers report few complaints regarding the transition of programs and services for EI clients from the federal government to the province. Given that most trainers also provide programs and services for other provincial clients (primarily Social Assistance clients), trainers are utilizing the same networks and programs as they do for EI clients.

Service providers' perspectives on the CS-LMDA include the following:³⁸

• The proposal process is very similar. In the pre-LMDA period, either trainers collaborated with HRDC when developing a program, or they would respond to a call for proposals. This process is similar under the CS-LMDA; however, several respondents noted that the province is more willing to provide funding upfront. They prefer this process when setting up their courses.
• The province has implemented a follow-up process, similar to that of the federal government. Follow-up is a costly process, and some would see lengthier follow-up as a burden on trainers. Currently, trainers complete follow-ups after four months.
• EI clients are reportedly a small proportion of their service load. Service providers seldom (if ever) provide training specifically for EI clients. The majority of clients are on Social Assistance (SA), although EI and SA clients often attend the same classes. Service providers submit proposals for generic training, of which a portion of the participants will be EI clients. Similarly, the grants used by trainers are for overall training overheads and costs, and not specifically for EI clients. It is impossible to separate the EI client share of a grant or contract for a training program.
• Mixing clientele with different needs can be difficult for trainers, as SA clients often require more counselling and personalized attention. The mixture of clients can make it difficult for trainers to deliver one program, when students have a wide range of needs and abilities. This suggests that more streaming may be appropriate.
• Prior to the LMDA, the federal government typically funded service providers (primarily CBOs) on a project-by-project basis. By comparison, the province commonly provides core money, as well as project-based funding. According to service providers, core funding stabilizes the training environment and makes it easier for trainers to consider longer-term training strategies. Stable funding also enables long-term planning and development of support services. Previously, most projects were on a contractual basis, making it difficult to plan in the long term. Finally, some service providers argued that continued long-term funding would make it easier to retain quality staff and provide better and continuous services to clients who cycle in and out of work.
• Several respondents noted that pre-LMDA, EI training allowances were easier to understand. Under the LMDA, the mixed training environment coupled with the variety of income support programs has confused some trainers. For example, clients in the same program could receive STB, PTA, or EI Part 1 benefits. Rules for income support programs differ, and trainers are asked by clients to explain why certain rules (such as attendance) apply to some students and not to others. Referrals to CSCES would resolve this.
• Philosophies differ regarding job search. During the pre-LMDA period, the federal government required mandatory job search. Under the LMDA, the province typically does not require people to search for work. The province often does not know who is an EI eligible client until the client applies for training or uses a service. This issue reflects the split in responsibility for EI Part 1 benefits (eligibility determined by the federal government) and Part 2 (programs and services provided by the province).
• Service providers see CSCES offices as effective. CSCES offices do a good job of directing clients to appropriate training, and have good referral processes in place. Overall, CSCES offices have functioned well in their role, and they have been proactive in communicating with service providers and accommodating their needs.
• According to several service providers, CBOs get less support and funding than regional colleges and SIAST. Several trainers maintain that government does not fully recognize the important role that CBOs play in the training environment. The province intends its new contracting strategy to offer multi-year support to CBOs to meet the needs of specific clientele. This should address many of the concerns expressed by these CBOs.
• Counselling is more developed and defined under the LMDA. Clients typically come to service providers with clearer goals and with career plans in place. Most counselling is on a one-to-one basis, a further benefit for EI clients.
• Trainers suggested listing the range of programs offered on the provincial web site. This would include who is offering the program, where it is located, etc. Providing a centralized list would match clients and trainers.

3.2.4 Community involvement in regional needs assessments

A second aspect of how the CS-LMDA meets the needs of communities can be seen in the regional needs assessment process.³⁹ These exercises involve a high level of community participation and are intended to support community decision-making. They are environmental scans that focus on particular sectors (e.g., mining, the motion picture industry, agriculture), groups (e.g., Aboriginal persons, youth, people with disabilities), or the design and delivery of programs and services (e.g., Basic Education, Bridging). These have important implications for EI clients, but EI clients are typically not the focal point for the research.

At this stage, it is difficult to determine whether the CS-LMDA and the needs assessments meet the needs of communities, simply because it is unclear what is meant by "community." For this evaluation, we define community as the organizations that participate in the local/regional needs assessment process. From this limited perspective, it is possible to identify accomplishments in terms of regional and local training needs assessments and the sectoral planning process. It is also important to note that the training needs assessments and sectoral planning processes predate the CS-LMDA. In effect, the Agreement incorporated a pre-existing planning process in which EI clients are not targeted specifically, but can expect to indirectly benefit from assessments and sectoral planning. At this early stage, no systematic evaluation exists to assess the contribution of these two processes.

The following provides an overview of these processes to date:

• Training needs assessments in different regions/communities have different goals. Our review of recent needs assessments identified the following goals:
  • providing support for identifying emerging jobs and training needs for local labour markets;
  • identifying training strategies to support specific sectors;
  • identifying industries and employers that have difficulty hiring and maintaining skilled workers;
  • improving access to programs and services; improving the delivery of PPSs;
  • assessing responsiveness to the needs of equity groups; fostering strong working relationships with community and regional partners;
  • determining resource allocation at the regional and sub-regional level;
  • maintaining current labour market information.

These goals all relate to PPSs generally, but EI clients are not the focal point.

• The CS-LMDA has changed the training needs assessment process, simply because regional CSCES offices now direct the process. Pre-LMDA, the main agencies involved in this process have typically included the province, HRDC, SIAST, regional colleges, and key industry representatives. With CSCES now having responsibility for the training needs assessment budget and approach, the process has changed in some regions. As this process continues to evolve, regions will likely adopt their own techniques most suited to the needs of the region.
• The regional needs assessment process contributes to the development of partnerships. Organizations most commonly involved in the assessment process include CSCES, SIAST, regional colleges, HRDC, Social Services, industry associations, First Nations and Métis representatives, employers, economic development organizations, health organizations, school representatives, and community-based organizations. Key informants who have been involved in the process cite stronger partnerships as a result, although some observed that it can be difficult to meet the needs of all partners, because each may have different interests. For example, training deliverers are often most interested in identifying the immediate demand for training to fill courses, whereas government may take a more long-term and broader perspective of regions' training needs. Government must also look at the needs of different client groups (EI clients, Social Assistance clients), and devise strategies to increase self-sufficiency and reduce dependency on income support. These strategies must integrate services while meeting the needs of diverse clientele.
• The provincial government has identified specific regional and provincial sectors. As of the 1999/2000 fiscal year, sectors of focus include: Agriculture, Mining, Forestry, Tourism, Motion Pictures, Outfitting/Eco-tourism, Trade and Retail, Export Manufacturing, Implement Manufacturing, Healthcare, Justice/Corrections, Information Technology, Food Processing, Trucking, Textiles, Recording Industry, and Aviation. It is likely that EI clients have received training in some or all of these sectors, but the focus of the planning process is industry sectors and specific target groups.
• Some target groups are the focus of the planning process. Three main groups form the focus of several regional assessments: youth, First Nations/Métis Nation, and persons with disabilities. These are also target groups of HRDC. Other assessments identify women, older workers, recent immigrants, small businesses, and the Francophone community. It is possible that EI clients present a less pressing challenge compared to these other groups, especially in the North, which is why the regional and sectoral planning process has not targeted them.
• Employers who have participated in the regional needs assessment support it as it currently operates. As noted in Table 4 below, employers (n=146) view the process as effective. Identifying industry/sector skills needs, planning workplace adjustment, developing training solutions, and developing programs and services all rated 3.7 or more on a 5-point scale. However, note that few respondents (approximately 9 percent) were able to respond to these questions. This suggests that only a minority of employers know of these processes. Typically, participants tend to be active in business and industry associations, which attract only a minority of employers.

The regional needs assessments are processes rather than outcomes. They vary in form and activity, based on the direction of regional CSCES offices. Using key informant interviews, a review of assessments and sector plans, and the employer survey, we conclude that these processes appear to meet the needs of communities. At the very least, these processes help strengthen partnerships, and ideally create opportunities for EI clients to find work in growth sectors of the economy.

Some regions are clearly more advanced in using the planning processes than others. As we note above, the regional planning process existed before the CS-LMDA. For example, the sectoral plan for forestry is the culmination of well over a decade of consultation.

3.3 Summary on rationale for the CS-LMDA

The Canada-Saskatchewan Labour Market Development Agreement replicates the services to EI clients that existed before enactment of the Agreement. Provincial programs and services align with the Employment Benefits and Support Measures. The province has created the Skills Training Benefit specifically to increase the income support available to EI clients.

Interviews and surveys with key informants, EI clients, employers, and service providers confirm that PPSs are relevant to their needs. In general, clients and employers accept the services, and express satisfaction with the service they receive. The "regionalization" of service delivery through CSCES centres is welcome for many clients.

The second major theme for the evaluation relates to design, delivery, planning and implementation.

4.1 Do Provincial Programs and Services (PPSs)reflect the intent of the Canada-Saskatchewan Labour Market Development Canada (CS-LMDA)?

4.1.1 Overlap and duplication

As already noted, key informants said both the federal and provincial governments offer services to Aboriginal people, youth, and persons with disabilities.⁴⁰ In some cases, programs receive funding from both orders of government. For example, both federal and provincial governments fund the Employability Assistance for Persons with Disabilities program (EAPD), but it is delivered through the province. In other areas, federal and provincial programs may appear to overlap, but as stressed by some respondents, these are instances where both orders of government coordinate complementary programming.

Provincially, income determines eligibility for training support. Therefore, youth, Aboriginal people, and persons with disabilities are able to receive support for training like other Saskatchewan residents,⁴¹ and may also be eligible for targeted support from the federal government. Federal programs are more likely to target specific groups. For example, the Youth Employment Strategy is a multi-departmental initiative that provides national training opportunities for youth. The Aboriginal Business Canada program funded by Industry Canada and the Aboriginal Human Resources Development Strategy of HRDC are examples of federal training programs for Aboriginal persons.

Federal and provincial managers stressed that close consultation between federal and provincial staff circumvents situations where the same client receives similar support from two programs. Redundancy in programming actually increases access. Key informants told us that close coordination avoids waste.

4.1.2 Community involvement

Federal key informants reported that many partnerships existed between government, training providers, employers, and the community prior to the CS-LMDA. According to those we interviewed (federal and provincial key informants), the Agreement has enhanced the development of new partnerships. Government, employers, Saskatchewan Institute of Applied Sciences and Technology (SIAST), regional colleges, Aboriginal groups, and community groups are involved in regional needs assessment and planning processes, as well as in program delivery. For example, the Northern Labour Market Committee, created in 1983, identifies and assesses labour market and economic development issues, and initiates action, enabling Northerners to benefit from regional activities. With three co-chairs, this committee includes representatives from First Nations communities, Métis Nation of Saskatchewan, industry/training institutes, and the provincial government Office of Northern Affairs.

Some key informants (federal and provincial) said the transition to the CS-LMDA has been challenging, because partnerships established by the federal government had to be re-established and redefined (n=12). The province has recently developed a framework to define the competitive process for selecting Community-based organization (CBOs) that will receive longer-term contracts.

The survey of employers shows a reported rise in partnerships among employers and the Saskatchewan government, Aboriginal groups, and community groups (n=146). However, the table below also shows that most employers using services under the CS-LMDA typically do not get involved in partnerships. This is quite common, as many businesses see this form of action as irrelevant to their immediate needs.

4.1.3 Service in French

The demand for service in French in Saskatchewan reflects the proportion of the Francophone population for Saskatchewan (2 percent). The client survey found only 11 people out of 1,250 (less than 1 percent) who said they received or wanted to receive services in French from Canada-Saskatchewan Career and Employment Services (CSCES) or Human Resources Development Canada (HRDC). Of these, nine respondents said they were satisfied with the level of service. When asked whether service had improved, stayed the same, or become worse after January 1999, four said it had improved, and three said it was the same. Only one thought it was worse, and two respondents could not compare (1 chose not to respond).

4.1.4 Promotion of Canada's contribution

According to provincial key informants, CSCES centres and promotional materials clearly identify the role of Canada. Early missteps and failure to mention the federal contribution to the CS-LMDA has created negative feeling among some HRDC managers. Federal key informants were particularly concerned about this lack of recognition, but acknowledged that the situation had improved.

Efforts to achieve public awareness of Canada's contribution to PPSs include:

• federal and provincial logos on all literature/advertising;
• joint orientation sessions for clients and employers at CSCES offices;
• the name "Canada-Saskatchewan Career and Employment Services," which implies cooperation.⁴²

Key informants disagreed about whether these efforts have been sufficient. Some federal key informants believe they have been insufficient, while most provincial representatives claim that the province has met its requirements. It is worth noting that using client surveys to test whether the federal profile remains high is not reliable. Most clients of CSCES make only a cursory distinction between the federal and provincial government, and typically think generically of government.

4.2 Information systems and monitoring client progress

At this point, information systems cannot monitor client progress to the extent outlined in the CS-LMDA. The province is currently in the second year of a four-year transitional plan to link Post-Secondary Education and Skills Training (PSEST) computer systems and provide information from all provincial programs. The plan also calls for connecting the provincial network with public post-secondary training institutions (SIAST, regional colleges). While development continues, staff will require training on a new system, and legacy information must be integrated. Our review of the database shows incomplete information in several cases, and some duplication. Comparing the Skills Training Benefit (STB) and One Client Service Model (OCSM) database shows that information is not completely shared between the two systems. Until this system is developed and discrepancies are resolved, it is difficult to exclusively rely on OCSM to provide information on how clients use provincial programs and services.

Key informants also reported that data sharing agreements between the federal government and the province limit the ability of the province to provide information on client progress. To obtain information on the EI status of clients, PSEST staff must use the "screen scraper" process⁴³. Provincial and federal staff report that this system is awkward, and believe strongly that the incomplete data sharing between HRDC and CSCES is a barrier to full service. The federal system (OLIS) currently provides "point-in-time" information on EI eligibility, and is not always up-to-date.⁴⁴ Ideally, the province would like historical information on clients that dates back at least one year. The current process is said to be cumbersome by key informants who have used it.

Provincial key informants reported that their staff frequently contacts HRDC by telephone to verify clients' EI status. In some offices, the communication between HRDC and CSCES is effective. Ex-HRDC personnel in these offices often have "pipelines" to HRDC information by virtue of their personal connections. However, in other centres, this close relationship does not exist, and clients noted that HRDC staff failed to direct them to CSCES.⁴⁵

Key informants (federal and provincial) highlighted other concerns about the current information systems, and the prospect of measuring client progress:

• Uploading provincial information to the National Job Bank has been problematic, because the national system can only display a limited number of characters.⁴⁶ As a result, the national system does not pull over all of the information that appears on the provincial system. This sometimes has consequences for employers who post job orders on the system (for example, individuals will apply for jobs for which they lack the necessary qualifications, because these requirements did not appear on the federal system).
• CSCES staff does not have timely and convenient access to HRDC data on EI clients. Accordingly, staff may not immediately know whether a person is EI eligible. Even when HRDC managed Part 1 and Part 2, several federal managers expressed scepticism that the information systems could support counsellors in directing active clients to Part 2 programs. This represents an ongoing issue in connecting the obligations inherent in receiving Part 1 benefits with Part 2 services, and is not a recent issue that arises with the CS-LMDA.

Key informants (n=25) reported that the provincial information system is still in the development stage.⁴⁷ The system is currently unable to generate aggregate reports that track client outcomes, but the intention is to eventually be capable of doing so. Some key informants observed that tracking client outcomes (i.e., post-program follow-up) is a human resource issue as much as it is a systems issue, and should be viewed in a broader context.

If service providers (regional colleges and SIAST in particular) play a greater role in delivering programs and services in the future, computer databases will need to be fully integrated and developed. Even if the province and the federal government proceed in this direction, several key informants suggested that including information from SIAST and regional colleges would be a daunting task. Although this issue is not central to the formative evaluation, it is critical for the implementation of the LMDA. This information lapse could also compromise the summative evaluation, if data on clients cannot be easily retrieved.

The summative evaluation will begin in April 2002. By then, system integration should be sufficiently advanced that all clients, including EI eligible clients, can be tracked in terms of PPSs received. Furthermore, federal information from the Status Vector file should be available to identify outcomes (Part 1 benefits received and savings to the EI Account).

4.3 Regional variation in service

The difficulty in linking to HRDC data notwithstanding, it is difficult to fully capture current activity in programs from the provincial databases, for three reasons:

1. The province retains information on EI client activity in two separate databases: the STB database (clients receiving the STB for training support⁴⁸) and the OCSM (One Client Service Model) database, which keeps information on "all clients." However, the OCSM database is still in development, and does not record all information about the STB.
2. The two databases serve different functions. The STB database assists with the administration of the program, rather than recording outcomes experienced by clients. OCSM tracks activity and outcomes, but is new and remains in development. These two databases do not similarly classify training interventions, and only OCSM records services.
3. A third challenge is also associated with the recording process. All program information should be stored in the STB database and/or OCSM database. Services, however, are only recorded when a client interacts with a counsellor and that counsellor updates the activity. Many interactions with the system will go unrecorded, particularly when a client drops in to CSCES and does not see a counsellor, or when he/she accesses the SaskNetWork, SaskJobs, or HRDC job bank from a remote site. We reconciled discrepancies by combining survey data with database information, but casual interactions may not be counted and may still be misrepresented overall. Unless general queries are logged (including phone calls) and associated with a client, it is likely that many services will be underreported. Short of asking every caller for a social insurance number (surely an intrusive process that would be rejected by the Privacy Commissioner), this undercount must simply be accepted as inherent in the service delivery system.

With these three cautions in mind, the sections below provide an account of activity to date based on analysis of project databases, the client survey, and interviews with key informants.

4.4 Variation in programs — client survey and databases

Saskatchewan's dispersed population and variations in industry across the province affect how clients use programs.

According to the STB database (n=4,019 clients) (Table 6), most EI clients using the STB for income support participate in a Skills Training program of more than 12 weeks (37 percent), or Apprenticeship (30 percent) or a Skills Training program (25 percent) of less than 12 weeks. Approximately 3 percent of EI clients receiving the STB participate in either a Bridging or Self-Employment program.

• Long-term skill training by those receiving the STB is lowest in Region 5 (Yorkton —SE), where Apprenticeship is more prevalent.
• Regions 1 (North) and 3 (Central) show similar program participation levels, though the latter has higher participation in Bridging programs.
• Region 2 (Prince Albert) and Region 4 (Regina — SW) also have participation levels similar to provincial averages.

The OCSM database (see Table 7, next page) supports a markedly different view of program participation than the STB. This is partly due to incomplete data recording. It also suggests that many who participate in Apprenticeship, Bridging, Community Works, and Work Placement do not receive the STB as a training support. Those in Apprenticeship will often collect EI benefits (Part 1) to support themselves. Work Placement and Community Works participants will typically support themselves through subsidized wages, while Bridging students may receive the Provincial Training Allowance (PTA), STB, or EI benefits, or receive no income support.

• In the OCSM database (n=4,079 programs), Apprenticeship programs were most common (71 percent), followed by Bridging (22 percent), Work Placement (22 percent), and Community Works (9 percent).
• From Table 7, Region 4 (n=1,227) had the highest proportion of Apprenticeship programs (95 percent), followed by Region 3 (n=1,885) (74 percent) and Region 2 (n=473) (64 percent). No Apprenticeship participation appears in Region 1 (n=227), while minimal participation exists in Region 5 (n=267) (8 percent). Apprenticeships are the result of participant choice and direction of clients by the Apprenticeship Board. Apprentices must complete a specified number of hours of work with an employer after completing their course. Kelsey (Region 3), SIAST Palliser (Region 1), and SIAST Woodland (Region 2) receive the highest number of designated apprenticeships.
• Bridging programs were most common in Region 5 (76 percent). This may reflect growth in the retail and tourism sector in the area, where employers need quick transitions to the workplace to meet the high demand for workers.
• For Region 1, which encompasses the northern part of the province, Community Works and Work Placement programs are most common. This likely reflects the lack of economic opportunities outside of key industries (mining, forestry, outfitting/tourism).

4.5 Variation in services — client survey and databases

Not all EI clients require training to re-enter the workforce, as many already have the skills required by industry. Others may combine a service (such as searching for work on the Internet or talking with a career counsellor) with short or long-term training. For the purpose of both the formative and summative evaluation, any EI client who uses either training or a service offered through a CSCES office is considered a "participant," as he/she is actively trying to get back into the labour market.

Two sources provide information on participation in provincial services: the OCSM database and the client survey. The OCSM database only records a program or service when a client interacts with a counsellor and that counsellor updates the records. The participant survey reported below reveals that many interactions with the system probably go unrecorded, particularly when a client drops in to CSCES and does not see a counsellor, or when he/she uses the SaskNetWork, SaskJobs, or HRDC job bank from a remote site (e.g., home computer). This under-recording of activity by OCSM is unavoidable, as staff cannot record all activity unless clients report all their activity to counsellors on a regular basis, and client service is interrupted for the counsellor to enter data.⁴⁹ The under-reporting makes it difficult to estimate the true value of services such as counselling, brief advice on the phone, or client use of the web sites.

According to the OCSM database (n=11,962 services recorded), the most commonly used service is counselling (86 percent). Other services recorded include orientation (19 percent), referrals (13 percent), job search (6 percent), and work preparation (4 percent).

Differences across regions may reflect the type of clients in each area, as well as differences in CSCES operations and the practices of counsellors when recording interactions. For example, based on the data, it would appear that counselling is mandatory for all EI clients in Region 3 (n=3,678 services recorded) (96 percent of clients have counselling listed), whereas it is much lower in Regions 1 (n=588 services recorded) and 4 (n=3,735 services recorded). Whether this is due to coding practices should be investigated further during the summative evaluation. Clearly, the reliability of data entry and the standards/practices used in different CSCES offices may have a profound impact on the evaluation.

• Region 1 (n=588) shows the highest proportion of referrals (23 percent) and job search activity (25 percent).
• Regions 2 (n=1,654) and 3 (n=3,678) have a similar proportion of clients accessing services, with most receiving career counselling.
• Region 4 (n=3,735) has the highest proportion of orientations recorded (46 percent), and a relatively high number of referrals (21 percent).

The client survey (sample of 1,250) also records use of CSCES services, capturing information in a different way than the OCSM database. Based on the survey, two-thirds (67 percent) report talking to a counsellor about training, while 62 percent said they read information on training and education programs. Approximately half used a computer at the centre to search web sites for a job, while the same proportion talked to a career/employment counsellor to plan a strategy for returning to work. One-quarter of participants (24 percent) phoned the centre to enquire about job openings, while the same number used multimedia products or the computer to write a resume.

4.6 Variation in program and services — key informants

Most key informants said that although all field offices deliver all programs and services, the actual PPSs used vary by region (n=15).⁵⁰ For example, clients in the North tend to be SA recipients rather than EI clients, and tend to have multiple barriers to employment compared to clients elsewhere.

Although some key informants expressed uncertainty about whether clients using PPSs represent eligible EI clients, many suspected that a large fraction of EI clients may not be using PPSs for the following reasons:

• they may not wish to take training while on EI (e.g., seasonal workers)
• clients may be unaware of the services available, especially in rural and remote areas
• those with more experience may require less help in locating employment, and may be waiting for suitable employment to turn up through their trade association or union hall.

Many provincial key informants noted that the province has no way of knowing the identity of eligible EI clients in a region, and therefore cannot invite them to use PPSs (n=13). Some also doubted whether all HRDC staff direct clients to the provincial offices. However, as we note below, the client survey indicates that an HRDC/EI counsellor referred 30 percent of the 804 respondents. HRDC is reportedly considering an insert with EI applications to advertise PPSs.

Key informants said a variety of promotional methods have been undertaken to inform clients and employers about PPSs, including distribution of written materials, TV and radio advertising, posting information on the Internet, grand openings of CSCES offices, presentations to stakeholders, and joint orientation sessions conducted by provincial and federal staff (n=28). They rated direct person-to-person contact and word-of-mouth as the most successful promotional techniques. Many key informants emphasized the need to reach rural and remote areas.

4.7 Demographics of EI clients

An important issue for the formative evaluation is to understand the demographic composition of EI clientele (n=12,753)⁵¹:

• Approximately 70 percent of EI clients are male, and 30 percent are female. Similar proportions exist across the regions.
• Of the 8,164⁵² participants almost all report receiving service in English, with less than 1 percent being served in French.
• Approximately 4 percent of EI clients are reported to have a disability, with little regional variation.
• Approximately 69 percent of EI clients have no dependents, 23 percent have one or two dependents, while 8 percent have three or more dependents.

• Approximately 7 percent of EI clients are of First Nations ancestry, 6 percent are Métis, and 1 percent is Non-Status Indian. Proportions differ considerably across regions, with a higher number of Aboriginal persons living up North in Region 1 (69 percent) of EI clients), compared to an average of 9 percent in Regions 3, 4, and 5.

• Of the 7,571 records in the OCSM database reporting level of education, approximately 26 percent of EI clients have less than a grade 12 education, 46 percent have grade 12 or its equivalent, 18 percent have a technical diploma or apprenticeship certification, and 11 percent have a university degree. Educational levels are lowest in Region 1, where 43 percent have less than grade 12. Southern parts of the province (Region 4 and Region 5) show relatively higher degrees of education. Educational information on clients is not available for all EI clients, or the information may be dated. The STB database does not record education levels.

4.8 Roles of partners

Although a few key informants (from federal and provincial governments) said the roles and responsibilities of the federal and provincial governments are clear to them (as senior managers), more said these are not completely clear, either to management or to frontline staff (n=11). They singled out federal/provincial responsibilities relating to Aboriginal people, youth, and people with disabilities as areas where ambiguity may exist (n=15).

A majority of key informants said federal and provincial roles and responsibilities are also unclear to clients and the public (n=13), and they encouraged better marketing of this relationship. Disagreement exists over whether this confusion is a problem, as some felt that current promotional efforts are adequate.

A key question is whether managers direct Part 1 clients to CSCES centres (Part 2 programs or the PPSs). Based on the client survey, 30 percent of those who have visited a CSCES office (n=804) were referred by HRDC/EI counsellors. More often, clients learned of CSCES through word-of-mouth or from family or friends (33 percent), or an advertisement (8 percent). The remaining 29 percent reported a wide variety of referral processes, or could not recall how they had first been referred.

4.9 Annex 3 and 7 arrangements

Service delivery arrangements outlined in Annex 3 of the CS-LMDA have been met. CSCES offices are in place across the province, delivering programs and services to clients.

Annex 7 of the Agreement outlines provisions for transferring federal staff to the provincial government, governed by the Employee Transfer Agreement and negotiated between the federal government, the government of Saskatchewan, and the Saskatchewan Government and General Employee's Union (SGEU). According to provincial data, the CS-LMDA resulted in the transfer of 97 HRDC staff (full-time equivalents) to the government of Saskatchewan.

Both federal and provincial key informants acknowledged that human resource/organizational culture issues exist with amalgamating federal and provincial staff. According to several key informants, some former HRDC staff working for the province wish to return to HRDC (n=17). Other ex-HRDC staff we interviewed apparently believe they have less discretion to make decisions. However, most ex-federal staff who reported this view also reported that provincial practices are changing, as experience is gained in delivering programs and services.

4.10 Collocation

In general, managers and staff from both orders of government support the concept of collocation. No empirical measure of the incremental impact is possible, but staff note that it should lead to better service. Technically, only one CSCES is truly collocated (North Battleford), but in other areas, HRDC offices and CSCES are in close proximity (close collocation).

In cases where close collocation has occurred, key informants believe that clients experience better service (n=25). This improvement is mainly due to the perception of increased convenience to clients for accessing several services or agencies in one location (n=21). Other positive effects of collocation reported by key informants include increased efficiency and promotion of partnerships/cooperation.

Perceived difficulties arising from collocation are sensitivities around information sharing and use of different computer systems. A few of the 41 key informants said collocation with the regional colleges has been problematic. Originally, the province wanted to deliver PPSs through the regional colleges and SIAST, but HRDC regarded these as third parties and did not permit their inclusion in the CS-LMDA as a government-to-government agreement. This has reportedly caused some bad feelings on the part of regional colleges.

Employers offered their views on collocation in their survey. Among those surveyed, 48 percent (n=70) reported awareness of the collocation of HRDC and provincial offices. Among these employers, 43 (61 percent) said they had visited one of these offices. This number seems high, given the limited number of collocated offices. It is possible that some thought that CSCES offices were all collocated given their name (Canada-Saskatchewan), and that they were comparing CSCES to the collocated sites.

Although the sample size is small, employers report that collocated offices provide better service, a higher level of service, and greater administrative efficiency than non-collocated offices.

4.11 Summary on implementation

The PPSs include all the major elements required in the CS-LMDA. Services exist throughout the province, although some variation is apparent. Given the newness of these services, this is expected. Service in French is available where numbers warrant (Prince Albert, Moose Jaw, Saskatoon, Regina), but insufficient recorded use of these services has occurred and their adequacy cannot be assessed. Some redundancy exists between the services offered by the province and the pan-Canadian initiatives (youth, Aboriginal, and persons with disabilities), but senior management (federal and provincial) believe that this does not imply overlap and duplication. Close coordination ensures that the same resources are not directed to the same clients. Staff support the redundancy as offering improved access.

The CS-LMDA seems to be associated with improved partnerships and consultation. It is difficult to attribute this to the Agreement. The needs assessment and sector partnership processes progressed independently, and predated the Agreement by several years. The nature of the process has evolved under the CS-LMDA, in that CSCES offices (created as a result of the Agreement) now lead the process.

Some areas of implementation work remain:

• The contribution of Canada remains under-recognized, according to some federal managers.
• The information systems and data sharing need substantial enhancement to meet the requirements for evaluation and accountability specified in the Agreement. Both the province and the federal government are aware of this, and recognize the urgency in enhancing the information systems and data sharing, both to improve the operation of the PPSs and to support the summative evaluation.

Many questions posed in the evaluation framework regarding success, impacts, and effects require information from Human Resources Development Canada (HRDC) on all Employment Insurance (EI) clients (active and reachback) to support a nonl./participant survey. Due to delays in obtaining a sample frame of non-participants, this work could not proceed as part of the formative evaluation. A non-participant survey will be completed as part of the summative evaluation, and in this report we address issues of outcome in a preliminary way, based solely on information provided by the clients.

5.1 Clients and employment/self-sufficiency

The self-reported impact of Provincial Programs and Services (PPSs) on participants' economic self-sufficiency varied. About half of the focus group participants said the training they took helped them to find a job, and that they found work in the area for which they were trained. Those who were involved in an apprenticeship program noted that their training began with their employment. With few exceptions, those who were still in training expected that the training would help them to find employment in the future. The client survey supports this finding, with 93 percent (231 of 248) of Apprenticeship participants reporting that their training increased their chances of employment; 83 percent (99 of 119) of Work Placement, Community Works, and Work-based Training participants report the same. Given that the client survey was based on those receiving training or services as of April 1999, outcomes are relatively short-term.

A few focus group participants said the training they took did not help them to find work. In some cases, participants believed they confronted "ageism" on the part of employers, and were pessimistic about their potential to find meaningful work in the future. Others observed that obtaining employment is difficult, and requires connections: "it's who you know, more than your qualifications." Participants in both urban and rural centres made this observation. The "local" nature of labour markets is a serious issue for some: "unless the employer knows you, you cannot get a job there."

Several focus group participants criticized what they perceived as an emphasis on obtaining employment, regardless of the type or quality of the work. However, others pointed out that building up one's resume is the most important consideration, no matter what kind of employment is sought. These focus group participants appeared willing to do short-term work as a stepping-stone to more permanent employment.

The summative evaluation will provide more concrete measures of self-sufficiency.

5.2 Factors in finding and maintaining employment

People apply for EI for different reasons. For some, EI represents a temporary break in employment, when an employer does not have enough work for the entire year. This benefits employers, who maintain access to skilled workers. Others find themselves in transitional jobs where skill development is minimal, and periods of unemployment are common. Still others work in a specific industry or with a particular employer for a long time, but find themselves out of work due to technological change or loss of competitiveness. Medical conditions can also cause an individual to be unemployed and/or to seek work in another profession or trade, particularly one that may be less physically demanding. Depending on how an EI client views employability and future job expectations, he/she may choose to upgrade their skills, look for work, or wait until their previous employer rehires them.

The client survey reflects these diverse situations (n=1,092). Half of the client survey participants said they were laid off either when their job ended (25 percent), or when the employer downsized or closed (30 percent). Nine percent said they quit to go to school, 8 percent left for medical reasons, and 5 percent went on maternity leave.

When comparing reasons for leaving jobs post-EI (n=298), it appears that many clients still work in transitional employment where lay-offs are common (either seasonal, casual, or contract work) or employment ends when a company closes or downsizes (35 percent). Others will leave their employer to upgrade their skills or take a better job (5 percent, 28 percent respectively). It appears that for the pre-EI job, employer downsizing (lay-offs) is the most common reason for leaving a job. In general, clients reported returning to school as the reason for leaving their most recent employment, which may be the last of several jobs held since their most recent EI cheque.

Clients surveyed also assessed their own job search behaviour when they qualified for EI (n=1,092). Approximately 22 percent reported looking for work right away, while 18 percent said they looked for ways to upgrade their skills. A further one-third of respondents said they waited for a period after becoming unemployed, either because they needed a rest (14 percent) or thought they would be recalled at some time in the future (22 percent). Nine percent (9 percent) noted they were off work due to medical reasons, while 5 percent were off work due to pregnancy.

5.3 Wages and job retention before and after PPSs

Almost 70 percent of respondents report being employed at some time since their most recent period on EI (n=864). Of these, 59 percent reported having one job, 23 percent had two jobs, and 16 percent had three or more jobs.

EI participants reported slight increases in earnings following their training. Prior to their most recent period on EI, clients reported average earnings of $1,881 per month (n=933). Following a period on EI in which they received training or accessed services, clients reported average earnings of $1,901 per month (n=736). These changes are not statistically significant.

Clients were also asked whether they had any other employment following their most recent period on EI, aside from their most recent employment. The average monthly wage for this job was $1,450 (n=255). Wages were lower than their other (most recent employment) because more of these jobs were part-time, contracts, or casual employment and may reflect interim employment.

With reference to Table 18 (next page), about one-quarter (24 percent) of the 908 clients that reported a duration of employment had worked for at least three years at a single job before going on EI. Of these, 6 percent had worked at the same job for more than 10 years, 26 percent had worked at their pre-EI job for one to three years, while half were employed for less than a year before going on EI.

Most EI clients (81 percent) report being employed at their most recent job for up to six months, while a further 9 percent have been employed for one year or less (n=173). These are expected findings given the time frame of the study, which focused on clients who have participated in a program (e.g., taken training) or received a service at some point after April 1, 1999.⁵³

Between the time they began their most recent period on EI and the present, most of the 1,092 clients reported that they relied on EI benefits (42 percent) or earnings from employment (39 percent) as their main personal source of income (see Table 18). A few used Social Assistance (5 percent), personal savings (3 percent), or Student Loans (3 percent). Approximately 45 percent of EI clients also reported income from their spouse or family.

Employers reported on trainee departures from Work Placement, on the job training (JobStart/Future Skills), or Community Works programs. Approximately one-third of the 124 employers surveyed found their employee unsuitable, and a similar proportion (32 percent) said that participants found other work. Changing business conditions (15 percent) and lack of funding for training (12 percent) were also commonly mentioned as reasons why employees leave after the program ends.

5.4 Results of self-employment programs

PSEST created the self-employment program to coincide with the introduction of the LMDA.⁵⁴ Self-employment programs are designed for clients with skills that can be directly targeted to a niche market. HRDC offered the self-employment program to EI clients in the pre-LMDA period.

Key findings on this program from the client survey are as follows (n=37):⁵⁵

• Most clients (78 percent) reported thinking about starting their own business at least one year ago; of these, one-third (32 percent) said they first thought about it 10 years ago.
• Seventy-six percent (76 percent) said they received training or workshops to help them start their own business while they were involved in the self-employment program.
• Of the 37 respondents, 27 (73 percent) started their own business since January 1999. In most cases, EI clients are sole proprietors (78 percent).
• Ninety-three percent (93 percent) of businesses (n=25) are still in operation. Of the two businesses that are no longer in operation, one closed due to illness and the other because the business was not profitable.
• Twenty percent (20 percent) of clients in this program report working up to 34 hours per week, 28 percent work between 35 and 40 hours per week, while 40 percent work more than 45 hours per week. On average, EI clients in self-employment programs report working 43 hours per week.
• Ninety-two percent (92 percent, n=23 of 25) employ only themselves. One company has three other paid employees, while the other has one other paid employee.
• Sixty-four percent (64 percent, n=16 of 22) said none of their household income is derived from earnings/drawings on the business, suggesting that these businesses are still unable to generate income for the owner. One person draws all their income from the business, while six people derive 5 percent to 40 percent of the household income from their business.
• Seventy-five percent (75 percent, n=18) said that they had no other employment aside from their own businesses. Six respondents earned $500 to $2,400 per month from other employment.

5.5 Satisfaction with PPSs

5.5.1 Overall client satisfaction with PPSs

Based on focus groups, case studies (dyads), and the client survey, EI clients experience high levels of satisfaction with PPSs. In particular, clients expressed support for CSCES offices, created as part of the CS-LMDA and the government of Saskatchewan's own reorganization of programs and services. Most clients report that counsellors are available, and provide helpful information. That said, a few EI clients reported difficulties with their counsellors. This is hardly surprising, given the sample size and the fact that most find unemployment to be a trying experience.

Clients are also generally satisfied with the training programs in which they have participated. The summative evaluation will offer clearer indicators of program satisfaction and effectiveness.

Key findings from the survey of participants include the following:

• Of the 1,017⁵⁶ participants surveyed, twenty-nine percent (29 percent) reported that the programs and services received during the most recent period on EI "met all their needs," while 49 percent said that it "met some of their needs." Sixteen percent (16 percent) reported that "none of their needs" were met, while 7 percent did not know.
• Almost half of the participants questioned (47 percent) reported that programs and services were accessible and conveniently located. A further 38 percent indicated that they were accessible, at least to the extent that they met some of their needs.
• One-third of 1,017 participants (34 percent) obtained enough information about programs and services to meet "all" of their needs, while half (51 percent) said the information available "met some of their needs." The remainder states that the program met none of their needs.
• Of the 576 EI clients who met with a counsellor, forty-three percent (43 percent) said the counselling they received "met all their needs," while 39 percent said it met "some of their needs." Eighteen percent (18 percent) reported it met "none of their needs."
• Most clients participating in Work Placement (Work Placement, Community Works, on the job training) and Apprenticeship programs reported on whether they thought the program had increased their chance of employment. Ninety-three percent (93 percent) of Apprenticeship participants (n=248) and 83 percent of Work Placement participants (n=119) responded affirmatively.
• Despite the overall positive rating of PPSs, some 40 percent of the 613 respondents currently working said the programs and services they received were "not at all helpful" to them in getting their present job. Compare this with 25 percent who said that PPSs were "very helpful," and 31 percent who said that they were "somewhat helpful." Those who took programs were more likely to rate PPSs as very helpful (110 of 359; 31 percent) compared to those who only accessed services (45 of 254; 18 percent).

• During focus groups, participants expressed varying levels of satisfaction with provincial programs and services; those trained and subsequently employed in their chosen field expressed largely positive opinions about their experiences. However, several participants expressed dissatisfaction with certain aspects of PPSs, such as the quality of training received, length of training, the poor advice offered by CSCES counsellors, and problems finding training that would lead to employment.
• Some participants noted that CSCES counsellors are very busy, and that it is difficult to meet with a counsellor in person. "You can't just drop in, you have to make an appointment." Others noted that the hours are not convenient for people who may work part-time, but still need to secure their employment.
• Although some participants praised the self-help resources available at CSCES centres, others commented that computerization can lead to a diminished rapport with people: "Some people need and want to have a greater amount of personal contact."
• A few participants said they have experienced ongoing follow-up and assistance from the organization that trained them, but many others remarked upon a lack of follow-up on the part of CSCES after completion of their training program. "The government spent a lot of money on me and they haven't done anything to determine whether it was worth that, and whether it was worth a year of my time." Some of these participants have taken the initiative to maintain a relationship with their counsellors, keeping them up-to-date on their progress.
• As already noted, some participants complained about training programs that failed to meet their individual needs, interests, or capabilities. Other sources of dissatisfaction included course prerequisites that changed as participants were completing a program; poor quality training programs ("dated and overpriced");and short-term funding for self-employment programs that did not accommodate the time it takes to establish and stabilize a business.

The dyads offer some evocative evidence of the CS-LMDAs and provincial programs and services. We interviewed five couples about their experience with the programs and services. We were especially interested in understanding how training affected the lives of trainees and their families. We also wished to identify factors in successful outcomes for families.

• All trainees and spouses endorsed the program and were very satisfied with the way they had been supported to increase their self-sufficiency. All trainees completed their programs and were either employed or had excellent prospects.
• Each family reported that the training required sacrifice for the family, but trainees all acknowledged that their spouses and other family members had been supportive. Several commented that this support was essential to their success.

5.6 Use of case/action plans

Based on the client survey, about one-third of participants (n=429) said counsellors required them to prepare a case/action plan before beginning training. A similar proportion of clients completed case/action plans in each region. Focus group participants indicated that the concept of a case/action plan is new to EI clients in some centres; in other locations, most PPSs participants had developed a case/action plan before beginning training.

The nature of the case/action plan varied. Some focus group participants reported that counsellors required them to complete a simple assessment process consisting of either a single test or a series of tests designed to assess their suitability for particular types of work (i.e., personality tests, aptitude tests, etc). Others remarked that they completed the plan to get into the courses, but then never looked at them again. Still other participants reported vague recollections of the plan.

At the other end of the spectrum, some focus group participants reported that staff asked them to conduct a market study, typically consisting of identifying and contacting potential employers to determine the demand for workers in that field, the prevailing salary levels, and the credentials expected. Those who completed the market study said it was "a good exercise — it gives you a good idea of the potential of getting a job and also whether the career is appropriate for you." Participants who prepared these types of case/action plans generally believed these were helpful in determining the type of training they should take, and gave them realistic expectations about work in the field they wished to enter.

The client survey supports focus group findings about preparation of substantive plans. When asked which steps were involved in their back-to-work plan, those clients with case/action plans reported the following (n=429):

• Seventy-one percent said they "prepared a list of potential employers," while a similar proportion "identified programs to upgrade their skills."
• Sixty-nine percent said they "set up steps along the way for meeting goals," while 68 percent said they "prepared a new resume."
• Fifty-eight percent reported they "researched wages or demand for people in the area that they were interested in training for," while 47 percent learned how to use the "HRDC kiosk or the Internet to search for job orders."
• Forty-two percent identified a program to upgrade their basic education, while one-quarter (26 percent) took an aptitude test.

• Of those who talked to a counsellor about a back-to-work plan, 28 percent had completed such a plan, 53 percent were part way through, and 18 percent stopped part way through or never started the steps.

5.7 Exit and attrition rates for training

Employment Insurance clients have relatively high completion rates for their training, irrespective of the type of training. For most programs, fewer than 10 percent of clients quit. As shown in Table 22, Work Placement programs (n=119) have the highest discontinuation rate at 13 percent, although this is still low when compared to PPSs that serve Social Assistance clients, where a 40 percent discontinuation rate is common.⁵⁷ Benchmarks for EI clients provide a better comparison, and a review of federal databases may provide this comparison in the summative evaluation.

Clients offered several reasons for discontinuing programs. Those in Work Placement programs said they left to take a better job, to take other training, or because they were laid off. In other programs, clients left due to lack of financial support, medical reasons, lack of childcare, or personal reasons.

View TABLE 22

The employer survey also measures post-program impacts.⁵⁸ As shown in Table 23 below, almost two-thirds of program participants are hired full-time (61 percent) or part-time (65 percent) following program completion when participating in a Work Placement/wage subsidy program. Retention rates are somewhat lower for Work-based Training programs, where just over half (53 percent) are retained after the program. This may reflect the relatively higher skills that EI clients receive in this program and their ability to find employment elsewhere. Employers participating in an Apprenticeship program report 94 percent continuing on after program completion.

5.8 Client financial participation in the PPSs

Four main types of income support exist for EI clients participating in training: EI Benefits (Part 1), the Skills Training Benefit, Student Loans, and the PTA. The type of income support depends on the type of training, and whether or not the client is still collecting Part 1 benefits. Clients enrolled in Basic Education are most likely to support themselves on EI benefits or to receive the PTA as income support. For other programs, the Skills Training Benefit is typically used, particularly when a client has exhausted their EI benefits and is classified as a "reachback" client. Although some list Student Loans for Basic Education and Bridging, this may reflect misclassification of training type in the database, as these programs are not typically supported by Student Loans. Some participants also list Social Assistance, but it is no longer used to support clients in training since the introduction of the PTA, except for particular circumstances (e.g., short-term training of less than four weeks).⁵⁹

As shown in Table 24, EI clients are more likely to take a skills training course (n=380) or an Apprenticeship program (n=248) while still collecting EI benefits (Part 1) (66 percent and 79 percent of participants respectively). While 52 percent of EI clients said they received EI benefits (39 percent) or the STB (13 percent) while in a Work Placement program (n=119), 39 percent said they received no income support/employment, which is expected, given that most are paid a subsidized wage while participating in these programs.

Additional financial support provided by the Skills Training Course took the form of grants for tuition or books (51 percent). (See Table 25.) Work Placement and Self-Employment participants report the least additional support, with some receiving coverage for transportation or tuition.

View TABLE 25

Although many participants surveyed said they would not have taken training without funding, others said they would have proceeded with training anyway (Table 26). Clients were least likely to proceed without support in a Skills Training (65 percent of 380), Bridging (58 percent of 81), or Self-Employment program (58 percent of 37).

View TABLE 26

Focus group participants supported this finding. Variation exists in the amount of income support received by participants and their willingness to contribute to the cost of training. Several participants thought the level of funding was inadequate, particularly those with families and spousal income. During client focus groups, some clients expressed uncertainty about the source of their funding. Given the range of programs, types of income supports, and circumstances of EI clients, this is not surprising.

Clients surveyed by telephone report contributing some money to support their own training. Those in Basic Education reportedly contribute $110 per month (n=87); those in Skills Training (n=380) contribute $321, while those in Bridging (n=81) contribute $84 per month.

As seen in Table 28, some clients reported supplementing their income by working while in training. Clients in a Basic Education (n=87) or Self-Employment program (n=380) were most likely to supplement their income. Higher proportions of clients in these programs receive income support (PTA or STB), and may be looking for additional income.

View TABLE 28

As shown in Table 29 (next page), EI clients who work while in training earn approximately $1,216⁶⁰ per month. Sample sizes for some courses are relatively small, and results should be viewed with caution.

View TABLE 29

5.9 Client service before and after the CS-LMDA

It is difficult to compare client service before and after the CS-LMDA without a baseline survey of client perceptions prior to the Agreement. Based on client feedback during focus groups and the survey of clients, most are satisfied with the services they have received in the period since the Agreement came into effect.

As noted in Table 30 below:

• of those who talked to a counsellor about training (n=539), 87% found it to be "very helpful" or "somewhat helpful";
• of those who read information on training and education (n=496), 88 percent found it to be "very helpful" or "somewhat helpful";
• of those who used a computer at the centre to search web sites for a job (n=412), 85 percent found it to be "very helpful" or "somewhat helpful";
• a similar proportion (85 percent) found CSCES to be "very helpful" or "somewhat helpful" when using multimedia products or the computer to write a resume (n=195);
• the lowest recorded rating was among those who phoned the centre to enquire about job openings (n=192); 71 percent found CSCES to "very or somewhat helpful," while a similar proportion (23 percent) found it "not at all helpful."

Clients also rated the national (HRDC) web site (n=626) and the SaskNetWork/SaskJobs web site (n=320). Seventy percent (70 percent) of respondents reported that the national/HRDC web site met all (19 percent) or some of their needs (51 percent). This compares with 71 percent who said that the SaskNetWork/SaskJobs web site met all (23 percent) or some of their needs (48 percent). Based on this indicator, the level of service is rated as being the same for both the federal and provincial web sites.⁶¹

On the survey, clients compared programs and services used most recently with those used prior to January 1999 (n=216). Thirty-eight percent said that programs and services were better, 39 percent said they were the same, 9 percent said they were worse, and 13 percent were unsure.⁶²

These preliminary findings suggest that the current level of service meets or exceeds the levels provided prior to the CS-LMDA.

5.10 Employer use and satisfaction with PPSs

Based on the survey, employers report satisfaction with programs targeted to EI clients. Programs have helped employers "fill job vacancies" (63 percent), "fill skill shortages" (49 percent), and "meet human resource needs" (54 percent). Other findings from the survey of employers survey are as follows:

• Forty-six percent (46 percent) of respondents said without the program, they would not have hired an employee ("strongly agree" or "agree"), while 55 percent said they wouldn't have "provided on the job training."
• Seventy-four percent (74 percent) of respondents said that "program participants fit into their workplace."
• Fifty-five percent of employers report that "the program helped improve their business," while 47 percent said that "having programs specifically targeted to EI clients helped them locate employees."

5.11 Satisfaction with Labour Market Information (LMI)

As part of the CS-LMDA, the governments of Canada and Saskatchewan agreed to develop a joint labour market information strategy that built upon the experience of the Canada-Saskatchewan Strategic Initiative projects. Under the CS-LMDA, Saskatchewan is responsible for producing the Saskatchewan labour market information needed to assume the responsibilities conferred by this Agreement, as well as disseminating labour market information (within Saskatchewan) relating to the implementation of this Agreement, and participating in and maintaining a link with the National Labour Market Information System. Some noted that unlike HRDC, the province does not have the resources to employ labour market analysts, which affects its ability to provide longer-range forecasting.

A number of projects currently support the continued development of the joint Canada-Saskatchewan strategy, such as:

• Saskatchewan Labour Market Planning and Information Strategy: Saskatchewan Post-Secondary Education and Skills Training has initiated a process for the development of a Labour Market Planning and Information Strategy for Saskatchewan, with HRDC Saskatchewan Region as an active participant. The strategy will support the provision of labour market planning and information within the province by facilitating linkages among the labour market planning processes and coordination of labour market information activities.
• Saskatchewan Employment Demand Forecast: This forecast provides a detailed five-year industry and occupation employment projection. A working group comprised of Post-Secondary Education and Skills Training, HRDC Saskatchewan Region, Saskatchewan Economic and Cooperative Development, and Saskatchewan Labour prepares the forecast annually.
• Overview of the Saskatchewan Economy and Labour Market: This is being developed to provide a provincial overview on demographic, economic, and labour market trends for job seekers, career explorers, students, employers, counsellors, planners, and analysts.
• Trades Forecasting: Work was initiated on developing and undertaking occupational and employment forecasting for the apprenticeable trades in Saskatchewan. This information will be useful for human resource and development planning, as well as for identifying training delivery and capacity issues. The working group is comprised of representatives from Saskatchewan Post-Secondary Education and Skills Training, the Saskatchewan Apprenticeship and Trade Certification Commission, HRDC Saskatchewan Region and Economic and Cooperative Development.
• SaskNetWork Web Site: An Internet web site, SaskNetWork, is the first stop for Saskatchewan career, employment, and labour market information. It is also part of a network of partners in all provinces and territories working together to develop an online career, employment, and labour market system known as Canada WorkInfoNet.
• Saskatchewan Job Futures: This is an on-line resource for finding out about education and training requirements, work duties, wage information, and employment trends for hundreds of Saskatchewan occupations.

The CS-LMDA also states that the Government of Canada "will continue to be responsible for the National Labour Market Information Service, including the National Labour Market Information (LMI) System with which it will produce and disseminate the labour market information it needs to exercise its responsibilities recognized under this Agreement and those associated with management of the Employment Insurance Account and with encouraging inter-provincial mobility."

In general, the overall LMI strategy in Saskatchewan has drawn a mixed reaction from managers and staff, though most made generally positive comments about these processes. Negative comments appear to reflect a disconnection between the activities of provincial economic planning units (e.g., Department of Finance) and the regional offices of PSEST. They may also reflect the difficulty in disaggregating labour market data to the regional level. National surveys, such as the Labour Force Survey, simply do not have much accuracy at the regional level. When looking to secondary data sources, those carrying out regional needs assessment are forced to rely on periodic surveys that may not encompass the entire labour market.

Under the CS-LMDA, the federal government's responsibilities include the development and dissemination of National Common Products (NCPs) as part of HRDC's Pan-Canadian local LMI Service. HRDC Saskatchewan Region's local LMI products can be found on the Saskatchewan HRDC web site. In addition, SaskNetWork links to HRDC's local LMI.

Employers are moderately satisfied with LMI provided through CSCES (n=39), but less satisfied with Internet-based information (n=50). Employers rate CSCES as 3.2 (on a 5-point scale) when it comes to "finding labour market information," compared to ratings of 2.8 and 2.9 for the HRDC web site and the Saskatchewan government web site. Several key informants (federal and provincial) also observed that the federal and provincial governments do not completely agree on who is responsible for labour market information (n=17).

5.12 Satisfaction with Labour Exchange processes (LEX)

The other resource for employers and clients is the Labour Market Exchange (LEX), where clients search for job openings and employers post positions or look for potential employees. Based on interviews with key informants as well as focus groups and surveys, evidence exists that the LEX process needs improvement. Clients participating in both the survey and focus groups state that word-of-mouth and classifieds remain the best information sources for job hunting. A smaller proportion rely on either the HRDC or Saskatchewan (SaskNetWork/Sask Jobs) government web sites to find work.

Regional differences also exist. According to focus group participants, small town labour markets are very personal, and job hunting requires that "you know someone." The isolation of the rural job seeker remains an important structural imperfection in the provincial labour market.

Employers also noted areas where the LEX process could be improved (Table 33). Again, they reported CSCES centres to be a useful source for posting a job vacancy (mean rating of 3.8), obtaining information about an education/training program, and locating potential employees. Both federal and provincial Internet services rated lower for all three categories of service.

Focus group participants also reported problems with Internet-based services. Several clients said that job postings were sometimes out of date. Approximately half of employers knew of the services, but few had used them. Those who had used the services found the job order process difficult, as it required sifting through too many applications. Several employers said they preferred referrals, as it made the hiring process more effective. Further promotion of the web site to employers could increase its use and functionality.

5.13 Summary of findings

In general, while clients are satisfied with the programs and services offered under the CS-LMDA, many did not believe PPSs assisted them in finding work. Clients in rural areas see themselves as isolated and caught in local labour markets, where personal contacts are essential to finding work. Many of these clients are unable or unwilling to leave their families to seek work in another centre.

The back-to-work experience post-EI is diverse. About 40 percent of the 1,092 participants who responded to this question reported seeking work or training upon their most recent qualification for EI. This suggests that 60 percent of EI clients may delay their use of PPSs.

Many EI clients have several jobs post-EI, reflecting the fact that casual and part-time work is common for these individuals. About half rely on spousal income as a source of income, in addition to the Part 1 benefits and employment.

Case/action plans varied in content and the extent to which clients and counsellors used them. Clients who reported doing a survey of the job market found this experience a worthwhile part of preparing a case/action plan.

Clients in Skills Training, Apprenticeship, and Bridging programs typically rely on Part 1 EI benefits for income support. Those in Basic Education tend to also use the PTA for support. Between 9 percent and 30 percent of clients used no income support for their training.

For the most part, employers are satisfied with PPSs, but are less satisfied with the labour exchange process. Clients see word-of-mouth and personal contacts as more useful in finding a job than the labour market exchange process.

The formative evaluation concentrates on process and implementation. Any comments on results and outcomes are preliminary and require confirmation by the next phase, where client follow-up surveys and non-participant data will allow us to infer program incrementality.

These observations are organized by evaluation theme.

6.1 Rationale

Fundamental to an understanding of the Canada-Saskatchewan Labour Market Development Agreement is appreciating that Employment Insurance (EI) Part 2 programming needed to be integrated into a pre-existing provincial labour market development program. The CS-LMDA is an example of a devolution process, as opposed to Human Resources Development Canada (HRDC) programs and services that are co-managed by the federal and provincial government.

Coinciding with the LMDA process, the province created the Saskatchewan Training Strategy (STS). In many ways, this strategy aligns with current thinking on optimal processes for training unemployed and economically disadvantaged persons to support their growth to self-sufficiency.⁶⁴

Central to the STS and its successor, the Sector Strategic Plan, are the following concepts:

• The publicly funded post-secondary education and training sector is pivotal to providing accessible, high quality training programs in Saskatchewan. An array of community-based organizations, private vocational schools, and private trainers augment the programs and services available to Saskatchewan adults through the publicly funded post-secondary institutions. All parts of the system should be supported to help ensure adequate capacity within it as well as quality and access to relevant labour market related training programs and services.
• The post-secondary education system must provide programs and services that are capable of addressing the needs of clients who may receive income support from a variety of sources including Employment Insurance, social assistance, student loans as well as their own resources. Saskatchewan is committed to designing programs around labour market and community need rather than income source.
• To support its move to an approach that links clients and funding sources with programs and services aimed at supporting labour market development and attachment, Saskatchewan embarked on a comprehensive information and tracking system known as the One Client Service Model (OCSM). This system is intended to offer an integrated capacity to record and share information about individual and organizational program and service provision.
• The final concept that guides the Saskatchewan approach to an integrated post-secondary education and training sector is an emphasis on regional planning underpinned by a sectoral approach to labour market analysis. Throughout the province there are venues that bring together business/industry, government, educational institutions, and community-based organizations to identify emerging labour market issues, assess the current array of programs and services available to meet these, and help plan actions that will enhance the programs and supports available to Saskatchewan adults.

The essential implication of the Saskatchewan approach for EI clients is that, in principle, they are to access the same portfolio of courses and supports open to any unemployed person in Saskatchewan. As the CS-LMDA evolved, some important differences were evident, and special programming has been implemented to serve EI clients. Most notable is the Skills Training Benefit, which offers educational financial support and in some cases an income supplement for EI clients.

The first year-and-a-half of the CS-LMDA has seen the two orders of government each endeavouring to fulfill the objectives of the EI Act and the Saskatchewan Sector Strategies Plan, which succeeded the STS. The process of negotiation and discussion is a process of searching for the common ground to allow the federal government to meet the requirements of the EI Act, and the province to integrate EI clients into a comprehensive labour market training initiative designed to serve all unemployed and underemployed residents.

An important goal for the evaluation of the CS-LMDA, particularly the summative evaluation, is to assess whether EI clients have been effectively trained, and whether they are returning to work. By comparing the experience of those who use programming under the province's approach and those who do not, the incremental benefit to the client can be measured.

6.2 Design, delivery, planning, and implementation

In general, despite the extended process of negotiating the Agreement, the design, delivery, planning and implementation has resulted in an effective system.

• Provincial programs and services reflect the intent of the CS-LMDA. A key issue in the early stages of the CS-LMDA was to ensure that provincial programs and services could be assessed as comparable to employment benefits and support measures (EBSM) offered in the pre-LMDA era. Federal and provincial officials are satisfied that this match now exists, especially with the creation of the Skills Training Benefit.
• Overlap and duplication of PPSs are minimal. Three areas exist where federal and provincial programming purposely overlap: programming for youth, persons with disabilities, and Aboriginal persons. Federal and provincial officials believe that this apparent overlap increases service to these clients.
• Partnerships with industry have continued to be strengthened under the CS-LMDA. Some provincial managers attribute this to the pre-existing sector partnerships program, which was part of the STS.
• Eleven clients reported requested services in French, and nine stated they were satisfied with services received. Insufficient recorded use of these services exists to support a conclusion on their adequacy.
• Federal managers reported that the province was slow to formally recognize the federal government's contribution to the CS-LMDA. Officials from both orders of government acknowledge that progress is evident and both federal and provincial contributions are recognized in program literature, and in the name given to Canada-Saskatchewan Career and Employment Services (CSCES) offices.
• The provincial information system needs further development. OCSM needs to be developed in order to support program monitoring and performance measurement. Development appears to be behind schedule, and information from various program sources has not been fully integrated. Reconciling existing databases and adopting standards across programs (e.g., type of program, outcome codes, dates for program start and end) are urgently needed to improve the system.
• Information sharing protocols between federal and provincial partners need to be enhanced. The inability to share information properly was criticized by both federal and provincial officials we interviewed. The process that provincial employees use to identify EI eligible clients is inefficient, and compromises effective program delivery. Without access to historical records (EI eligibility and training history), those responsible for counselling EI clients will be unable to prepare timely and complete training plans. As institutional memory fades, CSCES staff will have a more difficult time going through informal channels to access information about clients directly from HRDC staff.

6.3 Success/impacts/effects (Client satisfaction and preliminary outcomes

Some impacts can be discerned based on the client and employer surveys:

• Clients are satisfied with services provided at CSCES offices. Most who have used them have found them to be helpful - particularly career counselling, multimedia products for resume writing, and computer services for searching the Internet. Based on the client survey, there appears to be minimal variation in the use of CSCES services across regions. Those clients who have used services in both the pre-LMDA period and since the LMDA was signed report that services are the same or better. However, for the most part, clients are still using the same types of programs under the LMDA as under the federal system. They also report similar levels of income support; many remain on Part 1 EI benefits while in training.
• We also asked employers whether they believe that the CS-LMDA has improved training outcomes. However, employers find it difficult to distinguish between the former federal and current provincial training programs. Some feel that more programs are now available, and that wage and training subsidies have encouraged them to hire EI clients. Many employers who participate in a wage subsidy program or use the job order process to find employees are unaware that they have hired a former EI client. As such, they are unable to evaluate the current provincial programs with respect to serving EI clients. Due to the difficulty of "flagging" an EI client, the analysis in the formative evaluation could not "separate" these trainees and compare their experience with other non-EI clients.
• Service providers are aware of the changes brought on by the LMDA in terms of the use of community-based organizations. Some service providers report that they now receive some core funding, in addition to project-based funding. This has enabled some to plan for the long term, rather than providing services on a project-by-project basis. Others are concerned that the emphasis on using contract training has lessened. Some find it difficult to train Social Assistance (SA), EI, and non-income support students in a mixed class. Differences in income support regulations raise questions about equity among students.
• An intended outcome of the CS-LMDA is improved planning. Regional needs assessments and Sector Partnerships are easily identified processes, but these pre-date the agreement. Therefore, it is difficult to conclude on the extent to which the CS-LMDA has supported planning processes. Although these planning processes can have an impact on EI clients, they are not typically identified as a target group that needs further scrutiny. An important change resulting from the LMDA has been the shift in authority for planning to the recently created Regional Services Branch, which now controls the budget and has the decision-making authority.

6.4 Program impacts / effects are preliminary

Many questions posed in the evaluation framework regarding success, impacts, and effects require information from HRDC on all EI clients, which will be available for the summative evaluation. Program impacts are addressed in a preliminary and tentative way through client survey, focus groups, and dyads.

• Based on the client survey (n=1,250), about one-third of participants said they were required to prepare a case/action plan prior to beginning training. A similar proportion of clients completed case/action plans in each region. Of those who talked to a counsellor about a back-to-work plan (n=429), 28 percent had completed such a plan, 53 percent were part way through, while 18 percent stopped part way through or never started the steps. Based on the client and focus group responses, back-to-work plans (case/action plans) appear to be an effective and useful part of the training process.
• Clients that have used provincial programs and services express high levels of satisfaction. Although room for improvement certainly exists, clients rated CSCES centres highly. Clients expressed less support for the web sites (federal and provincial), largely because they saw them as often being out of date. Word-of-mouth and networking, particularly in rural areas, appear to be the most common way that clients reported getting work. This leads many clients to deny that the provincial programs and services helped them find employment.
• Employers are generally satisfied with programs targeted to EI clients. Programs have helped a majority of employers to "fill job vacancies" (63 percent), "fill skill shortages" (50 percent), and "meet human resource needs" (54 percent).
• Dyads revealed the sacrifices made by five families that had used provincial programs and services. The separation of the trainee from his or her family was the most important hardship. The role of a supportive spouse in completing successful training also emerged as a common theme in the dyads. All participants and their families strongly endorsed the value of the programming.

6.5 Other issues: the critical role of a comparison group and improved databases

The most important deficiency in the CS-LMDA is the weak information system to support both the provision of services to EI clients and the conduct of evaluations. The province is keenly aware of the need to improve OCSM.

Improving information in OCSM also supports the creation of intervention profiles for EI clients of provincial programs and services. This includes removing duplicate entries, ensuring that STB program interactions are recorded, and ensuring that information from other sources such as SIAST is recorded. For example, with complete client information, the survey can reference those provincial services that are part of the CS-LMDA and not other training interventions not funded by EI (Part 2). Further, the more detail available on the types of interventions used by participants, the more precise the estimate of incremental benefit arising from PPSs offered under the LMDA.

Footnotes

1	Clients of Provincial Programs and Services (PPS) will meet with counselling staff, usually at a Canada-Saskatchewan Career and Employment Centre, and develop a program of training, job search, and or job placement, the goal of which is to secure sustainable employment within a specified time period.
2	Bakvis, Herman, and Aucoin, Peter, "Negotiating Labour Market Development Agreements," Canadian Centre for Management Development, March 2000, offers a concise history of this process.
3	The provincial government dropped the retraining option under JobStart/Future Skills in April 2000.
4	The main exception is the Skill Training Benefit (STB), an income support program designed specifically for EI clients. Saskatchewan created this program, because the existing program (the Provincial Training Allowance) offered benefits that were lower than would be available to an EI client under the previous framework.
5	Self-employed persons do not contribute to the EI Account.
6	The federal government also delivers programs to "youth," people with disabilities, and Aboriginal people. This is significant in Saskatchewan due to the increasing Aboriginal population - the majority of whom are not EI eligible -and a large number of youth, the majority of whom are Aboriginal. Federal funding flows to both First Nations and Métis organizations, and is almost the same dollar value as that provided under the LMDA. Funding is provided from consolidated revenue, not EI funds.
7	A 5-year period is used if a person left work for paternity/maternity leave.
8	The term "employment benefit and support measure" is used in the Employment Insurance (EI) Act, in reference to specific types of employment programs established under Sections 59 and 60 (4) respectively. The term "benefits" is used in Section 63 of the EI Act in reference to provincial programs that are eligible for funding by the Canada Employment Insurance Commission (referred to hereafter as the Commission), provided they are similar to the Commission's employment benefits and are consistent with the purpose and guidelines of Part II of the Act. The term "measures" is used in Section 63 of the EI Act in reference to provincial programs that are eligible for funding by the Commission provided they are similar to support measures established by the Commission and are consistent with the purpose and guidelines of Part II of the Act. The CS-LMDA replaces the terms "employment benefits and support measures" with "provincial programs and services (PPSs)." "Provincial programs" means the programs referred to in Section 3 and described in Annex 1 of the Canada-Saskatchewan Labour Market Development Agreement. "Provincial services" means programs described in Annex 1 of the Canada-Saskatchewan Labour Market Development Agreement.
9	Community-based organization is the common term in Saskatchewan for a non-governmental organization.
10	The evaluation framework appears in Table 1, Section 2.
11	Throughout the report, the sample size is designated as (n = ). While the overall sample remains fixed, within a survey questionnaire, respondents may bypass certain questions that do not pertain to their situation or may decline to respond. Changes in the numbers of respondents is also noted as (n = ). In general, for a quantitative study such as the client survey reported here, smaller samples, especially those that are under 200, have lower reliability. Of course, management interviews have small samples, and we report the numbers of key informants who offered a particular view in the same way (n = ).
12	See Bakvis and Aucoin (op.cit.) for a more detailed discussion of co-management and devolution approaches to labour market development agreements. The New Brunswick LMDA is an example of another province that has created a comprehensive labour market development program, into which the EBSM defined in the Employment Insurance Act have been integrated.
13	Human Resources Development Canada (HRDC) tended to use more purchase of training and contracting with private trainers. PSEST favours using SIAST and regional colleges to deliver training to all learners.
14	Community-based organizations typically serve specific groups that require intensive assistance over a longer period of time. Examples include Aboriginal persons, persons with disabilities, and recent immigrants.
15	Dyads and focus groups with clients revealed many stories supporting this, indicating an important reason why EI clients may not use programs and services to retrain or find new jobs.
16	Client survey: 216 clients out of 1,250 were asked this question.
17	Except where noted, the term "key informant" refers to all respondents: federal, provincial, and those from community-based organizations.
18	We do not distinguish between federal, provincial, or other types of respondents in reporting these findings. As qualitative evidence, counting the numbers of different types of respondents would produce a false sense of precision.
19	Source: key informant interviews.
20	Source: client focus groups.
21	Client survey: 1,017 clients out of 1,250 were asked this question.
22	Source: client focus groups.
23	Client survey: 576 clients out of 1,250 were asked this question.
24	Source: client focus groups.
25	Source: survey of employers, sample size n=146.
26	Source: survey of employers, sample size n=146.
27	Source: survey of employers, sample size n=146.
28	Source: survey of employers, sample size n=146.
29	Source: survey of employers, sample size n=146.
30	Source: employer focus groups.
31	Source: survey of employers, sample size n=146.
32	Source: employer focus groups.
33	Source: survey of employers, sample size n=146.
34	Source: employer focus groups.
35	Source: survey of employers.
36	Source: survey of employers.
37	HRDC web site refers to sites such as the WORK Place. The Saskatchewan web site refers to sites such as SaskNetWork or the PSEST site. The survey listed the CSCES without any references. Clients may have accessed an HRDC web site and/or a Saskatchewan web site, and/or printed material at the location.
38	Source: service provider focus groups. Groups included representatives from community-based organizations, SIAST, and regional colleges, encompassing both major centres and rural areas.
39	The term "region" refers to a large center such as Regina, or one of the administrative regions as defined by PSEST.
40	Source: key informant interviews, n=20 out of a total of 41.
41	Quotas may be set that encourage participation among those eligible for training support.
42	Source: key informant interviews.
43	The "screen scraper" process allows provincial staff to obtain lists of clients who are eligible for Part 2 programs and services. HRDC prepares a regular extract of current EI eligible clients that is transferred to the province. A provincial staff member can use this to verify the eligibility of a client for certain PPSs, such as the STB. This extract is not linked to OCSM or federal databases, and is limited solely to determining potential eligibility for programs offered by the CS-LMDA.
44	Source: key informant interviews, n=25 out of 41.
45	Source: client focus groups.
46	Source: key informant interviews.
47	Federal key informants uniformly expressed concern over this, and most provincial key informants acknowledged that the provincial information system still requires considerable development.
48	The STB provides training support. It is used for tuition and books, with Student Loans or EI Part 1 benefits providing income support. STB can "top-up" either a Student Loan or Part 1 benefits, but it is intended for tuition and books for training.
49	The term under-reporting refers to fact that the true level of service interaction is higher than is reported or recorded by OCSM.
50	Respondents from PSEST had the most definitive opinions on this issue.
51	Source: OCSM and STB databases.
52	Source: OCSM database, 4,589 records did not contain data. Language of service is not recorded in the STB database.
53	Although the formal commencement of the CS-LMDA is January 1, 1999, we treated the first three months of the Agreement as a transitional period for the purpose of this evaluation.
54	The PSEST program adopted many of the same features as the previous HRDC self-employment program, which was a main element of the redesign of the Unemployment Insurance Program.
55	Sample size is small (n=37). Results should be viewed with caution.
56	Note that 1250 is the total sample of participants interviewed. For any specific question, some respondents may decline to respond, or may not be qualified. For this reason, the effective sample size will be somewhat smaller than the total.
57	This percentage is based on evaluation of other programs, such as the Provincial Training Allowance in Saskatchewan and Taking Charge! in Manitoba.
58	Note that the total sample for the employers survey is 146, but as shown in Table 23, 124 responded to the question about outcomes for EI participants.
59	These clients are recalling periods prior to the PTA.
60	This is a weighted average for the last row of Table 29.
61	The remaining 30 percent either refused or stated that the web site met none of their needs.
62	Based on 216 respondents who were active EI clients before and after January 1999.
63	Again note that the effective response is less than the total response of 146.
64	See Lessons Learned on Welfare Reform: Labour Market Perspectives (Prairie Research Associates (PRA) Inc., March 1999) and Lessons Learned on Aboriginal Social and Economic Development (Prairie Research Associates (PRA) Inc., December 1998) for reviews of these recent approaches.