Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Program evaluation was identified as an important component of the SI Contribution Agreement. The SI Program tested new and innovative approaches to increasing labour force participation, including the use of partnerships, community and industry involvement, multi-media, and other approaches to reduce employment barriers. In October of 1996, Calibre Consultants Inc. was contracted to conduct the evaluation under the direction of the Federal and Provincial Co-chairs of the SI Evaluation Working Group.

The evaluation was designed to be completed in two phases: the 'formative evaluation' which assesses the development phase, and the 'summative evaluation' which assesses the impacts after program implementation. The formative evaluation objectives are to identify:

• the extent to which the projects are contributing to the SI principles;
• the extent to which the process is efficient and effective; and
• key learnings for consideration in the next phases of the pilot projects and in future programs.

The formative evaluation was conducted in three segments corresponding to the three streams — W/S, LMI, and CS.

The W/S formative evaluation was conducted between November, 1996, and July, 1997.

The following methods were used to do the Work/Study formative evaluation:

• an extensive review of all of the Strategic Initiative W/S documents, including agreements, guidelines, application procedures, monitoring reports, fact sheets, newsletters, project tracking system reports, work flow plans, and project contracts;
• twenty-four in-depth key informant interviews with SI Senior Management and W/S Co-chairs, Working Group members, and staff;
• sixty participant interviews including learners, partners, employers, and delivery agents;
• twenty sponsor and partner interviews; and
• three workshops with W/S Co-chairs, Working Group members, and staff.

The first set of in-depth interviews was conducted to gain input on the methodology design and to identify performance indicators. The second set of in-depth interviews addressed questions regarding program appropriateness and process. The results of the preliminary research were consolidated in a draft document. Three stakeholder workshops were held to review the research findings, discuss issues, and develop common understandings regarding the key learnings.

The following methods were used to conduct the LMI formative evaluation, which took place between November of 1996 and September of 1997:

• an extensive review of all LMI documents, including agreements, needs assessment reports, histories, mid-term reviews and final reports, project contracts and statements of costs, newsletters, project proposals, terms of reference, and other descriptions of the LMI network and products;
• thirty-five in-depth interviews with SI senior management and the LMI Co-chairs, Working Group members, Project Co-ordinator, and IT staff;
• six focus groups with job seekers, career counsellors, career service representatives, LMI Working Group members, and PSEST senior management; and
• two workshops: one with SI Co-chairs from all three areas, communications and evaluation, and one with the LMI Working Group members.

Several LMI stakeholders provided input into the methodology design, including members of the Evaluation Working Group, the Evaluation Co-chairs, the LMI Working Group Co-chairs, and the LMI Project Co-ordinator. The results of preliminary research conducted through the document review, in-depth interviews, and focus groups were consolidated into a draft document. Two stakeholder workshops were held to review the research findings and discuss issues that had been raised to ensure that issues and recommendations were presented within the proper context, and to develop common understandings regarding the key learnings. One point of concern raised in the process was the absence of a formal needs assessment on LMI at the outset of the Strategic Initiatives process. No formal needs assessment was done as the LMI working group drew together experts with an interest in LMI from Saskatchewan government departments, HRDC, the Saskatchewan Institute of Applied Science and Technologies (SIAST), and regional colleges. The group felt a needs assessment was unnecessary in light of its collective expertise in the area.

The CS formative evaluation was undertaken between November 1996 and July 1997. The following methodologies were used to support the CS formative evaluation:

• an extensive review of all SI Career Services' documents, including technology analyses, survey research, project facilitators' reports, needs assessment reports, implementation plans, frameworks for evaluations, mid-term reviews and final reports, focus group results, workplans, and telephone survey data summary and analyses;
• twenty-five in-depth key informant and stakeholder interviews with SI Senior Management and CS Co-chairs, Working Group members, staff and Project Co-ordinators/Managers, and regional college staff; and
• one workshop with SI Co-chairs and CS Working Group members.

The three streams of Strategic Initiatives, W/S, LMI, and CS, share many of the same features with respect to relevance to the SI Program, design and delivery, success, and cost-effectiveness. These similar features may be examined in several areas that include, but are not limited to, the following:

• infrastructure;
• administration;
• communication;
• partnerships; and
• finance.

Four main categories have been used to frame the formative evaluation findings. These are: relevance, program design and delivery, program success, program cost-effectiveness.

Relevance in all SI Program streams was assessed by reviewing the processes that were used to determine: the social and economic needs for each stream, how individual projects were identified, the manner in which target groups were incorporated in each stream, the role of innovation both in planning and delivery of the stream, and linkages to client services.

In this area, infrastructure plays a key role. Infrastructure refers to the basic structural foundations of the Strategic Initiatives Program, including the three streams of SI and the projects within them, which form the basis of all operations. The features examined with respect to infrastructure include: the determination of project priorities, the human resource time required to fulfill program commitments, and finally, the linkages between the LMI and the CS initiatives. Assessment of administration consisted of the following aspects: timelines, flexibility, standard procedures, and working with two orders of government. In the area of communication, the specific features assessed relate to: meetings, communication with stakeholders, public awareness and relations, and communication strategies.

The common area of partnerships is assessed with respect to the strength and effectiveness of the partnerships, the broad perspective incorporated by the three streams, the increased access to services, and the leveraging of costs and reduction of duplication.

Financial considerations including significant common features such as resources, partners, funding allocations, and flexibility.

 

---

Last Modified: 1999-06-16	Important Notices