Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The Conclusions from this formative evaluation are provided in this chapter in relation to the three categories of issues addressed:

• the continuing relevance of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement;
• the design and delivery of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement; and,
• the success of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement.

6.1 Relevance

The relevance issues identified for the evaluation relate to the extent to which the EBSM's correspond to:

• the statutory requirements of the EI legislation;
• the priorities of federal and provincial governments;
• the needs of individuals; and,
• local economic development priorities.

Conformance to EI legislation

The EBSM's delivered under the Agreement correspond to the EI legislation and related policy directives of the Government of Canada. No significant changes have been made to either the design or the delivery of the EBSM's under the Canada-Newfoundland and Labrador Agreement on Labour Market Development.

Conformance to the Priorities of Federal and Provincial Government

The priority of the Government of Canada as regards EBSM's is to provide assistance to active EI claimants⁴³ to facilitate their return to employment. The Government of Newfoundland and Labrador accepts this priority and also seeks to address its own priorities:

• to ensure that reachback clients have the same degree of access to EBSM's as current EI claimants;
• to ensure that EBSM expenditures are made in a way as to be consistent with and complementary to the economic develop priorities and initiatives of the Government of Newfoundland and Labrador as well as those of Regional Economic Development Boards (REDB's) across the province; and,
• to enhance the capacity and ability of the Government of Newfoundland and Labrador to serve its citizens as regards employment programming and social and economic development.

Under the co-management approach, it is clear that each government has been able to pursue its respective priorities.

The Government of Canada's priority is dictated by the EI legislation which. Under the Agreement, HRDC has continued to be governed by this legislation.

The co-management process has enhanced the ability of the province to pursue its priorities in several ways which were noted by key informants:

• the business planning process includes targets (provincially and sub-provincially) for reachback clients and various industry sectors;
• provincial government representatives on local Project Assessment Committees examine the conformance of proposed projects to social and economic priorities of the area;
• staff of the provincial Department of Human Resources and Employment have received training in the Service Needs Determination (SND) Process which allows them to improve the referral of eligible SAR reachback clients to HRDC. It also improves their ability to assist other unemployed individuals who are not eligible for EBSM's.

Conformance to the Needs of Individuals

This evaluation has found short term post-project employment benefits for Targeted Wage Subsidies (TWS) clients and for Self Employment (SEB) clients which suggests that the needs of these clients have been addressed. This is also supported by the views of key informants and the high satisfaction of participants with the program and their experiences under the program.

Participants in Job Creation Projects (JCP) have not realized short-term employment gains after their project participation. Key informants indicated that such gains would not be expected. They saw the primary focus of JCP as being to support economic development initiatives as well as to provide short term income to participants. The providing of new skills and experience to participants was generally seen as a lower priority. It was beyond the scope of this evaluation to assess the effectiveness of JCP in stimulating economic development.

Conformance to Local Economic Development Priorities

The evidence from this evaluation suggests that local economic development priorities are an important consideration regarding EBSM expenditures. As noted, it was beyond the scope of the evaluation to assess the effectiveness with which these priorities have been pursued. However they clearly have been pursued:

• Business Plans (Provincial, District and Local) identify specific targets for priority sectors. Processes are in place to monitor the achievement of these targets and to report deviations from plans to the Provincial Management Committee and to District Management Committees.
• ACOA, the Department of Development and Rural Renewal and Regional Economic Development Boards participate in Provincial and District Management Committees. They also participate on local Project Assessment Committees.

6.2 Design and Delivery

Design and delivery issues for this evaluation address:

• the extent of co-ordination between the Government of Canada and the Government of Newfoundland and Labrador;
• the responsiveness to local conditions in the delivery of the EBSM's;
• the adequacy of administrative data for evaluation and monitoring; and,
• client characteristics and targeting.

While all these issues have been addressed, the Canada-Newfoundland and Labrador Agreement on Labour Market Development allows for significant innovation, flexibility and accountability at local and district levels. It was beyond the resources of this evaluation to examine the consequent variability in a detailed fashion.

Co-ordination Between the Federal and Provincial Government

Key informants consistently were positive about the extent and success of co-management under the Agreement. In particular, key informants noted:

• the value of the Business Planning process which has been applied at the district and provincial levels;
• the much improved knowledge that each level of government now has of the other's programs and priorities;
• the consensus decision making approach used by the provincial and district management committees and by some local Project Assessment Committees;
• the increasing integration of staff training across the two levels of government; and,
• the increased partnership between the two levels of government as regards programming not covered by the Agreement (e.g. Canada Jobs Fund; Fisheries Restructuring & Adjustment Measures (FRAM); and, summer employment programs).

Negatives noted regarding co-management and the resulting co-ordination included:

• delays in finalizing operational budgets attributed to the Business Planning process;
• need to re-allocate budgets to deal with under-commitment and/or over-commitment of available funds in some funding envelopes;
• initial difficulties faced by the Government of Newfoundland and Labrador as it needed to develop its capacity in order to fulfill its role under the Agreement; and,
• expectations that co-management is a temporary mechanism and will be replaced by devolution of employment programming to the Government of Newfoundland and Labrador in the medium term.

Responsiveness to Local Conditions

Key informants noted that local Project Assessment Committees have increased the range of input to decisions as to projects to be implemented in local areas.

Adequacy of Administrative Data

Electronic information for the EBSMs is maintained in two separate systems. Within the HRCCs, the staff use NESS to enter relevant information while outside of the HRCCs the counsellors at third party delivery sites use CATS. The CATS data contains information downloaded periodically from the NESS system, so that counsellors at these third party sites have access to client history similar to that available in the HRCCs.

Data from both NESS and CATS are used in the construction of the results data set, which uses a combination of EI data and the action plan information and follow-up to determine which clients have successfully returned to work, and to calculate any resulting EI savings. The results data set is then processed by the regional HQ to produce regional level reports on achievement of targets.

The current system provides a reasonable basis for tracking clients who are participants in EBSMs. In combination with EI data files a good client contact list can be produced. However there are two serious deficiencies in the current data management system.

• Successes (clients returned to work) recorded in the data do not correspond with actual client experiences as measured via the follow-up survey conducted in this evaluation. The calculation of success using EI information (clients who return to work within their EI period) is substantially more accurate than the calculations that have to rely solely on information entered by counsellors.
• Many clients who have TWS placements are not recorded in the data. This occurs since HRDC makes payments to the employers; not directly to clients so no client contract has to be set up. This problem is most prevalent with employers who have multiple placements. For the province as a whole, data is available for 46% of TWS participants. In the Avalon District data is available for only 32% of TWS participants.

Client Characteristics and Targeting

Under the Agreement, targeting provisions exist for priority sectors and for Social Assistance Recipients (SARs) who satisfy the eligibility provisions of the EI legislation. For the first two years of the Agreement, targets were set at relatively modest levels. These targets have guided the District Management Committees as well as local Project Assessment Committees and staff of HRDC and third party organizations contracted to provide services. However, key informants noted that the modest targets meant that major changes did not occur relative to the past. In particular, it was noted that project applications as well as participant applications were generally assessed based on their merit rather than on targeting provisions.

For the 1999/2000 fiscal year, targets for SARs were doubled relative to those for 1998/1999. Achieving these targets is likely to be a greater challenge and will necessitate co-ordination at the operational level between HRDC and the Department of Human Resources and Employment (HRE). Training has been provided to HRE staff across the province to allow them to fulfill their responsibility to identify and refer eligible SARs to HRDC.

6.3 Success

The success issues identified for the evaluation address:

• the short-term post-program employment experiences of participants;
• the effectiveness of case management;
• the satisfaction of participants and employer/sponsors;
• the extent to which the experiences of SARs differed from those of other participants; and,
• the extent to which targets for employment and savings to EI have been met.

Short-term Post-Program Employment

Based on 1,494 participants surveyed during this evaluation, participants have been employed on average 49% of the time since completing their intervention. This varies substantially by intervention type:

• SEB participants have been operating their business or employed for 89% of the time since HRDC funding support was terminated;
• TWS participants have been employed 64% of the time since their wage subsidy ended; and,
• JCP participants have been employed 46% of the time since the project ended.

Statistical regression models indicate that for SEB and TWS this post-program employment is higher than would otherwise be expected. The models employed adjusted for other measurable characteristics which are known to affect employment success (e.g. age, education, prior labour market experiences, etcetera) but do not adjust for selection bias. Separate models were employed for males and females. These models indicated the following:

• SEB participants have been in business or employed more than 35 percentage points more of the time than would be expected based on their other characteristics (36 percentage points for males and 39 for females).
• TWS male participants have also been employed 26 percentage points more than would be predicted from the models. The post-program employment for female TWS participants is estimated at 6 percentage points higher than would otherwise be expected. This result is not statistically significant.

JCP participants post-program experiences are closer to what would be predicted based on their other characteristics. Male JCP participants have been employed an estimated 12 percentage points more of the post-program period than the model predicts while females have been employed 9 percentage points less that the model predicts.

All the above data is short term in nature with most participants interviewed within a year of their program completion date. It is likely that this short-term orientation explains at least part of these results. In particular SEB participants are likely to continue in business after their subsidy ends and many TWS employers retain participants after termination. Consequently, participants from both interventions can be expected to do well in the short term. Conversely, almost all JCP participants will be unemployed at the end of their project and poor short-term post-project employment is expected.

Effectiveness of Case Management

The evaluation did not examine case management processes and their effectiveness in depth. Based on the evidence obtained, the following conclusions can be drawn:

• 64% of HRCC staff describe the current method of case management as very effective compared to only 32% of staff of third party providers;
• the "Action Plan" which participants must complete is not, in retrospect seen as a valuable process. Only 12% of participants surveyed, recalled preparing an action plan; and,
• participants were generally satisfied with the level of service they received from the HRCC or Outreach office with 80% rating their satisfaction at 8 or higher on a 0 to 10 scale.

Satisfaction of Participants and Employers

Participants were satisfied with HRDC staff and service but less so with the value of their project experiences:

• 80% were very satisfied⁴⁴ with the level of service received; and,
• 72% were very satisfied with the variety of services available.

but, only,

• 47% gave equivalent ratings for how close the work was related to their career objectives; and,
• 28% indicated the program was very useful in helping them find a job.

Employers and sponsors were highly satisfied with all aspects of the program:

• 76% were very satisfied with the participant's skills;
• 89% were very satisfied with HRDC administration; and,
• 81% rated the value of the participant's work very high.

Success of Social Assistance Recipients

The regression models of percentage time employed after the program indicate that SAR participants have somewhat less success in the post-program period than individuals who were otherwise similar. However, these effects were small (13 percentage points for males and 7 percentage points for females).

The evaluation compared the experiences of "reachback" participants to those of other participants in order to address this question. Late in the evaluation, however, the validity of the "reachback" indicator in HRDC's administrative data was determined to be questionable.

Employment and EI Savings Targets

HRDC administrative data indicates that employment and EI savings targets have been met. The evaluation indicates that HRDC's estimate of "returns to employment" is low based on data from participants surveyed. TWS, SEB and JCP do not account for a large share of EI savings attributed in HRDC's administrative data.

Footnotes

43	Defined as those with a current active claim as well as those who satisfy the reachback provisions of the EI legislation.
44	8 or higher on a 0 to 10 scale.