Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The Canada/Yukon Labour Market Development Agreement (LMDA), like all provincial and territorial labour market development agreements, prescribes the use of primary measures to monitor short-term LMDA success in three primary measure areas: number of Employment Insurance (EI) clients served by LMDA interventions and the percentage who are active claimants, participants returning to work, and unpaid EI benefits of participants returning to work. The Agreement further states that results targets are to be set annually for these measures and adjusted to reflect target attainment and changing economic conditions during the previous fiscal year.

It would appear that these targets are in fact sensitive to changing conditions, realities and restraints in the Yukon. Between 1997/98 and 1998/99, the targets set for the number of EI client participants served by the LMDA and the proportion who are active EI claimants were reduced, while the targets for the number of returns to work and the amount of unpaid EI benefits were increased. The flexibility in the setting of targets is a positive development, adhering to a principle laid out in the Canada/Yukon LMDA.

Two concerns were raised with respect to the targets, however. First, they include Apprentices who are not "typical" EI clients returning to work and possibly generating unpaid EI benefits. Second, the targets are focused on Part I EI benefits, but ignore incidences where a participant may subsequently receive Part II benefits, which effectively reduces the estimated "savings" to the EI account. Exclusion of Apprentices in the targets and a consideration of both Parts I and II benefits are suggested.

As for target attainment, Table 7.1 shows that, based on the administrative data, there were 619 EI clients who participated in LMDA interventions in 1998/99, which is fairly close to the target for that year (759). As well, 57 percent of these participants were active EI claimants, which almost meets the target of 60 percent. Of those who participated, 283 returned to work, which again is close to the target (318). However, in 1998/99 there were only $1.31 million in unpaid EI benefits resulting from EI clients returning to work before the end of their claim. This is well short of the target of $2.35 million.

An important component of the evaluation work was the validation of the information systems and methods used to measure LMDA performance on the basis of the primary measures. The presence ofsound information systems is deemed important for accurate monitoring and measurement of program performance. This evaluation study uncovered a number of problems, consideration of which would contribute to the ongoing process of improving the data and the methods used to measure Employment Benefits and Support Measures (EBSM) performance.

One major concern raised was with regard to the participant count, the first primary measure (columns 3 and 4 of row 1 of Table 7.1). A major source of participant data is the third-party contractors' files. The analysis revealed potential data-entry problems, that almost one third of all EBSM participants were accounted for in these files and not in the Human Resources Investment Branch (HRIB) files. This would indicate a significant underestimation of the number of participants, which has implications for the ability of the information systems to measure performance. Moreover, accounting for contractor-only participants (row 2) drives down the active claimant share to a lower level (47 percent) than is apparent in the HRIB files (65 percent). At the same time our analysis also revealed that almost all those who were indicated as being participants in the files did self-identify as participants in the survey conducted for this evaluation.

Regarding the second primary measure, returns to work, two different investigations were conducted in this evaluation. First, consideration was given to the issue of the extent to which the performance picture presented annually to program managers based on the administrative data misrepresents actual performance of that year's interventions over the full course of the post-intervention period. Our work indicated that, assuming the administrative data were accurate, a manager using data available at the end of the year would obtain a fairly accurate picture of LMDA success for interventions started in that year. For example, in 1998/99 the number of actual returns to work for interventions started in that year would be under-estimated by only 10 percent using fiscal year-end data.

Second, concerns were raised in this analysis about return to work (RTW) capture and accuracy. Using self-reported employment results in the evaluation survey to corroborate the RTWs recorded in the administrative data indicated that a large proportion of actual returns to work were not in fact captured in the administrative data (row 3 of columns 3 and 4 of Table 7.1). The inaccuracy was particularly great for participants on the contractors' files who were not originally in HRIB's files. The fact that actual returns to work appear to be underestimated indicates that the return to work target was likely exceeded by an even greater amount than indicated above. More importantly, this finding raises concerns, once again, over the ability of the administrative data systems to measure LMDA performance. On the other hand, a large majority of the returns to work recorded on the file were accurate, according to the evaluation survey.

A number of potential causes were suggested for the underestimation of returns to work. Among them are the previously mentioned problem of not all participants being captured in the files, a failure to contact clients in a follow-up telephone call, reachbacks and active claimants who had not returned to work before the end of claim but who might have returned to work after their intervention, and a follow-up call question that may be subject to misinterpretation and inconsistency across the regions. As for active EI claimants, there may be measurement problems as well, such as participants who return to work but who fail to report doing so in order to "protect" remaining entitlement weeks, which leads to an under-estimation of returns to work.

Our analysis also revealed some overestimation of unpaid EI benefits (UB), the third primary measure. Using alternative measures explicitly accounting for EI benefits actually paid, it was shown in this analysis that unpaid benefits were overestimated by 10-15 percent (row 4 of columns 3 and 4 of Table 7.1).

Potential problems were identified in the way HRIB measures unpaid EI benefits, which could account for this overestimation. In its calculation of unpaid EI benefits, HRIB ignores benefits paid in the return to work period (12 consecutive weeks receiving 25 percent of entitled EI benefits), and considers lapsed EI benefits as unpaid. These factors could lead to an over-estimation of unpaid benefits by HRIB.

Finally, the total return to work performance of the LMDA was considered in this analysis. It was found that active EI claimant participants are more than twice as likely to return to work and the latter are four times as likely as non-EI clients (56 versus 27 versus six percent). The return to work rate of income assistance recipients (who are non- or reachback EI claimants) was only somewhat higher than the reachback rate. The return-to-work rate for Apprenticeship was higher than the five main EBSMs, though concerns were raised about the inclusion of Apprentices (who artificially raise the return to work count) in an exercise to measure RTW success for EI clients.

The purpose of this exercise was to point out potential shortcomings in HRIBs accountability process. In light of the possible sources of error in measuring participation and performance identified in this analysis and bearing in mind the need for HRIB to still deliver information in a timely and cost-effective fashion, it is suggested that there needs to be a re-assessment of the methods of gathering information on participants and the measurement of returns to work and unpaid EI benefits. In particular, there needs to be greater effort and scrutiny in the gathering of information on participants and greater stringency in the capture of returns to work in the follow-up procedure. Furthermore, to reduce possible errors in measuring unpaid EI benefits, it is suggested that consideration be given to more intensive examination of administrative data. Without good information systems and methods, accurate measures of LMDA success cannot be achieved.