Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Joint Management Response — Formative Evaluation of EBSMs under the Canada-Newfoundland and Labrador Agreement on Labour Market Development Agreement

The Canada-Newfoundland/Labrador Agreement on Labour Market Development (LMDA) is an agreement between the Government of Canada and the Government of Newfoundland and Labrador relating to the co-management of federal employment programs and services specifically under EI Part II.

Section 12 of the LMDA outlines the requirement of the Labour Market Partners to cooperate in developing and implementing a two-phase evaluation framework. To this end the Management Committee is responsible for the completion of all evaluations related to the employment benefits and support measures. The Management Committee is pleased that the first phase of the evaluation process, namely the formative evaluation, has been completed. The formative evaluation is comprised of three associated reports. The first report (June 2000) covers three Employment Benefits, i.e., Targeted Wage Subsidy (TWS), Self-Employment Benefit (SEB) and Job Creation Partnership (JCP). The second report (June 2000) deals with the Training Benefit (currently Skills Development), and the final report (June 2001) deals with Support Measures, i.e., Local Labour Market Partnerships (LLMP) and Employment Assistance Services (EAS).

The evaluation evidence from the three formative evaluation reports completed to date has shown that in the near term, clients served by the employment benefits and support measures have been moderately successful in achieving their employment goals. Labour Market Development Agreement accountability targets for unpaid EI benefits and client returns to employment have been met in each fiscal year since the inception of the agreement. The Management Committee anticipates that the summative evaluation report, which will contain more definitive evidence, will confirm these findings in the longer term. The Management Committee looks forward to the commencement of the summative evaluation in 2002.

The formative evaluation reports have identified many specific findings. The Management Committee suggests that we can deal with these findings under general categories: the functioning of the co-management relationship, findings related to programs/services and findings related to client impacts. A joint Management Committee response to each group of findings is reported below:

Co-management:

The evaluation reports suggest that "under the co-management approach, it is clear that each government has been able to pursue its respective priorities." The Management Committee suggests that this is perhaps one of the most fundamental achievements of this co-managed agreement, given the unique cultures and priorities of both levels of government. The Management Committee also feels that the effectiveness of the co-management approach is primarily attributable to the level of communication and co-operation among partners.

The reports suggest that co-management has increased administrative responsibilities in terms of business and financial planning. The Management Committee feels that it has resulted in a more targeted response to the needs and opportunities of unemployed individuals. The Management Committee also believes that these pressures are abating as the partners become more knowledgeable of each other's programs and collective priorities.

Programs & Services Finding:

The evaluation evidence suggests that eight out of ten individuals who received services from the HRCCs and Employment Assistance Services offices had a high level of satisfaction with that service. The Management Committee feels that this evidence reflects a strong service commitment of HRCC staff. The committee also believes that despite a challenging economic climate in many rural parts of the province, HRCC staff are making a valuable contribution to the lives of individuals in these communities.

With respect to the training benefit two findings are noteworthy from the Management Committee's perspective: Ninety percent of the sample of training participants indicated that participation in the training benefit made them more employable. However, it is recognized that a lack of participant information at this early stage of evaluation, particularly with two and three year programs, did not allow a complete analysis of the impact of training on employment. The Management Committee is interested in the long-term impact of training on subsequent employment gains and will examine this issue in the summative evaluation. The Management Committee has also noted that graduation rates are somewhat unclear given the limitations of administrative data. This issue must be addressed so we can more closely track individual impacts resulting from training.

Participant surveys highlight that the support measures offered were well received by program participants. The Management Committee feels that the support measures are essential elements of the LMDA. The measures allow the Management Committee to provide support to incremental initiatives that have many positive impacts for individuals and communities throughout the province. The evaluation suggested that Employment Assistance Services participants welcome the convenience of their being served in their geographic area. " The main tangible benefit emerging from the first three years of EAS seems clearly to lie in the improved employability of the client group." However, additional evidence is needed to determine the overall impact on employment gains that these services are providing. This work will be undertaken in the summative stage of the evaluation process.

Local Labour Market Partnerships (LLMPs) have also furnished the Management Committee with the capacity to engage numerous community partners in various incremental activities. The evaluation evidence suggests many valuable activities have taken place, which, in the absence of LLMP, might not have come to fruition. The Management Committee feels that the employment benefits of these initiatives will be realised in the longer term.

Client Impacts:

The evaluation evidence suggests that SEB and TWS participants experience positive employment gains in the post-program period. This incremental impact shows that investment in these benefits is a prudent expenditure of public funds. The Management Committee desires to maximize the potential of these benefits while recognizing their finite growth potential, given the modest employer and market base in the province.

The evaluation indicates that JCP participants did not realise significant post-program employment gains, however, JCP earnings did improve the lives of these participants, their families and the communities in which they live. These results must be interpreted with the knowledge that JCP participants had the lowest levels of pre-program earnings and education of all participant groups. Given this information, the management committee submits that the impact of JCPs on longer-term employment gains must be established before a judgement can be made about the overall utility of JCP participation.

The engagement of Social Assistance Recipients in LMDA programs was a priority of the LMDA partners. The report outlines that these targets were met and that SARs achieved positive outcomes from the programs offered. However, the disproportionately higher level of SAR participation in JCP (compared with other interventions) and the subsequent lack of post program employment are of concern to the Management Committee. Improvement of assessment, selection and referral of SAR clients within the EBSM model is a priority of the LMDA partners. It is believed that this approach will lead to a more balanced level of participation of SAR clients in all benefits and measures. A determination of the utility and impact of this new approach should be one of the priorities of the summative evaluation.

Aspects to monitor and evaluate:

While the Management Committee is generally pleased with the findings of the three Formative Evaluation reports, challenges remain. The following is a list of LMDA dimensions that the Management Committee feels is in need of additional monitoring and evaluation: The Training Benefit report suggests that in the early stages of implementation of Negotiated Financial Assistance (NFA), there was a perception of unequal treatment of clients in terms of the levels of funding received. At the time of the evaluation, Negotiated Financial Assistance was a new process for both clients and staff alike. The problems with dissemination of information and the generally low level of understanding of the process lead to concerns on behalf of both clients and staff. The Management Committee agrees with the consultant's recommendation that this area of concern should be closely monitored to ascertain whether NFA principles are used consistently. Nevertheless, the Management Committee senses that with the passage of time, the intent of NFA has been more widely understood and accepted by all parties. The NFA issues that were identified in the formative evaluation should however be re-examined in the summative evaluation to determine the extent to which these concerns persist.

The evaluation evidence suggests a negative employment gain for female JCP participants. This is of concern to the Management Committee who believes that JCP has much economic and social merit. Perhaps more careful targeting of JCP must take place to ensure positive impacts for participants in the post-program period. Again, the summative evaluation should provide more definitive evidence as to the long-term employment impact of JCP. In addition, decisions regarding SAR participation should not be made based on point estimates from the initial evaluation.

The evaluation report outlines a data concern associated with Targeted Wage Subsidy administrative data. While this issue has not limited the effectiveness of this benefit, the absence of reliable data for administrative and evaluation purposes may bias our assessments of success or failure, so the data must be improved. The consultant also reported that based on the employers' survey responses, a proportion of the participants would have been hired without the subsidy. The Management Committee recognizes that this finding may outline a potential weakness in the current TWS model. The Management Committee does not however feel entirely comfortable with the integrity of this initial finding. Prior to making any potential adjustments to the program, a more precise and rigorous examination of this issue must be explored.

Support measures continue to provide excellent employment assistance and partnership opportunities within the local labour market. The Management Committee considers that the increased level of funding to these activities over the initial period of the LMDA will necessitate further detailed analysis and ongoing monitoring of the effectiveness of these support measures, through the summative phase of the LMDA operational and evaluation plan.