HRDC - Formative Evaluation of Employment Benefits and Support Measures in the Ontario Region

Directorates

· Director General's Office

· Data Development

· Strategic Evaluations

· Partnership and Continuous Evaluations

Publications

· Evaluation Reports

· Evaluation Briefs

· Labour Market Development Agreement Series

· Lessons Learned Series

· Evaluation Tool Kit Series

· Evaluation Forum Newsletter

· From Theory To Practice

Appendix A: Evaluation Research Issues and Questions

Implementation and Planning Issues

Under implementation and planning, five issues are addressed in this evaluation. The first is the extent to which the Ontario Region Employment Benefits and Support Measures (EBSMs), as implemented, reflect the basic principles of the Employment Insurance (EI) Act, including elimination of waste, reduced dependence on income support and increased self-sufficiency, integration of plans at the community level, local flexibility and partnership. The second planning and implementation issue is the extent to which the mix of locally provided EBSMs addresses the needs of individuals, employers, and communities. The third issue the evaluation examines is the extent to which EBSMs were designed and implemented in partnership with others in the community. The fourth issue is flexibility, i.e., the extent to which EBSMs permit local decision-making and reflect the needs of different communities. Whether locally delivered Employment Assistance Services (EAS) agreements were used to support active employment benefits is also considered. The final planning and implementation issue is annual targets, including how targets were set, what criteria are appropriate, and the feasibility of reporting against targets at the local (regional/Human Resource Centres of Canada [HRCC]) level.

Service Delivery Issues

The second set of issues concerns service delivery. The first delivery issue is the extent to which specific client groups received benefits and measures, including whether EBSMs were targeted towards specific clients (e.g., job-ready clients or diverse client groups) and the criteria used to select different clients. The second delivery issue concerns the extent to which clients were served in the language of their choice (in areas of the province that are designated bilingual) and how service may have differed according to language. The final issue addresses how the EBSMs affected the way of doing business. This includes the evolution of service delivery to service/case management; differences between the case management and traditional approaches of delivering services; the delivery of employment benefits through the EAS vehicle; and the impact of the altered policy on office staff and their understanding of their roles and responsibilities.

Impacts Issues

Under the category of impacts, six issues are addressed by this formative evaluation. The first five: client satisfaction, employment, attitudinal, client responsibility and community impact lend themselves to be addressed in the short-term. The latter issue: income dependence is best assessed in the longer-term. Notwithstanding the difficulty of measuring this last impact issue, this evaluation has included its assessment in order to test methodologies and data availability in preparation for the summative evaluation.

Client satisfaction is measured in terms of the extent to which clients are satisfied with the EBSMs, and the extent of and the reasons for discontinuance. The second impact issue is employment, including whether clients found employment or became self-employed, unemployment duration, the differential employment effects of individual EBSMs and the characteristics of successful clients. The third impact issue concerns attitudinal impacts, i.e., the extent to which the EBSMs enable clients to develop a positive attitude to employment. The fourth impact issue addressed is the extent to which the EBSM encouraged clients to take responsibility in the development and implementation of their action plan, including contributing to the cost of the intervention and following up on the action plan. The fifth issue is community impact and serves to re-focus results reported elsewhere in the report but within a community perspective. The six impact issue is reduced income-support dependence, i.e., the extent to which participants were able to reduce reliance on EI benefits and social assistance by obtaining employment or becoming self-employed and possibly increasing their earnings. Included under this issue are the ratio of employment income to income from EI, the extent to which the primary targets of EI unpaid benefits were met, and how program and delivery factors contributed to the meeting of targets.

Evaluation Research Questions


1.	Implementation and Planning
1.1	Does the implementation of the EBSM reflect the basic tenets/principles underpinning the EI legislation: elimination of duplication and waste? reduce dependency on EI benefits and help clients to obtain employment? integration of community labour market and economic development plans? commitment to partnership approach? flexibility at local level to make decisions? clients assuming more responsibilities?
1.2a	Was the mix of EBSM provided locally consistent with the employment needs of client groups, employer skill needs, and community labour market and economic development plans?
1.2b	In particular, to what degree did TWS and training purchases meet employer needs?
1.3a	To what extent have the EBSM been designed and implemented in partnerships with others in the community? Have the EBSM led to the development of partnerships or enhancement of existing partnerships among the various levels of government, employers, and community groups?
1.3b	What factors have facilitated/hindered partnership development for EBSM planning and delivery?
1.3c	How have Labour-Management Partnership Program (LMPP) and local boards influenced EBSM planning and delivery?
1.4a	To what extent were there flexibility to allow significant decision-making about implementation at the local level?
1.4b	To what degree were EAS used to support delivery of employment benefits?
1.4c	To what extent do the use of Part II initiatives reflect the needs of the community?
1.5a	How did the Region/HRCCs set their annual targets?
1.5b	Were the criteria appropriate?
1.5c	To what degree was it possible to report against the targets the needs of the community?
2.	Impact
2.1a	How satisfied are participants with delivery of EBSM and with programs and services provided under Part II initiatives?
2.1b	To what extent did participants discontinue before their anticipated completion date?
2.1c	What were the main reasons for discontinuation?
2.2a	How many clients have become employed or self-employed?
2.2b	What is the duration of employment and unemployment for clients?
2.2c	Are there differences based on the type of intervention?
2.2d	To what extent did the employment accountability measure capture the impact of EBSM on actual client employment?
2.2e	What is the profile of clients gaining employment?
2.3a	To what extent have the Part II initiatives helped participants to reduce their dependency on EI benefits and/or Social Assistance? obtain or keep employment? increase their earnings?
2.3b	How much did clients earn from employment, employment and EI benefits combined, and from other sources?
2.3c	Have the primary targets in terms of employment and unpaid benefits been met?
2.3d	What factors (system, delivery, other) affected meeting the targets?
2.6a	To what extent does the EBSM encourage participants to assume more responsibility in the development and implementation of their action plan?
2.6b	Have clients who have received assisted services prepared action plans?
2.6c	Are they taking responsibility for their action plans?
2.6d	Did participants become involved in decisions related to their interventions?
2.6e	Are they following through on action plans?
2.6f	Did participants contribute to the cost of the intervention? Was the contribution appropriate to their circumstances?
2.6g	Is HRDC following-up?
2.7	Did the EBSM assist clients to develop a positive attitude to finding and maintaining employment?
3.	Service Delivery
3.1a	To what extent have specific client groups received the benefits and measures?
3.1b	Did the new policy result in preference for job ready clients and/or diverse client groups served by HRCCs?
3.1c	If the former holds, how can this situation be improved?
3.1d	What was the criterion used to select clients?
3.1e	How did the selection criterion influence delivery noted in 3.1a?
3.1f	To what extent has the profile of EI/reachback clients changed? Is this change reflected in clients receiving EBSM?
3.2a	Are the Part II initiatives accessible in the official language of choice where there is significant demand?
3.2b	Were those who received Part II initiatives in a minority official language, satisfied with the service?
3.3a	To what extent have EBSM resulted in "cultural" change in the HRCCs way of doing business? Specifically, to what extent has service delivery evolved to service management?
3.3b	To what extent is the case management approach a departure from the traditional approach? To what extent have the changes in case management been an improvement?
3.3c	How has delivery of employment benefits been affected by use of EAS? Has EAS increased access? Has EAS resulted in contributions from other organizations?
3.3d	Has client awareness of HRDC role/responsibility for EBSM been affected by EAS (and alternate service delivery)?
3.3e	To what extent have HRCCs/RHQ staff responded to changes in service delivery? Do staff understand their new roles and responsibilities? What factors have influenced the transition to service management?

Last Modified: 2002-11-29	Important Notices