Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The formative evaluation of Employment Benefits and Support Measures (EBSMs) presents an interesting perspective on the early results from the implementation of EBSMs in Human Resources Development Canada (HRDC's) Ontario Region. This perspective comes from both the breadth of its review (all of Ontario) and the inclusion of the voice of the client. While the evaluation did review a number of internal processes through document review and staff interviews, it is its detailed presentation of client issues and client impacts that are most unique. The retrospective assessment provided by the evaluation is an important supplement to day-to-day operational experience and plays a key part in both accountability and continuous improvement.

In reviewing the evaluation's key findings, it is encouraging to note evidence of positive client results. The report's findings also suggest several areas for improvement. The remainder of the management response will focus on addressing some of the key findings. It should be noted that some of the findings fall within themes explored in separate reviews undertaken by the Auditor General (AG) and HRDC's Internal Audit Bureau (IAB). Actions planned or in progress arising from these reviews will be referenced in our response where they relate to the evaluation findings.

Key Finding on Participant Access

Key Findings # 1 and # 2 fall within the broad theme of client access. The evaluation's analysis indicates that EBSM take-up is broadly based across client groups. For some groups, participation appears aligned with the proportions existing in the broader unemployed population, but for others participation is less.

HRDC acknowledges the report's finding on the relatively lower participation by designated group members, specifically persons with disabilities and visible minorities. The issue of under representation of some designated groups was identified in HRDC's 1999 Employment and Insurance Monitoring and Assessment report. It noted that past data capture and reliance on self-identification of designated group members were in part contributing factors to the reported under-representation.

HRDC also acknowledges the absence of information on the extent and characteristics of the reachback clients to analyze their participation in EBSM. The management response wishes to note that such analysis depends on information sources external to HRDC. HRDC takes great effort to undertake a demographic and needs analysis to plan and guide all its programs. From an operational perspective, HRDC strives to ensure that previous EI claimants have access to EBSM through approaches such as referrals from local social service offices and promotion in the community.

Key Finding on HRCC Business Planning

Key Finding # 3 notes that at the time of the evaluation there was a perception of "... a wide variation in the approach and quality of business planning between Human Resource Centres of Canada (HRCCs) that may limit effective programming." The variation in business planning approach and reporting is acknowledged. Given the emphasis in EBSM design on local flexibility, sound local business planning is essential.

HRDC Ontario Region is taking steps to strengthen business planning. In the fall of 2000, the Region commenced management consultations to promote the business planning process and establish key components for sound local business plans. The consultations are one element in a broader review of the HRCC business planning process. This review will examine the current state of business planning and methods for improvement.

Key Finding on Monitoring and Accountability

Key Finding # 5 presents a number of observations related to adequacy of information to monitor activity, measure results and plan. In general, the weakness identified in the monitoring and accountability of EBSM is acknowledged. This key finding is of similar theme to that identified by the Auditor General and the Internal Audit Bureau and is currently being actioned by HRDC. These actions will help alleviate weakness also noted in the EBSM evaluation.

In its response to the Auditor General's Report, HRDC indicated that in May 2000, it embarked on a program management study to develop results-based management and accountability frameworks for all of its programs (AG recommendation 3). It also committed to develop a new information system for Grants and Contributions, of which EBSM is part (AG recommendation 5). Finally, the Internal Audit Bureau (IAB) report on HRDC's contracting procedures has resulted in a "Six Point Action Plan" that has improved monitoring of the activity and results of third party service contracts.

Key Finding on: Client Opinion of EBSMs and Services

Key Findings # 4, # 6, # 8 and # 9 present client opinions on their experiences with the process and service delivery associated with EBSMs. These opinions, collected within a structured, independent surveying of clients, provide valuable insights that extend beyond the anecdotal. The clients' recognition of their personal responsibility in EBSM (Finding # 4) and awareness of HRDC's support (Finding # 8) is very gratifying.

The level of satisfaction expressed by clients (Finding # 6) and the access by franco-Ontarians to EBSM in their preferred official language phone is also heartening, however as the analysis indicates, improvements are required. Quality of client service is a key element of HRDC's vision and quality service standards are in place. Within its quality service process HRDC will take steps to address these issues raised in the evaluation.

Key Finding on Employment Outcomes

Key finding # 10 highlights the employment outcomes of EBSM participants. The evaluation results are taken as encouraging signs that the EBSM program is meeting its key objective. The variation in employment existing between EBSMs is expected. In particular, the evaluation's finding of lower impact from short-term group sessions supports the change made in 1999 that tightened the method of calculating the found work performance indicator for Group Information Sessions. This change resulted in a lower employment estimate being attributed to participation in these sessions.

The management response acknowledges the report's caution in interpretation of the incremental impact analysis due to the relatively short time span after completion. The need for supplementary impact analysis is recognized. At present, it is intended to conduct a summative evaluation of EBSM in Ontario in 2001/2002. In addition and as the evaluation report refers, HRDC has commenced a research project that will endeavor to measure medium-term impacts from participation on EBSM. Both projects will examine the client implications of EBSM beyond the limited time horizon explored in the formative evaluation.