Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Formative Evaluation of Employment Benefits and Support Measures Delivered Under the Canada/Newfoundland LMDA
Phase 1 — Employment Benefits

Final Report

Evaluation and Data Development
Strategic Policy
Human Resources Development Canada and the
Government of Newfoundland and Labrador

June 2000

SP-AH190-06-00E
(également disponible en français)

The Canada-Newfoundland and Labrador Agreement on Labour Market Development (LMDA) is an Agreement between the Government of Canada and the Government of Newfoundland and Labrador relating to the delivery of federal employment programs and services within the Province of Newfoundland and Labrador. The Agreement was signed on March 24, 1997 and came into effect as of that date. The timeframe for the Agreement is unlimited and the parties agreed to review it in the 2000/2001 fiscal year.

Under the Agreement, Human Resources Development Canada (HRDC) remains responsible for delivery of federal employment programs and services in the Province. The major change relative to past practice is that the Province of Newfoundland and Labrador co-manages these programs and services with HRDC.

The LMDA included commitments to evaluate "the results of the employment benefits and support measures". The current evaluation is a formative evaluation which addresses three employment benefits — Job Creation Partnerships (JCP), Targeted Wage Subsidies (TWS) and Self-Employment (SEB).

The evaluation examines the effectiveness of these employment benefits from the signing of the agreement in March 1997. The evaluation does not address the effectiveness of training during that time which will be addressed via a separate evaluation.

Key findings for the evaluation relate to three groups of evaluation issues: relevance, design/delivery and success.

Relevance

The relevance issues identified for the evaluation relate to the extent to which the EBSM's correspond to:

• the statutory requirements of the EI legislation;
• the priorities of federal and provincial governments;
• the needs of individuals; and,
• local economic development priorities.

Conformance to EI legislation

The EBSM's delivered under the Agreement correspond to the EI legislation and related policy directives of the Government of Canada. No significant changes have been made to either the design or the delivery of the EBSM's under the Canada-Newfoundland and Labrador Agreement on Labour Market Development.

Conformance to the Priorities of Federal and Provincial Government

The priority of the Government of Canada as regards EBSM's is to provide assistance to active EI claimants¹ to facilitate their return to employment. The Government of Newfoundland and Labrador accepts this priority and also seeks to address its own priorities:

• to ensure that reachback clients have the same degree of access to EBSM's as current EI claimants;
• to ensure that EBSM expenditures are made in a way as to be consistent with and complementary to the economic develop priorities and initiatives of the Government of Newfoundland and Labrador as well as those of Regional Economic Development Boards (REDB's) across the province; and,
• to enhance the capacity and ability of the Government of Newfoundland and Labrador to serve its citizens as regards employment programming and social and economic development.

Under the co-management approach, it is clear that each government has been able to pursue its respective priorities.

The Government of Canada's priority is dictated by the EI legislation which. Under the Agreement, HRDC has continued to be governed by this legislation.

The co-management process has enhanced the ability of the province to pursue its priorities in several ways, which were noted by key informants:

• the business planning process includes targets (provincially and sub-provincially) for reachback clients and various industry sectors;
• provincial government representatives on local Project Assessment Committees examine the conformance of proposed projects to social and economic priorities of the area;
• staff of the provincial Department of Human Resources and Employment have received training in the Service Needs Determination (SND) Process which allows them to improve the referral of eligible SAR reachback clients to HRDC. It also improves their ability to assist other unemployed individuals who are not eligible for EBSM's.

Conformance to the Needs of Individuals

This evaluation has found short term post-project employment benefits for Targeted Wage Subsidies (TWS) clients and for self employment (SEB) clients which suggests that the needs of these clients have been addressed. This is also supported by the views of key informants and the high satisfaction of participants with the program and their experiences under the program.

Participants in Job Creation Projects (JCP) have not been as successful in the labour market after their project participation. Key informants indicated that such gains would not be expected. They saw the primary focus of JCP as being to support economic development initiatives as well as to provide short term income to participants. The providing of new skills and experience to participants was generally seen as a lower priority. It was beyond the scope of this evaluation to assess the effectiveness of JCP in stimulating economic development.

Conformance to Local Economic Development Priorities

The evidence from this evaluation suggests that local economic development priorities are an important consideration regarding EBSM expenditures. As noted, it was beyond the scope of the evaluation to assess the effectiveness with which these priorities have been pursued. However they clearly have been pursued:

• Business Plans (Provincial, District and Local) identify specific targets for priority sectors. Processes are in place to monitor the achievement of these targets and to report deviations from plans to the Provincial Management Committee and to District Management Committees.
• ACOA, the Department of Development and Rural Renewal, the Department of Education, and Regional Economic Development Boards participate in District Management Committees. They also participate on local Project Assessment Committees.

Design and Delivery

Design and delivery issues for this evaluation address:

• the extent of co-ordination between the Government of Canada and the Government of Newfoundland and Labrador;
• the responsiveness to local conditions in the delivery of the EBSM's;
• the adequacy of administrative data for evaluation and monitoring; and,
• client characteristics and targeting.

While all these issues have been addressed, the Canada-Newfoundland and Labrador Agreement on Labour Market Development allows for significant innovation, flexibility and accountability at local and district levels. It was beyond the resources of this evaluation to examine the consequent variability in a detailed fashion.

Co-ordination between the Federal and Provincial Governments

Key informants consistently were positive about the extent and success of co-management under the Agreement. In particular, key informants noted:

• the value of the Business Planning process which has been applied at the district and provincial levels;
• the much improved knowledge that each level of government now has of the other's programs and priorities;
• the consensus decision making approach used by the provincial and district management committees and by some local Project Assessment Committees;
• the increasing integration of staff training across the two levels of government; and,
• the increased partnership between the two levels of government as regards programming not covered by the Agreement (e.g. Canada Jobs Fund; Fisheries Restructuring & Adjustment Measures (FRAM); and, summer employment programs).

Negatives noted regarding co-management and the resulting co-ordination included:

• delays in finalizing operational budgets attributed to the Business Planning process;
• need to re-allocate budgets to deal with under-commitment and/or over-commitment of available funds in some funding envelopes;
• initial difficulties faced by the Government of Newfoundland and Labrador as it needed to develop its capacity in order to fulfill its role under the Agreement; and,
• expectations that co-management is a temporary mechanism and will be replaced by devolution of employment programming to the Government of Newfoundland and Labrador in the medium term.

Responsiveness to Local Conditions

Key informants noted that local Project Assessment Committees have increased the range of input to decisions as to projects to be implemented in local areas.

Adequacy of Administrative Data

Electronic information for the EBSMs is maintained in two separate systems. Within the HRCCs, staff use NESS to enter relevant information while outside of the HRCCs the counsellors at third party delivery sites use CATS. The CATS data contains information downloaded periodically from the NESS system, so that counsellors at these third party sites have access to client history similar to that available in the HRCCs.

Data from both NESS and CATS are used in the construction of the results data set, which uses a combination of EI data and the action plan information and follow-up to determine which clients have successfully returned to work, and to calculate any resulting EI savings. The results data set is then processed by the regional HQ to produce regional level reports on achievement of targets.

The current system provides a reasonable basis for tracking clients who are participants in EBSMs. In combination with EI data files a good client contact list can be produced. However there are two serious deficiencies in the current data management system.

• Successes (clients returned to work) recorded in the data do not correspond with actual client experiences as measured via the follow-up survey conducted in this evaluation. The calculation of success using EI information (clients who return to work within their EI period) is substantially more accurate than the calculations that have to rely solely on information entered by counsellors.
• Many clients who have TWS placements are not recorded in the data. This occurs since HRDC makes payments to the employers; not directly to clients so no client contract has to be set up. This problem is most prevalent with employers who have multiple placements. For the province as a whole, data is available for 46% of TWS participants. In the Avalon District data is available for only 32% of TWS participants.

Client Characteristics and Targeting

Under the Agreement, targeting provisions exist for priority sectors and for Social Assistance Recipients (SARs) who satisfy the eligibility provisions of the EI legislation. For the first two years of the Agreement, targets were set at relatively modest levels. These targets have guided the District Management Committees as well as local Project Assessment Committees and staff of HRDC and third party organizations contracted to provide services. However, key informants noted that the modest targets meant that major changes did not occur relative to the past. In particular, it was noted that project applications as well as participant applications were generally assessed based on their merit rather than on targeting provisions.

For the 1999/2000 fiscal year, targets for SARs were doubled relative to those for 1998/1999. Achieving these targets is likely to be a greater challenge and will necessitate co-ordination at the operational level between HRDC and the Department of Human Resources and Employment (HRE). Training has been provided to HRE staff across the province to allow them to fulfill their responsibility to identify and refer eligible SARs to HRDC.

Success

The success issues identified for the evaluation address:

• the short-term post-program employment experiences of participants;
• the effectiveness of case management;
• the satisfaction of participants and employer/sponsors;
• the extent to which the experiences of SARs differed from those of other participants; and,
• the extent to which targets for employment and savings to EI have been met.

Short-term Post-Program Employment

Based on 1,494 participants surveyed during this evaluation, participants have been employed on average 49% of the time since completing their intervention. This varies substantially by intervention type:

• SEB participants have been operating their business or employed for 89% of the time since HRDC funding support was terminated;
• TWS participants have been employed 64% of the time since their wage subsidy ended; and,
• JCP participants have been employed 46% of the time since the project ended.

Statistical regression models indicate that for SEB and TWS this post-program employment is higher than would otherwise be expected. The models employed adjusted for other measurable characteristics which are known to affect employment success (e.g. age, education, prior labour market experiences, etcetera) but do not adjust for selection bias. Separate models were employed for males and females. These models indicated the following:

• SEB participants have been in business or employed more than 35 percentage points more of the time than would be expected based on their other characteristics (36 percentage points for males and 39 for females).
• TWS male participants have also been employed 26 percentage points more than would be predicted from the models. The post-program employment for female TWS participants is estimated at 6 percentage points higher than would otherwise be expected. This result is not statistically significant.

JCP participants post-program experiences are closer to what would be predicted based on their other characteristics. Male JCP participants have been employed an estimated 12 percentage points more of the post-program period than the model predicts while females have been employed 9 percentage points less that the model predicts.

All the above data is short term in nature with most participants interviewed within a year of their program completion date. It is likely that this short-term orientation explains at least part of these results. In particular SEB participants are likely to continue in business after their subsidy ends and many TWS employers retain participants after termination. Consequently, participants from both interventions can be expected to do well in the short term. Conversely, almost all JCP participants will be unemployed at the end of their project and poor short-term post-project employment is expected.

Effectiveness of Case Management

The evaluation did not examine case management processes and their effectiveness in depth. Based on the evidence obtained, the following conclusions can be drawn:

• 64% of HRCC staff describe the current method of case management as very effective compared to only 32% of staff of third party providers;
• the "Action Plan" which participants must complete is not, in retrospect seen as a valuable process. Only 12% of participants surveyed, recalled preparing an action plan; and,
• participants were generally satisfied with the level of service they received from the HRCC or Outreach office with 80% rating their satisfaction at 8 or higher on a 0 to 10 scale.

Satisfaction of Participants and Employers

Participants were satisfied with HRDC staff and service but less so with the value of their project experiences:

• 80% were very satisfied² with the level of service received; and,
• 72% were very satisfied with the variety of services available.

but, only,

• 47% gave equivalent ratings for how close the work was related to their career objectives; and,
• 28% indicated the program was very useful in helping them find a job.

Employers and sponsors were highly satisfied with all aspects of the program:

• 76% were very satisfied with the participant's skills;
• 89% were very satisfied with HRDC administration; and,
• 81% rated the value of the participant's work very high.

Success of Social Assistance Recipients

The regression models of percentage time employed after the program indicate that SAR participants have somewhat less success in the post-program period than individuals who were otherwise similar. However, these effects were small (13 percentage points for males and 7 percentage points for females).

Employment and EI Savings Targets

HRDC administrative data indicates that employment and EI savings targets have been met. The evaluation indicates that HRDC's estimate of "returns to employment" is low based on data from participants surveyed. TWS, SEB and JCP do not account for a large share of EI savings attributed in HRDC's administrative data.

Joint Management Response — Formative Evaluation of EBSMs under the Canada-Newfoundland and Labrador Agreement on Labour Market Development Agreement

The Canada-Newfoundland/Labrador Agreement on Labour Market Development (LMDA) is an agreement between the Government of Canada and the Government of Newfoundland and Labrador relating to the co-management of federal employment programs and services specifically under EI Part II.

Section 12 of the LMDA outlines the requirement of the Labour Market Partners to cooperate in developing and implementing a two-phase evaluation framework. To this end the Management Committee is responsible for the completion of all evaluations related to the employment benefits and support measures. The Management Committee is pleased that the first phase of the evaluation process, namely the formative evaluation, has been completed. The formative evaluation is comprised of three associated reports. The first report (June 2000) covers three Employment Benefits, i.e., Targeted Wage Subsidy (TWS), Self-Employment Benefit (SEB) and Job Creation Partnership (JCP). The second report (June 2000) deals with the Training Benefit (currently Skills Development), and the final report (June 2001) deals with Support Measures, i.e., Local Labour Market Partnerships (LLMP) and Employment Assistance Services (EAS).

The evaluation evidence from the three formative evaluation reports completed to date has shown that in the near term, clients served by the employment benefits and support measures have been moderately successful in achieving their employment goals. Labour Market Development Agreement accountability targets for unpaid EI benefits and client returns to employment have been met in each fiscal year since the inception of the agreement. The Management Committee anticipates that the summative evaluation report, which will contain more definitive evidence, will confirm these findings in the longer term. The Management Committee looks forward to the commencement of the summative evaluation in 2002.

The formative evaluation reports have identified many specific findings. The Management Committee suggests that we can deal with these findings under general categories: the functioning of the co-management relationship, findings related to programs/services and findings related to client impacts. A joint Management Committee response to each group of findings is reported below:

Co-management:

The evaluation reports suggest that "under the co-management approach, it is clear that each government has been able to pursue its respective priorities." The Management Committee suggests that this is perhaps one of the most fundamental achievements of this co-managed agreement, given the unique cultures and priorities of both levels of government. The Management Committee also feels that the effectiveness of the co-management approach is primarily attributable to the level of communication and co-operation among partners.

The reports suggest that co-management has increased administrative responsibilities in terms of business and financial planning. The Management Committee feels that it has resulted in a more targeted response to the needs and opportunities of unemployed individuals. The Management Committee also believes that these pressures are abating as the partners become more knowledgeable of each other's programs and collective priorities.

Programs & Services Finding:

The evaluation evidence suggests that eight out of ten individuals who received services from the HRCCs and Employment Assistance Services offices had a high level of satisfaction with that service. The Management Committee feels that this evidence reflects a strong service commitment of HRCC staff. The committee also believes that despite a challenging economic climate in many rural parts of the province, HRCC staff are making a valuable contribution to the lives of individuals in these communities.

With respect to the training benefit two findings are noteworthy from the Management Committee's perspective: Ninety percent of the sample of training participants indicated that participation in the training benefit made them more employable. However, it is recognized that a lack of participant information at this early stage of evaluation, particularly with two and three year programs, did not allow a complete analysis of the impact of training on employment. The Management Committee is interested in the long-term impact of training on subsequent employment gains and will examine this issue in the summative evaluation. The Management Committee has also noted that graduation rates are somewhat unclear given the limitations of administrative data. This issue must be addressed so we can more closely track individual impacts resulting from training.

Participant surveys highlight that the support measures offered were well received by program participants. The Management Committee feels that the support measures are essential elements of the LMDA. The measures allow the Management Committee to provide support to incremental initiatives that have many positive impacts for individuals and communities throughout the province. The evaluation suggested that Employment Assistance Services participants welcome the convenience of their being served in their geographic area. " The main tangible benefit emerging from the first three years of EAS seems clearly to lie in the improved employability of the client group." However, additional evidence is needed to determine the overall impact on employment gains that these services are providing. This work will be undertaken in the summative stage of the evaluation process.

Local Labour Market Partnerships (LLMPs) have also furnished the Management Committee with the capacity to engage numerous community partners in various incremental activities. The evaluation evidence suggests many valuable activities have taken place, which, in the absence of LLMP, might not have come to fruition. The Management Committee feels that the employment benefits of these initiatives will be realised in the longer term.

Client Impacts:

The evaluation evidence suggests that SEB and TWS participants experience positive employment gains in the post-program period. This incremental impact shows that investment in these benefits is a prudent expenditure of public funds. The Management Committee desires to maximize the potential of these benefits while recognizing their finite growth potential, given the modest employer and market base in the province.

The evaluation indicates that JCP participants did not realise significant post-program employment gains, however, JCP earnings did improve the lives of these participants, their families and the communities in which they live. These results must be interpreted with the knowledge that JCP participants had the lowest levels of pre-program earnings and education of all participant groups. Given this information, the management committee submits that the impact of JCPs on longer-term employment gains must be established before a judgement can be made about the overall utility of JCP participation.

The engagement of Social Assistance Recipients in LMDA programs was a priority of the LMDA partners. The report outlines that these targets were met and that SARs achieved positive outcomes from the programs offered. However, the disproportionately higher level of SAR participation in JCP (compared with other interventions) and the subsequent lack of post program employment are of concern to the Management Committee. Improvement of assessment, selection and referral of SAR clients within the EBSM model is a priority of the LMDA partners. It is believed that this approach will lead to a more balanced level of participation of SAR clients in all benefits and measures. A determination of the utility and impact of this new approach should be one of the priorities of the summative evaluation.

Aspects to monitor and evaluate:

While the Management Committee is generally pleased with the findings of the three Formative Evaluation reports, challenges remain. The following is a list of LMDA dimensions that the Management Committee feels is in need of additional monitoring and evaluation: The Training Benefit report suggests that in the early stages of implementation of Negotiated Financial Assistance (NFA), there was a perception of unequal treatment of clients in terms of the levels of funding received. At the time of the evaluation, Negotiated Financial Assistance was a new process for both clients and staff alike. The problems with dissemination of information and the generally low level of understanding of the process lead to concerns on behalf of both clients and staff. The Management Committee agrees with the consultant's recommendation that this area of concern should be closely monitored to ascertain whether NFA principles are used consistently. Nevertheless, the Management Committee senses that with the passage of time, the intent of NFA has been more widely understood and accepted by all parties. The NFA issues that were identified in the formative evaluation should however be re-examined in the summative evaluation to determine the extent to which these concerns persist.

The evaluation evidence suggests a negative employment gain for female JCP participants. This is of concern to the Management Committee who believes that JCP has much economic and social merit. Perhaps more careful targeting of JCP must take place to ensure positive impacts for participants in the post-program period. Again, the summative evaluation should provide more definitive evidence as to the long-term employment impact of JCP. In addition, decisions regarding SAR participation should not be made based on point estimates from the initial evaluation.

The evaluation report outlines a data concern associated with Targeted Wage Subsidy administrative data. While this issue has not limited the effectiveness of this benefit, the absence of reliable data for administrative and evaluation purposes may bias our assessments of success or failure, so the data must be improved. The consultant also reported that based on the employers' survey responses, a proportion of the participants would have been hired without the subsidy. The Management Committee recognizes that this finding may outline a potential weakness in the current TWS model. The Management Committee does not however feel entirely comfortable with the integrity of this initial finding. Prior to making any potential adjustments to the program, a more precise and rigorous examination of this issue must be explored.

Support measures continue to provide excellent employment assistance and partnership opportunities within the local labour market. The Management Committee considers that the increased level of funding to these activities over the initial period of the LMDA will necessitate further detailed analysis and ongoing monitoring of the effectiveness of these support measures, through the summative phase of the LMDA operational and evaluation plan.

The Canada-Newfoundland and Labrador Agreement on Labour Market Development (LMDA) is an Agreement between the Government of Canada and the Government of Newfoundland and Labrador relating to the delivery of federal employment programs and services within the Province of Newfoundland and Labrador. The Agreement was signed on March 24, 1997 and came into effect as of that date. The timeframe for the Agreement is unlimited and the parties agreed to review it in the 2000/2001 fiscal year.

The Agreement is a comprehensive one, which addresses a variety of issues. It differs from agreements in several other provinces. In particular Human Resources Development Canada (HRDC) remains responsible for delivery of federal employment programs and services in the Province. The major change relative to past practice is that the Province of Newfoundland and Labrador co-manages these programs and services with HRDC.

The LMDA included commitments to evaluate "the results of the employment benefits and support measures". The current evaluation is a formative evaluation. A summative evaluation is scheduled for 2000/2001 and will cover the first three years of the agreement. A brief description of the EBSMs that are examined in this formative evaluation is given in Section 1.2.

This evaluation has been conducted based on the evaluation issues and methodologies specified in Evaluation Framework for the Formative Evaluation of EI Part II under the Canada-Newfoundland and Labrador Labour Market Development Agreement, August 1998.

1.1 Structure of the Report

This report describes the results of the formative evaluation and is structured as follows:

• Chapter 1 describes the LMDA, the EBSMs delivered under the Agreement, the scope of the evaluation and the methodology applied to conduct the evaluation;
• Chapter 2 provides a profile of participants in the EBSMs as well as profiling employers and sponsors;
• Chapter 3 describes the impact of the EBSMs on participant employment and employability in the post-program period;
• Chapter 4 addresses management and process issues;
• Chapter 5 addresses the relevance and appropriateness of the EBSMs to the current labour market conditions in Newfoundland and Labrador; and,
• Chapter 6 provides the Conclusions.

1.2 Employment Benefits and Support Measures

1.2.1 Employment Benefits

There are five employment benefits identified in Part II of the Employment Insurance Act:

• Self-employment Benefit (SEB);
• Job Creation Partnerships (JCP);
• Skills Loans and Grants and Training;
• Targeted Earnings Supplements (TES);
• Targeted Wage Subsidies (TWS).

The TES Benefit has not been used in Newfoundland and Labrador and training is being addressed via a separate evaluation. The remaining three employment benefits are defined as follows:

Self-employment Benefit

The Self-employment intervention is designed to help individuals with sound business ideas to start their own businesses by offering financial assistance, mentoring/coaching and technical help, including advice on business planning and operations. Clients accepted into SEB continue to receive their insurance benefits for the duration of their claim and most clients are eligible to receive financial assistance for up to an additional 52 weeks under Part II of the Employment Insurance Act. Participants with a disability may receive financial assistance for up to an additional 78 weeks under Part II of the Employment Insurance Act if the disability represents a barrier to making the business sustainable within one year. Clients may also receive assistance for personal expenses related to establishing the business such as childcare.

Job Creation Partnerships

JCP involves the creation of incremental and meaningful work opportunities for clients, through partnerships involving HRCCs, the provincial government, the private sector and community groups. Participation is aimed at allowing participants to maintain and develop skills in order to re-enter the labour market. Projects are expected to be linked to provincial and local community plans and priorities. Participants continue to receive insurance benefits, which are topped up to the level of a prevailing wage. Earnings are non-insurable.

Targeted Wage Subsidies

TWS is designed to encourage employers to hire targeted clients they otherwise might not employ and provide them with on-the-job experience. The subsidy is for a maximum of 78 weeks and for a maximum of 60% of the wages paid to the individual.

1.2.2 Support Measures

Two support measures are also delivered under the Canada-Newfoundland and Labrador Agreement on Labour Market Development.

Employment Assistance Services

Employment Assistance Services (EAS) are offered through agreements with sponsors —organizations and community groups — to help unemployed individuals become employed. Local Human Resource Centres of Canada (HRCC's) determine the type of EAS agreements they will arrange with sponsors after they have consulted with the local community and in the process of developing the HRCC business plan. For example, one organization might be contracted to provide a full range of services to blind and visually-impaired individuals while another might provide services to all unemployed individuals in a remote community. The services provided by these organizations are not restricted to EI-eligible individuals.

Local Labour Market Partnerships

LLMPs are mechanisms to help communities and employers adjust to changing circumstances. LLMPs promotes community partnerships to develop strategies expanding a community's employment base, enhancing the potential for economic success. LLMPs can support partnership development, leadership development, communications and promotion. Financial assistance for LLMP is provided through contribution agreements for overhead costs and can include:

• wages and employment associated costs for staff;
• professional service fees;
• disbursements for research and technical studies;
• commuting, rental of equipment or of premises, etcetera.

1.3 Scope and Methodology of the Evaluation

The evaluation examines the effectiveness of the EBSMs from the signing of the agreement in March 1997. The evaluation does not address the effectiveness of training during that time which will be addressed via a separate evaluation. This evaluation also does not directly address the Self-Employment Benefit in the Western Region since a recent evaluation had been conducted³ and it was not reasonable to re-interview participants.

The major focus of the evaluation has been on determining the short-term post-program employment experiences of participants and assessing the accountability framework results.

The evaluation has relied on a variety of methodologies as follows:

• An extensive survey of 1,494 participants. The survey interviews averaged 15 to 20 minutes in length and addressed in detail the labour market experiences of participants subsequent to program participation as well as collecting information on pre-program and program experiences. The survey comprised three cohorts as follows:

• Cohort 1 — a stratified random sample of 448 participants from the pre-LMDA period; i.e. whose interventions commenced between July 1, 1996 and March 31, 1997;

• Cohort 2 — a stratified random sample of 819 participants under the LMDA. These individuals were selected from those who had started an intervention on or after April 1,1997 and had completed their intervention by February 28, 1999;

• Cohort 3 — a supplemental sample of 227 reachback clients who had started an intervention on or after April 1, 1997 and had completed their intervention by February 28, 1999. This cohort comprised an exhaustive attempt to interview as many reachback clients as possible⁴.

  Analysis of survey results is based on weighted data. The weights adjust for the sampling design used and ensure that estimates are unbiased estimates of the whole population. The weights have been normalized based on the expected number of completed interviews (1,502).

• a comparison sample of 400 individuals matched to post-LMDA participants⁵;

• regression modelling to estimate the differences between the short-term post-program employment experiences of participants and the comparison sample. These models adjust for several characteristics which affect labour market success but do not adjust for selection bias;

• extensive administrative data analysis to examine the workings of the Accountability Framework

• a survey of 98 employers/sponsors from TWS and JCP;

• a survey of 76 front-line staff at HRCCs and third party providers⁶;

• a detailed review of relevant documents;

• a focus group with HRCC employment counsellors and needs determination officers in Corner Brook;

• five case studies of Job Creation Partnership projects in three regions (Avalon, Central and Western). These case studies included interviews with project sponsors, HRDC officials and project participants;

• case studies⁷ examining the implementation of co-management at the local level (Stephenville, Gander, Happy Valley-Goose Bay and St. John's). The Stephenville case study is based on key informant interviews (individually and in groups) with representatives of district and local management committees as well as interviews with HRDC project officers. The Gander and St. John's case studies addressed the same issues but involved interviews only with HRDC and HRE officials. The Happy Valley-Goose Bay case study is based on telephone interviews with the co-chairs of the District Management Committee and the co-chairs of the local Project Assessment Committee.

This chapter of the report profiles the characteristics and experiences of EBSM participants. Section 2.1 is based on administrative data for all participants in the July 1996 to December 1998 period. Section 2.2 is based on the survey of participants conducted for this evaluation.

2.1 Administrative Data Profile of Participants

HRDC administrative systems relating to EBSMs consist of two primary files:

• Contracts File. This consists of information on the sponsors (and the project sponsored) for TWS and JCP. The primary purpose of the Contracts file relates to financial accountability but individual records also contain information on the project (e.g. expected number of participants and expected start and end dates). Non-financial data is not necessarily updated, however, as the project evolves.
• Intervention File. This file has a record for each individual participant. For JCP and SE, coverage is complete since there are direct financial payments (in addition to amounts paid in EI Benefits) to individuals. For TWS, however, there is no direct financial relationship between HRDC and participants and coverage of TWS participants on the Intervention File is incomplete⁸.

Determination of the number of participants in any given time period is thus inexact. In the time period April 1, 1997 to December 31, 1998, an estimated 7,029 new participants came into one of the three employment benefits addressed by this evaluation. JCP and SE participant counts are based on the Interventions file. The TWS counts below were obtained as follows:

• for TWS projects where there are participants on the Interventions file, it was assumed that coverage was complete and the No. of Participants field (no_of_part) on the Contract file was ignored;
• for TWS projects where there are no participants on the Intervention field, the no_of_part field from the Contracts file was used.

As noted in the table, HRDC information systems only specifically identify 1,004 of the estimated 2,161 TWS participants. Overall, 46% of TWS participants were on the Interventions file. This varied by region⁹ from a low of 32% in the Avalon District to a high of 59% in the Central District. Coverage for the other districts was 51% for Labrador and 56% for Western.

Table 1 also identifies the number of reachback clients as well as the number of Social Assistance Recipients (SARs) who participated by intervention and by region. These numbers are somewhat underestimated since whether the unknown TWS participants are reachback clients or SARs¹⁰ is unknown.

View Table 1

For the participant survey, not all of these individuals could be included on the survey frame. In particular:

• as noted, no information was available for 1,157 TWS participants;
• individuals for whom contact information was not available were dropped;
• individuals who would not have completed their intervention by the end of February 1999 were dropped;
• when multiple records existed for an individual, only one case was retained.

This left a total of 4,586 data records corresponding to participants who were known to have taken and completed one of the three interventions since April 1997. The survey frame also included 1,089 individuals who commenced an intervention between July 1, 1996 and March 31, 1997 (i.e. after the effective date of the new Employment Insurance legislation but prior to the effective date of the Labour Market Development Agreement).

Tables 2 and 3 provide data on the age (at the start of their intervention) and gender of these 5,567 individuals. Both the age breakdown of the clients and the gender breakdown vary by the type of intervention.

Age

The most notable result in Table 2 is that TWS participants are younger than other participants:

• 17% of TWS participants are under 25 and 38% are under 30;
• only 14% of TWS participants are 45 or older compared to 19% of SEB participants and 23% of JCP participants.

Also notable is that reachback participants are somewhat older than other JCP participants although this difference is not marked.

Gender

There is also variation by gender between programs. Overall, 66 percent of the participants were male which is slightly higher than for the EI population generally (61 percent of active EI claimants are male). The specific data for each intervention indicates that males are more likely to have participated in JCP. For TWS and SEB, more females have been enrolled (46% and 43% respectively) than their representation in the EI population (39%) would suggest.

Reachback Participants and Social Assistance Recipients

Table 1 indicates that 783 clients were classified as reachback clients in HRDC's information systems. This represents an estimated 11% of all participants. Table 1 also indicates that 1,015 participants had received Social Assistance in either the quarter their intervention started or the previous quarter. Despite the similarity in these totals, these are generally two distinct groups of clients as indicated in Table 4. The table indicates that only 24% of reachback clients had received Social Assistance in the quarter their intervention started or in the previous quarter. This compares to 17% of non-reachback clients having received Social Assistance in the equivalent time period.

Especially pertinent in Table 4 is that only 185 of 5,515 participants (3%) are recent SARs who were classified as reachback clients in HRDC data.

2.2 Survey Data Profile of Participants

Table 5 outlines demographic and social characteristics of participants as collected in the EBSM participant survey. As the table shows, Job Creation Partnership participants are more likely to live in small communities and have lower levels of formal education than participants in other types of interventions. Reachback clients are essentially similar to other JCP participants in terms of these characteristics.

2.2.1 Pre-program Situation

Participants were asked a series of questions about their situation before participating in the program and their motivation for participating. There are some differences between the participants of the different programs. Respondents provided the following information about the 12 months prior to their program participation:

• Full-time school attendance. Targeted Wage Subsidies participants are the most likely (29 %) to have spent time in school in the 12 months before participating in the program compared with 16 percent of SEB participants and 12 percent of JCP participants. Almost all of these participants were in school full time. Reachback clients were somewhat less likely than other JCP participants to report attendance at school with only 10% reporting 1 or more months at school in the 12 months before entering the program.
• Full-time employment. Self-Employment Benefit participants are the most likely (64%) to have been employed full time at some point in the 12 months before starting the program. This compares with 44 percent of TWS participants and 46 percent of JCP participants having been employed full time at some point during the year before starting the program. For reachback clients, only 35% reported a period of full-time employment in the preceding 12 months.
• Lack of Unemployment. SEB participants are the most likely (41%) to report 0 months of unemployment in the 12 months before commencing the program. For TWS participants surveyed, 25% reported 0 months employed and for JCP, 23% did. Among reachback clients, 16% did not report any period of time unemployed. Presumably, many of these individuals experienced periods of unemployment of less than 1 month. Others may have not included their waiting time to get into an intervention in the pre-program 12 months. Since this result had not been anticipated, the questionnaire did not investigate this situation. Clarification of this surprising finding should be an important goal for the summative evaluation.
• Out of labour force. A very small group of participants identified time out of the labour force in the 12 months prior to their participation. Specifically, only 5% of JCP participants, 4 % of SEB participants and 3% of TWS identified any period of time when they were a homemaker or otherwise out of the labour force for reasons other than school attendance. Among reachback clients, 7% reported some time out of the labour force with about half of these identifying the entire 12 months as out of the labour force.

Those who had been unemployed at some point during the year before participating in the program were asked how many hours they spent looking for work in a typical week. JCP participants tended to report having spent the longest, reporting a median of 10 hours per week compared with 8.6 hours for SEB participants and 5.6 hours for TWS participants.¹⁶ By far the two most common job search methods were checking at the HRCC (whether using job boards, kiosks or job banks) and sending out resumes and applications.

Respondents were also asked why they thought they had been unemployed during this period. A large majority of 86 percent of respondents mention high unemployment rates or a lack of jobs as the reason. Only 6% cited their occupation or training as the issue and only 2% cited lack of experience.

Table 6 outlines how participants first heard about the program and the main reason why they decided to participate. As the table shows:

• TWS participants were much more likely to have found out about the program from the employer while JCP participants (especially reachback participants) became aware of the program from a notice in an HRCC or employment centre; and,
• JCP participants (especially reachback participants) were the most likely to have participated in order to increase their income.

Participants were asked whether the experience they gained through the program has made them more employable. Most responded positively. Specifically, 84 percent of TWS participants, 78 percent of SEB participants and 74 percent of JCP participants say yes. When asked in what way, almost all respondents say either that they now have the work experience or that they now have specific work experience in a particular field. However, since only 2% had identified lack of experience as a reason for their pre-program unemployment, these very positive results must be discounted somewhat. Sections 2.2.2 and 3.1 provide results on participant's actual post-program experiences.

2.2.2 Post-program Situation

A series of questions was asked to assess the work patterns of participants once they finished the program. Table 7 outlines some of the key results.

Participants who have found employment since they finished the program were asked what job search method led to their job. JCP participants were more likely to have found their job through an HRCC than were the participants of other types of programs.

In addition to finding employment, returning to school for additional training is sometimes considered a successful outcome. About 10% of participants (8 percent of TWS, 11 percent of SEB and 7 percent of JCP participants) have returned to school full or part time since they finished the program. For half of these TWS participants and 42 percent of these JCP participants the training is related to their work experience from the program. For virtually all of the SEB participants who subsequently took training, the training relates to their business.

Participants who are unemployed were asked why they think they are unemployed. For all interventions, participants cited economic conditions above all other reasons. Overall, 85 percent of respondents give a high unemployment rate or general lack of jobs as the main reason. TWS participants are slightly less likely to give this answer with 73 percent giving this reason compared to 86 percent of JCP participants and 79 percent of SEB participants. Among unemployed TWS participants, 13 percent say a lack of demand for people with their skills or training is the reason compared with just 3 percent of JCP participants giving this answer.

2.3 Employers/Sponsors

The evaluation also included a survey of 98 employers/sponsors. Specifically, interviews were conducted with 59 TWS employers and 39 JCP employers/sponsors.

Characteristics of Employers

Table 8 provides data on the characteristics of the firms and organizations surveyed. These data are primarily based on survey estimates. The exception is that data on the number of individuals placed with the employer/sponsor is derived from HRDC administrative data. Notable results are as follows:

• 95% of TWS employers were private firms. By contrast the majority of JCP sponsors/employers were not for profit (51%) with only 23% private firms and 26% municipal government.
• TWS employers generally have a small number of full-time employees — 52% have four or less employees and an additional 18% have 5 — 9 full-time employees. JCP employers are even smaller with 71% having 4 employees or less.
• TWS typically involve placement of one individual with 72% of employers having only one participant at a given time. Only 5% of TWS employers had 5 or more participants. By contrast, larger projects are the norm for JCP with 44% of participants involving five or more participants.
• the jobs in both TWS and JCP appear to be largely incremental or growth related. The large majority of employer/sponsors of single participant projects (82% for both TWS and JCP) indicated that the hiring was for a new position. For multiple participant projects this is likely to be more the case. JCP hirings appear to be largely incremental with 82% of employers/sponsors with a single participant indicating that no one would have been hired without the subsidy. Conversely, deadweight loss appears to be a concern with TWS since 46% of employers indicated they would have hired the same individual without the subsidy.

Satisfaction of Employers

Employers were asked to rate their satisfaction on a 10-point scale with the quality of the match between the participant's skills²² and the organization's needs. As can be seen in Table 9, employers were generally satisfied with the skills of the participant. For TWS, 79% of employers surveyed rated their satisfaction 8 or more on a scale of 0 to 10. For JCP, satisfaction was slightly lower with 71% of surveyed employers rating their satisfaction at 8 or more.

Employers rated their satisfaction with HRDC administration even more positively with 89% rating their satisfaction at 8 or more and only 1% rating satisfaction at 3 or less.

TWS employers were satisfied with the amount of the wage subsidy. Specifically, 66% considered the amount of the subsidy about right and 25% considered it a bit low. Only 9% said the subsidy was way too low and no respondents said it was too high. Finally, a large majority (81%) of employers considered the value of the participant's work to be very good.

Participant Retention

Table 10 provides data on the extent to which participants were retained throughout and after their projects.

In this respect, the two interventions are very different. The majority (58%) of TWS employers continued to employ the participant at the time of the survey and several others (19%) had retained the participant after the end of the subsidy. For JCP these outcomes were very uncommon with only 8% of participants still employed at the time of the survey while 79% ended their project employment at or before their scheduled completion date compared to 23% of TWS participants.

Table 11 provides information on why employment ended in those cases where the participant was no longer employed with the project employer. For both TWS and JCP, the lack of work was the most common reason. For JCP, the inability to pay the participant once HRDC funding ceased was also commonly cited.

For those TWS participants who continue to be employed by the project manager, 91% continued in the same job while 9% had new duties. In terms of salary, 62% were paid the same salary as they received during the subsidy period and the remaining 38% received an increase in salary.

Training Provided to Participants

Table 12 provides data on training provided by employers. As can be seen, TWS employers (85% versus 51% of JCP employers) more commonly provided on-the-job training. Other types of training investments were relatively uncommon (23% for TWS and 18% for JCP).

3.1 Impacts on labour market activity

Participation in an Employment Benefit affects short-term employment. The size and direction of this effect varies among the interventions and between genders.

Table 13 provides the results of the regression modelling performed in this evaluation. The dependent variable used was percentage of time employed²³ in the post-program period.

The independent variables that were used to explain post-program labour market success are defined below. They include participation variables to allow estimation of the apparent effect of each of the three interventions. To enhance the precision of those estimates, 17 other independent variables are included. One of these (Months since program) is included since past research indicates that the effect of employment programs (whether positive or negative) changes over time. The other 16 independent variables adjust for other factors that are known to affect the success of individuals seeking employment (e.g. age, education, location, previous employment history, past use of EI or social assistance, etcetera). Consistent with the approach used in past evaluations of Canadian and U.S. employment programs, separate regression models were employed for males and females.

The regression models used did not include adjustment for selection bias. Selection bias occurs because candidates are not selected randomly for participation. Instead selection decisions are made by individuals who decide to apply for the program; by HRDC or third party officials; and/or employers or sponsors who decide whether to accept them. All three groups will make their decisions based on factors which can only be imperfectly quantified and are thus not typically included in an Ordinary Least Squares (OLS) regression model. Since participants may differ from the comparison sample on these unmeasured characteristics, regression models that do not account for selection decisions — such as those used in this evaluation - produce potentially biased estimates of program impact.

1. JCP Participation.
2. Participation in Targeted Wage Subsidies.
3. Participation in the Self-Employment Benefits program.
4. Reachback status. Applies both to participants marked as reachback and the comparison group members matched to those participants.
5. Graduation from high school.
6. Graduation from a college or university.
7. Average total income from 1991 to 1996 inclusive.
8. Attachment to the fishery (1 if any fishing income in any of the three previous years, 0 otherwise).
9. Urban/Rural (1 if St. John's, Mount Pearl, Grand Falls Windsor or Corner Brook).
10. Age and Age squared.
11. Aboriginal status (self-declared).
12. Visible minority (self-declared).
13. Disabled (self-declared).
14. Dependence on EI 1991 — 1996²⁴.
15. Central district.
16. Western District.
17. Labrador District.
18. Social Assistance Recipient (SAR) in quarter enrolled or previous quarter.

Interpretation of these coefficients depends on the variable and its units. For example, the coefficient of 17.729 for post-secondary graduation in the female regressions means that, on average, the model estimates that females who have a post secondary degree or diploma will be employed 17.729% more of the time than females who do not have such a qualification.

For both males and females the models are significant and explain a large amount of the variation in post-program employment with R²=.196 for males and R²=.257. These are strong results in explaining variation in employment of individuals and consistent with what is generally seen in the literature and other evaluations. For example, in Goss Gilroy's 1989 Evaluation of the Job Development Program — a national program targeting the long-term unemployed, extensive econometric modelling was conducted for various sub-populations. In models with a similar set of dependent variable, R²s in the 18% - 25% range were typical. At the extremes, individual regressions had R²s as low as 14% and as high as 36%.

As noted in the Job Development Evaluation Report: "As a rule, regressions using disaggregated data (such as observations on individuals as opposed to, say, observations on aggregates such as the average employment rate for the labour force) will produce lower R-squared values, so the value of 0.20 is not surprising and compares favourably with other regressions using similar data²⁶".

While these regression results are strong and explain significant variability in post-program employment, their interpretation is complicated by the presence of selection bias.

Table 14 below summarises the estimated effects of participation for the three interventions for each gender. Results are provided in terms of percentage time employed and the estimated increase (or decrease) in number of weeks worked per year. Note that the estimated effects of participation for all three interventions are statistically significant for males. For females the small estimated benefit for TWS is not significant and the small estimated negative effect for JCP is significant at the 5% level but not at the 1% level.

Effect of Job Creation Partnerships

Males who participated in a JCP project were employed after the program an estimated 12.1% more of the time than their previous work history and other characteristics would predict according to our models. A likely interpretation is that participation does significantly increase the likelihood that males will work more after the program — at least in the short term. However, as noted earlier, it is possible that some of this effect is due to selection bias or otherwise attributable to unmeasured characteristics of participants rather than the intervention itself. Correspondingly, females who participated were employed an estimated 9.0% less of the time after their program than the model would predict. A possible explanation is that at the end of a JCP project, many individuals are unemployed²⁹, and that female participants surveyed had not yet recovered from this initial unemployment³⁰. Another possible explanation could be differences (on average) in the projects which males and females have participated in. An important question for the summative evaluation will be to determine if this apparent negative effect persists and whether the gap between males and females remains.

In the key informant interviews, it was noted that improving short-term post-program employment of participants has not been a major focus for JCP. While JCP projects are expected to provide participants with skills and experience that may help their future employment, two other important goals were also noted³¹:

• Projects which offer economic development advantages may be approved at least partially on that basis. For example, one of the case studies was the Stephenville Theatre Festival. Part of the rationale for funding this project is for skill development of individuals. However, the rationale also includes the benefits to other industries of tourists attending the festival as well as the longer term potential for growth in cultural industries; and,
• Providing short-term income to participants who otherwise have limited opportunities for income. Particularly in small communities with limited (or seasonal) employment opportunities, participants may be largely motivated by short-term income considerations.

Effect of Targeted Wage Subsidies

Participants in the Targeted Wage Subsidies employment benefit have been employed significantly more in the post-program period than the models suggest based on their past labour market success and their other characteristics. This is especially the case for males who were employed 26.1% more of the post-program period than would have been predicted if they had not participated. The apparent benefit for female participants is much smaller at 5.6% of the post-program period. This estimated benefit for females is not statistically significantly different from zero.

Wage subsidy programs typically provide, at least, short term employment benefits since it is not uncommon for employers to retain some participants after the subsidy ends. Also some participants move immediately to a new job with another organization which they dealt with in their project employment. As noted in Table 7, 59% of TWS participants surveyed had a job immediately upon the end of the project. The important evaluation question — which can be addressed via the summative evaluation - is whether these short-term benefits persist.

Effect of Self-Employment Benefit

The Self-Employment Benefit intervention has an apparent strong positive effect on subsequent employment for both men and women:

• Men participating in SEB were employed an estimated 35.5 percent of the time more than if they had not participated. This amounts to an estimated 18.5 weeks of additional employment on an annual basis; and,
• Women participating in SEB were employed an estimated 38.8 percent of the time more than if they had not participated — an additional 20.2 weeks of employment on an annual basis.

Past evaluations of similar programs in Canada and other countries, have demonstrated that the positive benefits of such programs come from the survival of some businesses after the program and above-average employment success of those individuals who elect to close their businesses.

While the above estimates of program impact are extremely positive they are largely due to short term survival of the established businesses (75% of participants continued to operate their businesses at the time of their survey interview — 86% of those operated their businesses on a year round basis). Important questions for the summative evaluation will be the longer-term survival of those businesses and the employment success of individuals who elect to close their business.

3.2 SAR and Other Reachback clients

Reachback clients have mostly participated in JCP. Their post-program employment has not differed substantially, on average, from other JCP participants.

Employment Benefits are available to active EI claimants. They are also available to "reachback clients". Reachback clients are defined as those who do not have an active EI claim but who:

• had an active claim within the past three years; or,
• had an active claim or established a benefit period for parental or maternity benefits or sick benefits within the past 5 years.

Within the models, an indicator variable for reachback status was included to test whether the results varied for clients in these groups. For both males and females, the coefficient of this variable was not statistically significant indicating that no such effect exists.

The raw data in Table 7 indicates a somewhat more complicated picture. In terms of percent employed at the time of the survey (40% for reachback versus 38% for all JCP) and the average percent time employed since the program (47% for reachback versus 46% for all JCP) there seems to be very little difference between the reachback clients and other clients. However, reachback clients are more likely to have not worked at all (34% versus 23% of all reachback clients) and quite unlikely to have had limited work (3% were employed 1 to 24% of the time versus 9% of all JCP clients).

Recent receipt of social assistance is a more useful predictor of post-program employment success than reachback status. As can be seen in Table 13, for males, recent receipt of Social Assistance has an estimated reduction of 12.5 percentage points in the percentage time employed after the program. The regression model also estimates a negative effect for females although this estimate is not statistically significantly different from 0.

3.3 Effects of Interventions Pre- and Post-LMDA

As noted in the description of the methodology, the survey of participants concentrated mostly on participants since the Labour Market Development Agreement became effective but did include some participants in the time period between the effective date of the new EI legislation and the LMDA. In particular, of 1,493 participants interviewed, 448 started their intervention between July 1, 1996 and March 31, 1997 while 1,045 started after the latter date. The reason for including the first group was to test the hypothesis that effect of the interventions varied in the time periods.

4.1 Information management

Electronic information for the EBSMs is maintained in two separate systems. Within the HRCCs, the staff use NESS to enter information on action plans, set up contracts and track client achievement of action plan goals. Outside of the HRCCs the counsellors at third party delivery sites use CATS to perform substantially the same function. The CATS data contains information downloaded periodically from the NESS system, so that counsellors at these third party sites have access to client history similar to that available in the HRCCs.

Data from both NESS and CATS are used in the construction of the results data set, which uses a combination of EI data and the action plan information and follow-up to determine which clients have successfully returned to work, and to calculate any resulting EI savings. The results data set is then processed by the regional HQ to produce regional level reports on achievement of targets.

The current system provides a reasonable basis for tracking clients who are participants in EBSMs. In combination with EI data files a good client contact list can be produced. However there are two serious deficiencies in the current data management system.

• Successes (clients returned to work) recorded in the data do not correspond with actual client experiences as measured via the follow-up survey conducted in this evaluation. The calculation of success using EI information (clients who return to work within their EI period) is substantially more accurate than the calculations that have to rely solely on information entered by counsellors.
• Many clients who have TWS placements are not recorded in the data. This occurs since HRDC makes payments to the employers; not directly to clients so no client contract has to be set up. This problem is most prevalent with employers who have multiple placements. For the province as a whole, data is available for 46% of TWS participants. In the Avalon District data is available for only 32% of TWS participants.

Key informants indicated that both of these issues are attributable to resource shortages. In particular, full-time staff in many offices have not had the time to collect and enter the data.

4.2 Targeting and participation

There are several goals that Employment Benefits are intended to achieve, and the optimum strategy for meeting one goal may not be the best way of achieving another. For example, the desire to reach Accountability Framework targets for "clients returned to work" may encourage creaming; i.e. selection of participants who are most likely to succeed with or without the intervention and do not actually need the assistance provided by HRDC programming. Conversely, the desire to help those most in need may adversely affect overall success rates as measured by the Accountability Framework.

A second example relates to the contrasting implications of economic development goals and providing individuals with skills and experience. Projects that target economic development may do little to develop the skills of individual participants and vice versa.

This section looks at some of the issues that arise out of these targeting conflicts.

4.2.1 Targeting of individuals, groups and/or business sectors

Key informant interviews indicate that targeting strategy varies by intervention.

Self-Employment Benefit

Applications for self-employment are assessed based on merit rather than on priority sector match. Individuals who meet eligibility criteria, have a credible business plan (including market research) and who are not expected to harm existing local businesses are accepted into the program (subject to funding availability).

Targeted Wage Subsidies

HRDC personnel typically review applications for a TWS involving a single participant and a decision is made based on the merit of the application and funding availability. TWS applications involving multiple participants are reviewed by local assessment committees as per JCP projects (see below) in some jurisdictions (e.g. Stephenville) but are processed solely by HRDC in most locations.

Job Creation Partnerships

Local Project Assessment Committees review applications for JCP projects. Membership of these committees includes:

• local HRDC manager;
• local HRE manager;
• local Department of Development and Rural Renewal (DDRR) representative;
• Regional Economic Development Board (REDB) representative;
• local Community Business Development Corporations (CBDCs);
• an ACOA representative (if available); and,
• a College of the North Atlantic representative (if available).

HRDC program officers bring applications to this committee for consideration. HRDC retains signing authority for individual project approvals but has moved to a highly consultative process with its partners under the Labour Market Development Agreement. This evaluation examined how these committees function in four areas (Stephenville, Gander, St. John's and Happy Valley-Goose Bay).

In Stephenville, the local committee has developed as follows:

• committee meetings are held monthly;
• all parties noted above are represented and active; and,
• decisions on project applications are made via a consensus model based on economic plans and priorities for the region.
• decisions on project applications are considered jointly by all relevant government organizations. This is seen as assuring that proponents make a direct investment.
• projects without potential for economic development would not be approved in Stephenville even if they offered other benefits.

In the other three locations, the last two points do not apply. The interviews with HRDC personnel in Gander indicated a more flexible set of priorities is in place in that area. While economic development is an important criterion for project assessment, projects that provide only income and/or skill development may also be considered especially in communities where unemployment is very high.

In St. John's, we were informed by HRDC personnel that — like in Stephenville — the Project Assessment Committee operates using a consensus model and places significant emphasis on economic development due to the involvement of DDRR and the REDB. An increasing focus on the needs of Social Assistance Recipients was also noted. Projects that focussed on SARs as well as initiatives to improve the access of SARs to Labour Market Information (LMI) were noted.

In Happy Valley — Goose Bay, geography has presented some unique challenges. In particular the Happy Valley - Goose Bay Project Assessment Committee considers applications from coastal Labrador, which have been recommended by one of four sub-committees, which involve local officials in smaller communities. These local sub-committees are still in a development phase.

The survey of front line staff also provides evidence of a focus on economic development. In total, 87 percent of HRCC staff and staff of third party providers gave examples of EBSM projects that have been tied into local economic development initiatives, indicating that sectors which fit into local economic development plans are being targeted. Some also mentioned that a positive aspect of change with LMDA has been more of a focus on local economic development.

According to multiple lines of evidence, HRDCs spending on Employment Benefits is increasingly focused on economic development

As noted above, the extent of this increase has varied based on intervention type and locality.

Key informants at the regional level noted that the industry sector targets in the regional business plans were set at levels which:

• necessitated some greater focus on sectors deemed to offer the best opportunities; but,
• left sufficient flexibility to allow accommodation of applications that offered economic development opportunities in sectors which had not been initially targeted or which offered advantages for eligible clients.

In the survey of front line delivery staff, 28 percent of HRCC staff and 32 percent of third party provider staff indicated that they think the identification of priority sectors has meant that some candidates who would benefit from EBSMs have not had access to them. Key informants, however, indicated that the priority sector targets were low enough so that applications of merit from outside these sectors could typically be funded.

The identification of SAR reachback targets has only affected the participation of SAR clients in HRDC Employment Benefits in a limited way to date.

Staff at both HRCCs and third party providers commonly commented that the LMDA had broadened their client base with the inclusion of SARs, but other evidence suggests that by the end of 1998-99 progress in this direction was quite limited.

Interviews with key informants from HRE and HRDC conducted in the fall of 1999 established that effective steps to truly broaden the client base to include SARs were anticipated but had not yet occurred. Two primary reasons were cited:

• SAR reachback targets in 1998-99 were low enough that they were easily achieved without special effort;
• HRE has undergone significant restructuring. This restructuring includes staffing actions to increase the department's ability to increase its focus on employment of SARs. This restructuring was not fully complete in 1998-99 and the department had only limited ability to effect this new focus during the time period addressed by this evaluation. In particular, HRE did not have the resources to market the HRDC programming to its clients, nor to properly assess and counsel applicants.

HRE and HRDC as co-chairs of the LMDA have recognized and are addressing the first two points. In regard to the first point, SAR reachback targets have been doubled for 1999/2000. On the second point, development of HRE's capacity is proceeding according to its strategic priorities. As well, how to best identify and case manage SAR clients has been considered at provincial, district and local management committees. In the fall of 1999, HRE personnel were receiving training in Service Needs Determination to better equip them to identify appropriate candidates for referral to HRDC. HRE has also acted to reduce financial disincentives by extending drug cards for families with dependents. Nevertheless, based on key informant interviews and the survey of delivery agents conducted late in fiscal 1998-99 the policy issues presented by "shared clients" were not fully resolved at that time. The most common issue raised by counsellors at both HRCCs and third parties is that there are financial disincentives to SARs participating in employment benefit programs. SARs with dependants (who need to arrange child care) and single SARs with health problems in particular may end up to be worse off financially if they give up social assistance to take an employment benefit program.

As part of the survey of front line delivery staff, respondents were asked whether they have found that SAR reachback clients have different needs than other clients. Overall, 65 percent of respondents indicate yes and another 16 percent say "yes and no". These latter respondents mainly want to distinguish between two types of social assistance recipients. The longer term SARs have additional needs compared with other clients, but short-term, usually first-time SARs do not. Delivery staff are split on the extent to which SAR needs are met with 36 percent of respondents answering that the needs of SARs are being met fully, 41 percent that they are being met somewhat and 22 percent that they are not being met to any great extent. Those at third party providers were more likely to indicate that the needs of SARs are not being met, with 29 percent giving this answer compared to 13 percent of HRCC staff. Similarly HRCC staff were more likely to indicate that they are being met fully with 44 percent giving this answer compared to 31 percent of third party staff.

Counsellors also discussed how SARs have more self-esteem issues, emotional problems and other barriers that require individual counselling and often family counselling. Here HRCC and third party counsellors had a different view on how to deal with these sorts of issues. HRCC counsellors want the provincial Department of Human Resources and Employment (HRE) to continue to play a role with these clients, particularly when there are family related issues. Counsellors at third parties generally feel more able to help these clients, but worry that with increasing workloads they will not be able to continue to do this in the future. Counsellors at both HRCC offices and Outreach Offices³² also worry that they do not have the training to assess the needs of SARs. This was not raised as a concern with other types of third party providers. Both types of counsellors raised a lack of literacy programs as an issue when dealing with SARs. Many also brought up a lack of local economic development initiatives and suggested when there are initiatives there should be some JCP-type positions or quotas for SARs to give them a bit of an edge.

Suggestions for improvement made by counsellors include:

• That HRE continues supports and makes sure that clients do not become worse off by participating in an employment benefit program. Clear guidelines and rules needs to be established so that SARs can be confident about what support they will receive.
• Better co-ordination between HRE and HRDC.
• More promotion of HRDC programs and services to SARs who are eligible to avail of these programs and services.
• More literacy programs.

Some suggested that guarantees could be made to SARs who take training such as if they successfully complete the training they will be guaranteed that funding would be available for a targeted wage subsidy for the year after their training. To avail of this opportunity, it would be necessary to recruit a willing and suitable employer.

4.2.2 Are program participants representative of the unemployed labour force?

Income tax data was used to examine differences between EBSM participants and the Newfoundland population of EI recipients³³. The major differences are that:

• The EBSM participants in general (except SEB clients) have lower income levels than the general EI population. The average 1995 incomes (from all sources) for the four groups are:
  • Main EI population $19,850
  • TWS participants $14,550
  • SEB participants $22,500
  • JCP participants $12,900
• EI benefits received by participants was (on average) higher than the EI population ($4,400 per year, compared to $3,900)³⁴

4.2.3 Is the selection of clients affected by the accountability framework?

Respondents to the survey of front line delivery staff noted that the accountability framework has led to an emphasis on helping clients who can be more easily placed in an employment benefit program. These clients are helped as opposed to those who require in-depth counselling or who have greater need such as those with disabilities or those who require literacy training. The accountability framework means that counsellors are primarily judged on short-term success. Many third party counsellors also expressed frustration that the accountability framework does not look at the starting point of clients.

4.2.4 Achievement of targets

Each of the four HRDC districts (Avalon, Central, Western and Labrador) has a target defined for number of clients served, clients returned to work and EI savings. These targets are specified in the Labour Market Development Agreement and are defined for the 1998/99 fiscal year.

Clients Served

Table 15 shows the district targets for clients served for fiscal 1998/99. These targets were well beyond activity in fiscal 1997/98 (the first year of the LMDA) since activity was curtailed in 1997/98 by multi-year commitments to training clients in the previous fiscal year.

Return To Work Targets

Table 16 identifies the overall (including training) success targets for returns to work. It was not possible to validate these numbers since the data available was only for participants in JCP, TWS and SEB whose interventions started between July 1, 1996 and December 31, 1998 and whose interventions had ended by February, 1998.

Table 17, however, examines the HRDC results data for those individuals surveyed and compares the HRDC definition of successes with the survey estimates.

Using the survey data on jobs after the intervention, for TWS and JCP, anyone who started a job within 3 months of the end of the intervention or who was still working for the same employer was counted as employed. For SEB clients anyone who was still running his/her own business or who got a job within three months of completing the program was counted as a success. As indicated in Table 17, the survey results indicate that the number of successes is greater than the number of successes recorded in the HRDC results database.

In addition to this apparent undercounting of successes, the two sources did not agree well. They agree in about 56 percent of cases, and disagree in 44 percent (identified as mismatches in Table 17).

Table 17 illustrates the dual nature of the problem for each of the three interventions. Results in Table 17 are based on all individuals surveyed (excluding the extra reachback clients surveyed).

In summary this table indicates that:

• Success rates are substantially underestimated by HRDC for all three interventions. For SEB, survey results indicate more than twice as many returns to work as indicated in HRDC's data. For JCP, success estimates from the survey data are 2.5 times higher than those of HRDC.
• Success rates from HRDC data as well as being low also involved incorrect classification of successes.

To illustrate this dual problem, Table 18 illustrates the detailed comparisons for Targeted Wage Subsidies participants. For 221 of the people the survey and the results data agree on the result, including 146 that both indicate were a success. For 152 people (41%), the two sources disagree, the majority (103) being people not counted as a success in the results file, but who are "successes" based on the survey data. However, there are an additional 49 individuals classified as successes in the results data base who did not return to work within three months of completing their subsidized employment.

EI Savings Targets

The business plans for each district also specify the EI savings targets that are expected from getting people successfully employed. These savings are counted only if the client becomes employed within the period in which they are entitled to collect EI. There are thus no savings attributed to getting reachback clients employed. As well, individuals who are active claimants when they participate but whose claim expires before returning to work do count as "return to work" successes but do not count as "savings to EI account" successes.

Based on Table 19, it is clear that the SEB, JCP and TWS interventions do not account for a significant share of EI savings targets.

Table 20 shows the breakdown of the source of successful clients in 1998/99. The information is based on matching the 98/99 successes with the HRI client data for 97/98 and 98/99.

Clients often receive several types of assistance from HRDC. Where a client had multiple interventions, the client is assigned to a single intervention in the same order as they are presented in the table. For example, if a client participated in TWS and also attended a counselling group, he/she would be assigned to TWS.

The table confirms the information in Table 19 that SEB, TWS and JCP account for a small proportion of the EI savings calculated by HRDC. Clearly training (both seat purchase³⁶ and enhanced feepayers) account for a large proportion of the estimated savings. The Other EBSM category is also large. The major component in this is Govt. to Govt.—RBA. Note that although clients have been counted as being in group services and counselling group sessions only if they participated in no other interventions, that these two items account for a substantial proportion of savings (almost double the combined contribution of SEB, JCP and TWS).

4.2.5 Validity of Accountability Framework Data

The variations between the Results Data and the survey data raise serious questions as to the accuracy of HRDC's data. We assume that one source of the missed successes are problems in case management. In particular, many of the missed "return to work" successes are likely due to:

• counsellors failing to reach individuals to determine their post-program employment situation (either initially or after three months or both); or,
• counsellors failing to enter successes into CATS or NESS due to heavy workloads.

The results file differentiates between four groups of clients who "return to work" as illustrated in Table 21.

The results file contains successes for any case-managed client and for people who attended group sessions. For the EBSM evaluation we are primarily interested in the three employment benefit measures (TWS, JCP and SEB). The results file shows successes for 1422 people (of 5567) who participated in an employment benefit after July 1996. The success rate using the data from this file are given in Table 22 along with the total numbers of interventions and reachback clients for each intervention.

There are two major apparent inaccuracies in Table 22. Firstly, the number of Unit 43 successes in TWS is impossibly high. This intervention is generally one year in duration, so in most cases the end-of-intervention action plan indicator of success would be entered after the client's claim ended.

The other inaccuracy is the large number of Unit 46 successes. For a Unit 46 success the following must be true:

• There is no client record in BNOP (i.e. the client has not received EI benefits), or they are in BNOP but the Action Plan or equivalent does not match the benefit period;
• The result is recorded in NESS (or CATS);
• Outcome dates or end dates fall within the fiscal year range.

The Unit 46 code is designed to capture reachback clients, i.e. those who received help even though they had no active EI claim. There are obviously other clients being captured in this category. Note for example that the results file classifies 250 successful TWS placements as Unit 46 successes even though the administrative data identifies only 3 TWS clients as reachback clients³⁷. The likely problem here is that even though a client really did start an intervention while they were on EI, some problem with the data resulted in the match not being made in all cases³⁸.

4.3 Co-management

Under the Canada-Newfoundland and Labrador Agreement on Labour Market Development (LMDA) the two levels of government agreed on the following:

• the federal government continues to be responsible for the delivery of Employment Benefits and Support Measures (EBSMs) and the National Employment Service; and,
• both levels of government are jointly responsible for the design, management and evaluation of EBSMs and some elements of or certain functions of the National Employment Service.

The governing structures stipulated in the LMDA were as follows:

• a provincial Management Committee with equal numbers of representatives from each level of government was struck to manage the delivery of EBSMs at the provincial level. The Committee is co-chaired by the Regional Executive Head of HRDC and the Deputy Minister of Human Resources and Employment (HRE). The Deputy Ministers of the Department of Development and Rural Renewal (DDRR) and the Department of Education (DOE) are also on the Committee.
• four District Committees for Labour Market Development were established for the Avalon, Central, Western and Labrador Districts. Each Committee is co-chaired by the appropriate district managers of HRDC and HRE. The specific responsibilities of these committees are not spelled out in the LMDA and are left to the discretion of the provincial Management Committee. The LMDA does include a commitment to business planning at the District level and in practice these plans have been drafted by the District Committees.

The LMDA stipulates that decisions of all five committees are to be made by consensus and if consensus cannot be reached than decisions move up to a higher level.

The LMDA also includes a commitment to review the design of the EBSMs covered by this evaluation as well as to develop a new training benefit.

The LMDA does not stipulate how co-management is to be implemented at local levels. In practice, local Project Assessment Committees have been instituted which examine applications for Job Creation Partnership Projects, Local Labour Market Partnership and in some cases, larger Targeted Wage Subsidies.

In general, it is our understanding that local HRDC managers retain responsibility for management of local HRDC operations and that the local Project Assessment Committees are generally not involved in other aspects of HRDC operations³⁹.

Front line staff at HRDC and third party delivery organizations were asked how the introduction of federal-provincial co-management of EBSMs has changed their day to day work. About a third (30 percent) of respondents with three or more years of work experience indicate that it has not changed their day to day work. Respondents at third party organisations were more likely to indicate that it has not had an influence, with 37 percent indicating there has been no change compared with 20 percent of HRCC staff.

The evaluation included four case studies of co-management (Stephenville, St. John's, Gander and Happy Valley — Goose Bay). Based on interviews with representatives on the Project Assessment Committee, HRDC personnel, and the co-chairs of each of the four District Management Committees, these local committees are functioning effectively without any serious complications. In Happy Valley — Goose Bay, however, processes for considering projects from coastal Labrador are not yet fully effective.

4.3.1 Advantages of co-management

Key informants saw co-management as an effective approach. In particular, they noted that review of projects is much improved under co-management. In the past, they noted:

• projects would sometimes receive independent support from many branches/departments of government as well as from third party organizations funded by government. Since there was no overall assessment, each aspect of support might be sensible in isolation but the sum total of support might be difficult to defend; and,
• projects which offered short term advantages in a particular community might be supported when better opportunities were available.

Under co-management, all parties can consider a project together which offers the potential to eliminate these difficulties. In particular, by having the various government organizations working together, it is much easier to:

• ensure that the sponsor benefits from the full range of non-financial assistance offered by all of the government organizations;
• ensure that proponents or sponsors are truly making their own contribution and/or have the support of conventional lenders; and,
• select the best available projects since all parties are aware of available project proposals in the same time frame.

In Stephenville, all parties were convinced that these potential advantages are currently being realized. In the other three locations, the full potential advantages from working together has yet to be entirely realized in the view of those interviewed. All expected continuing progress.

Key informants also noted that the co-management of EBSMs has also led to several other advantages in other areas of programming:

• similar co-operation has occurred in related areas of programming — e.g. Transitional Job Fund, summer employment programming;
• improved co-ordination and communication within HRE and DDRR;
• improved co-operation among Regional Economic Development Boards; and,
• provides relevant learning for the province's Strategic Social Plan.

In terms of positive aspects of the change, both HRCC and third party staff most commonly mentioned that duplication of services is being avoided and that there is more communication and co-operation between the province and HRDC. Respondents discussed how both levels of government now promote each others services more and are generally more aware of what each other has to offer. That they could now help a broader client base was also commonly mentioned by both kinds of staff.

Third party staff mentioned several positive aspects of co-management that HRDC staff did not. Some have found that since co-management they have been getting more support from HRDC and that they can do more for specific client groups, including SARs and disabled clients.

4.3.2 Disadvantages of co-management

Key informants saw few disadvantages to co-management. Those difficulties which were identified were considered surmountable or to have compensating advantages.

For example, the delays in approval of Regional Business Plans for 1999/2000 was lamented and noted as a barrier to best implementation. However, the perceived solution was commitment of all parties to a timely approval process rather than changes to the co-management process. Also the possible delays in review of projects by the Project Assessment Committees was noted by some. However, this concern did not exist in Stephenville where the PAC functions smoothly and is perceived by all parties as an improvement on the previous process.

The biggest disadvantage of co-management identified by key informants from HRDC is that co-management is perceived by many as a temporary situation and this perception has caused serious morale problems for HRDC personnel. In particular, most HRDC personnel believe that either the province will decide to take full responsibility for employment programming under a devolution approach as has been applied in most other provinces or that Ottawa will pressure the province to make such a decision.

Respondents to the staff survey tended to give more examples of negative changes than positive ones. HRCC and third party staff gave fairly different responses. The most commonly mentioned negative change by HRCC staff is that making decisions has become much more time consuming because they have to consult and co-ordinate with so many outside agencies. Some expressed frustration that they can no longer make decisions and that they do not have the authority anymore to make decisions about they type of supports that would best help their clients.

Some HRCC staff members have also found that under co-management they see a shift in focus as the year goes by. In the first quarter they are encouraged to try to help clients find sustainable employment by whatever means is best and by the last quarter they are just trying to get clients in programs so they can spend the budget.

Key informants noted a particular problem in 1998/1999 when substantial additional funding became available late in the year when the Pan-Provincial Budget was not fully spent. Several informants with both HRDC and HRE noted that this damaged HRDC's credibility with its partners as well as undermining the efforts of all parties to communicate that HRDC support is now limited to projects/initiatives providing the potential for sustainable employment.

Both HRCC and third party staff noted an increased workload with no additional staff, and in some instances reduced staff, as a problem under co-management. This, together with having to spend more time on data entry to ensure accountability framework measures are tracked, is, according to many of those interviewed, causing a situation where counsellors cannot always get to know a client well enough to know what is best.

4.4 Case Management and Client Counselling

4.4.1 Is a tracking system in place?

All front line staff surveyed use either CATS or NESS (or both) to track clients. Some counsellors at third party providers commented that while they use CATS to supply the necessary information to HRDC to track for accountability framework purposes, they do their actual case management from their paper files.

As Table 23 shows, HRCC counsellors are more positive about the effectiveness of their current method of case management than are counsellors at third party providers.

4.4.2 Participant satisfaction

Participants were asked a series of questions about their satisfaction with the services they received and with their job/placement in the Targeted Wage Subsidies or the Job Creation Partnership programs. As Table 24 shows, there is a high level of satisfaction with the level of service received at the HRCC or Outreach Office. The satisfaction level in Labrador is somewhat lower than in other regions. Satisfaction with the variety of services offered also varies by region, with participants in the Avalon and Central regions being more satisfied than those in the Western Region and Labrador.

Satisfaction with the actual work placement is lower, in particular in terms of the work being related to career objectives and being useful in helping participants to find a job. Provincially, 47 percent of respondents were satisfied with how closely work was related to their career objectives. However, TWS participants were more likely to indicate satisfaction with career-relatedness with 58 percent indicating high satisfaction compared to 45 percent of JCP participants.

Those who are no longer working for their project employer were asked how useful the program has been in helping them find a job. Only 28 percent have found the experience to be useful. However, 37 percent of TWS participants have found the program very useful in helping them to find a job compared with 27 percent of JCP participants.

4.4.3 Participant involvement

Counsellors were asked about what percentage of EBSM participants develop action plans. As Table 25 shows, the clients of third parties are less likely to develop action plans than the clients of HRCCs.

When asked to rate the extent to which the development of action plans benefits clients, 57 percent of HRCC staff rate the benefit 9 or above on the 0 to 10 scale (0 not at all and 10 very much so) and 40 percent of third party provider staff rate the extent of benefit a 9 or above on the same scale.

When participants themselves were asked "with the help of an employment counsellor at an HRCC, did you develop an action plan for the future?" only 12 percent indicated yes. The percentage was the highest in Central where 16 percent of participants said they had developed an action plan and was lowest in Western where only 6 percent said yes. These low percentages show that clients themselves are not necessarily aware of the action plan process and therefore are likely not as involved in this process as they might be. Based on these statistics, the primary benefit of Action Plans to clients appears to be that they are required to access EBSMs.

4.5 Labour Market Information

Both the survey of front line staff and the participant survey asked questions about the availability and use of labour market information.

4.5.1 Availability of Labour Market Information to front line staff

In the survey of Delivery Staff, 52 percent of HRCC staff and 67 percent of third party staff indicate that they have the labour market information resources and tools that they need to be able to help clients. When counsellors were asked what was missing a range of responses was given. Those at HRCCs tended to give more specific answers than those at third party providers.

HRCC staff mentioned they were missing (in order of frequency of mention):

• An LMIA (Labour Market Information Analyst) in their office (mentioned by far the most frequently). Some at offices with part time LMIA indicate that they need a full time LMIA;
• Updated information provided regularly;
• Regular updates from LMIA — these should be distributed automatically on a regular basis. Now they have to go looking for information;
• Information on the local labour market;
• Tools such as Choices to help clients make career choices;
• Regularly updated success rates for graduates of specific programs;
• More time to use resources;
• Internet access for clients;
• A Career Information Resource Centre (CIRC) in the local area.

Staff at third party providers indicated that they were missing (in order of frequency of mention):

• More local information;
• Updated information provided regularly;
• Information about what LMI sources are available;
• Internet access for staff;
• Need more specific LMI — what is available is too general;
• Need internet access for clients;
• An LMIA at HRCC;
• A full time LMIA at HRCC rather than part time that they have now;
• Need more user-friendly LMI/ LMI in plain simple language that clients can understand;
• Need access to updated LMI from HRCC/regularly updated information;
• Would like to have Choices Program;
• More printed materials;
• Regular workshops;
• More provincial information;
• Need to have information in schools.

Most respondents indicated that they would benefit from additional training on labour market information. In particular, 83 percent of those at HRCCs and 93 percent of those at third party providers indicate that they would benefit from LMI training.

Staff members at HRCCs are interested in the following training opportunities (in order of frequency of mention):

• Training on accessing specific information on the internet/ how to use internet tools;
• Anything would help/general training/ how to find out what is out there;
• How to do LMI research more efficiently;
• Training on how to get and keep up to date local LMI;
• All they need is an LMIA then training would not be necessary for everyone else;
• Just periodic presentations to update them — not actual training;
• How to find specific LMI;
• Job prospects for main occupations/growing sectors;
• Training on new Departmental Pay System;
• Knowing what LMIA role is and what is the process to get information;
• Training on how to advise people to seek out LMI on their own;
• Training on how to follow LMI trends.

A few people in HRCCs commented that the same people get all the training and in particular that those in acting positions never get to take any training. Many commented that time was the big issue for LMI. They just don't have the time to stay up to date with all of the other pressures they have.

Staff members at third parties are interested in:

• Quarterly workshops to help keep up to date/Sessions to keep up to date;
• Training on best ways to keep up to date;
• Anything would help/ General LMI workshops;
• Internet training;
• Training on LMI trends/ growth areas/analysis;
• Training on how to interpret results (for example what jobs fall into what categories);
• How to access LMI;
• A day orientation at the CIRC/on labour exchange/kiosk;
• Sessions with industry to let them know what various industries need/ Session on offshore — up and coming jobs in the offshore;
• Training on local LMI sources;
• Training on how to access information on specific jobs;
• Getting together with HRCC to talk and share information would help;
• Training on how to use LMI to help clients/ on what tools are available (Choices software, etc.).

Some third party staff at outreach offices commented that they should have access to updates and training provided to HRCC staff.

Interestingly counsellors at third party providers are more likely to indicate that they have had useful LMI training than have those at HRCCs with 57 percent of third party staff indicating that they have received useful LMI training compared to 20 percent of HRCC staff.

4.5.2 Client satisfaction with the availability of Labour Market Information

As Table 26 shows, participants are generally satisfied with the LMI available to them at local offices. A higher percentage of clients are satisfied with information available on the local labour market than they are on the provincial or national job market.

5.1 Consistency with EI legislation, provincial and federal government priorities, HRDC national and regional priorities

The delivery of the EBSMs covered by this evaluation (JCP, TWS and SEB) has been done in a fashion that is entirely consistent with national guidelines. Indeed, while the LMDA stipulates that the Management Committee would initiate reviews of the appropriateness of the EBSMs and make modifications to enhance their responsiveness to local terms and conditions, no such reviews have occurred. Consequently, JCP, TWS and SEB continue to be delivered as they were prior to signing of the LMDA⁴⁰ and as laid out in the national guidelines⁴¹.

Key informants (both federal and provincial) considered these Employment Benefits to be appropriate to the province's circumstances and did not perceive a need for significant revisions. It was generally perceived that the tools offered sufficient flexibility to allow their appropriate use in all locations.

The perspectives offered by key informants for the three measures were as follows:

• Several informants noted that TWS and SEB were excellent and proven tools that were currently under-utilized in the province. In particular, severely depressed local economies in many areas of the province limit the use of these employment benefits, which are designed to respond to emerging opportunities. Several informants noted a need to improve the marketing of these tools so that their use is maximized.
• JCP was noted by informants and by some sponsors (in the case studies) as an improvement on previous "project" interventions. In particular, the implementation of this tool in the co-managed environment was seen as much better than make-work projects of the past for three reasons:
  • focus on economic development. The review of project applications focuses to a greater extent on economic development opportunities and infrastructure development. While these projects may not offer any short term advantages to participants, development of local economies — if successful — offers advantages to those communities and their members;
  • combating the cycle of EI dependency. Informants noted that since JCP earnings are non-insurable and may be minimal (depending on the individual's benefit rate), the cycle of being on a project every year was starting to break because of the decreasing appeal of projects⁴². One sponsor from an organization that had sponsored HRDC projects for several years noted that two current participants had requested time off for a job interview. In previous projects participants had not been seeking work since they were content to qualify for benefits as a result of their project employment. On another project, 4 individuals had to be replaced when they received employment.
  • sponsor contribution. Informants noted that the requirement for a sponsor contribution was a powerful disincentive for "make-work" projects. This was especially the case since in a co-managed environment, it was not possible for a sponsor to obtain funding from another branch of government as their "contribution".

Nevertheless, some informants remained sceptical about the value of Job Creation Projects for either economic development or for improving employment opportunities for individuals.

5.2 Directed toward labour market opportunities and economic development

Key informants also indicated an increased focus on labour market opportunities through Job Creation Partnership Projects and Local Labour Market Partnerships (LLMPs). Many of these investments are seen as long term rather than short term. As well, in communities where opportunities for future economic development have not yet been identified, projects that maintain or improve existing physical infrastructure are being supported to maintain a base on which to build in the future.

Respondents to the survey of front line staff were asked to rate their agreement with the statement "Employment programs are delivered in ways that are more appropriate to your local labour market than prior to LMDA". More than half agreed. In particular, 52 percent of staff at HRCCs and 60 percent of third party staff agreed with this statement. Many respondents commented that even more of a focus was needed on local economic development because local labour markets are so poor.

Front line delivery staff were asked whether they were aware of EBSM projects that were tied into local economic development. Most (87 percent) respondents were able to give examples of initiatives that have been tied in with local economic development.

5.3 Matched to individual needs of unemployed

Front line staff interviewed indicated that there are gaps in meeting the needs of unemployed individuals. When asked their agreement with the statement "The mix of programs and supports under LMDA meets the needs of unemployed individuals", only 40 percent of HRCC counsellors and 30 percent of those at third parties agreed. Further, when counsellors were asked about gaps in the current EBSMs in terms of their ability to meet the needs of unemployed Newfoundlanders, almost all mentioned gaps. By far the most commonly mentioned gap is that now they cannot help people who are not EI eligible and do not have reachback status. Some groups were commonly singled out as having a particular need including youth, parents returning to work, injured workers who want to retrain and the disabled. Counsellors also mentioned that they cannot do much for displaced workers with low education levels and that there is not enough money in the opportunity fund to do as much as is necessary for disabled people. Counsellors raised here again the issue of there being financial disincentives for SARs to take training. Finally some counsellors mentioned that they could no longer help people in dead-end jobs, particularly those in low paying part time jobs.

When asked what should be done to fill these gaps most suggest that there should be separate funding for non-EI eligible clients and, at a minimum, support measures such as counselling and job finding clubs should be open to everyone. Others indicate that the reachback time period should be extended. More youth programming, more ABE and literacy programs and more money for the opportunity fund were also suggested. Removing financial disincentives for SARs was also raised as a priority.

Key informant interviews approached this issue from a different perspective. Issues of eligibility were not discussed in any depth with key informants. Instead key informants were asked whether the increased focus on economic development and strategic economic sectors under co-management meant that there was a decreased focus on client needs. Key informants consistently noted that this was not a concern for two reasons:

• the focus on growing sectors of the economy provides individuals with the skills and experience that are expected to be in demand in future; and,
• the business plan targets for priority sectors have — at least to date — been set at moderate levels and do not impede the approval of strong applications for investments outside of these sectors.

5.4 Language of choice

Only one respondent to the participant survey indicated that the language he or she was most comfortable speaking was something other than English. This person's language of choice was French and service was provided in French.

The Conclusions from this formative evaluation are provided in this chapter in relation to the three categories of issues addressed:

• the continuing relevance of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement;
• the design and delivery of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement; and,
• the success of the Employment Benefits and Support Measures (EBSM's) provided under the Canada-Newfoundland and Labrador Labour Market Development Agreement.

6.1 Relevance

The relevance issues identified for the evaluation relate to the extent to which the EBSM's correspond to:

• the statutory requirements of the EI legislation;
• the priorities of federal and provincial governments;
• the needs of individuals; and,
• local economic development priorities.

Conformance to EI legislation

The EBSM's delivered under the Agreement correspond to the EI legislation and related policy directives of the Government of Canada. No significant changes have been made to either the design or the delivery of the EBSM's under the Canada-Newfoundland and Labrador Agreement on Labour Market Development.

Conformance to the Priorities of Federal and Provincial Government

The priority of the Government of Canada as regards EBSM's is to provide assistance to active EI claimants⁴³ to facilitate their return to employment. The Government of Newfoundland and Labrador accepts this priority and also seeks to address its own priorities:

• to ensure that reachback clients have the same degree of access to EBSM's as current EI claimants;
• to ensure that EBSM expenditures are made in a way as to be consistent with and complementary to the economic develop priorities and initiatives of the Government of Newfoundland and Labrador as well as those of Regional Economic Development Boards (REDB's) across the province; and,
• to enhance the capacity and ability of the Government of Newfoundland and Labrador to serve its citizens as regards employment programming and social and economic development.

Under the co-management approach, it is clear that each government has been able to pursue its respective priorities.

The Government of Canada's priority is dictated by the EI legislation which. Under the Agreement, HRDC has continued to be governed by this legislation.

The co-management process has enhanced the ability of the province to pursue its priorities in several ways which were noted by key informants:

• the business planning process includes targets (provincially and sub-provincially) for reachback clients and various industry sectors;
• provincial government representatives on local Project Assessment Committees examine the conformance of proposed projects to social and economic priorities of the area;
• staff of the provincial Department of Human Resources and Employment have received training in the Service Needs Determination (SND) Process which allows them to improve the referral of eligible SAR reachback clients to HRDC. It also improves their ability to assist other unemployed individuals who are not eligible for EBSM's.

Conformance to the Needs of Individuals

This evaluation has found short term post-project employment benefits for Targeted Wage Subsidies (TWS) clients and for Self Employment (SEB) clients which suggests that the needs of these clients have been addressed. This is also supported by the views of key informants and the high satisfaction of participants with the program and their experiences under the program.

Participants in Job Creation Projects (JCP) have not realized short-term employment gains after their project participation. Key informants indicated that such gains would not be expected. They saw the primary focus of JCP as being to support economic development initiatives as well as to provide short term income to participants. The providing of new skills and experience to participants was generally seen as a lower priority. It was beyond the scope of this evaluation to assess the effectiveness of JCP in stimulating economic development.

Conformance to Local Economic Development Priorities

The evidence from this evaluation suggests that local economic development priorities are an important consideration regarding EBSM expenditures. As noted, it was beyond the scope of the evaluation to assess the effectiveness with which these priorities have been pursued. However they clearly have been pursued:

• Business Plans (Provincial, District and Local) identify specific targets for priority sectors. Processes are in place to monitor the achievement of these targets and to report deviations from plans to the Provincial Management Committee and to District Management Committees.
• ACOA, the Department of Development and Rural Renewal and Regional Economic Development Boards participate in Provincial and District Management Committees. They also participate on local Project Assessment Committees.

6.2 Design and Delivery

Design and delivery issues for this evaluation address:

• the extent of co-ordination between the Government of Canada and the Government of Newfoundland and Labrador;
• the responsiveness to local conditions in the delivery of the EBSM's;
• the adequacy of administrative data for evaluation and monitoring; and,
• client characteristics and targeting.

While all these issues have been addressed, the Canada-Newfoundland and Labrador Agreement on Labour Market Development allows for significant innovation, flexibility and accountability at local and district levels. It was beyond the resources of this evaluation to examine the consequent variability in a detailed fashion.

Co-ordination Between the Federal and Provincial Government

Key informants consistently were positive about the extent and success of co-management under the Agreement. In particular, key informants noted:

• the value of the Business Planning process which has been applied at the district and provincial levels;
• the much improved knowledge that each level of government now has of the other's programs and priorities;
• the consensus decision making approach used by the provincial and district management committees and by some local Project Assessment Committees;
• the increasing integration of staff training across the two levels of government; and,
• the increased partnership between the two levels of government as regards programming not covered by the Agreement (e.g. Canada Jobs Fund; Fisheries Restructuring & Adjustment Measures (FRAM); and, summer employment programs).

Negatives noted regarding co-management and the resulting co-ordination included:

• delays in finalizing operational budgets attributed to the Business Planning process;
• need to re-allocate budgets to deal with under-commitment and/or over-commitment of available funds in some funding envelopes;
• initial difficulties faced by the Government of Newfoundland and Labrador as it needed to develop its capacity in order to fulfill its role under the Agreement; and,
• expectations that co-management is a temporary mechanism and will be replaced by devolution of employment programming to the Government of Newfoundland and Labrador in the medium term.

Responsiveness to Local Conditions

Key informants noted that local Project Assessment Committees have increased the range of input to decisions as to projects to be implemented in local areas.

Adequacy of Administrative Data

Electronic information for the EBSMs is maintained in two separate systems. Within the HRCCs, the staff use NESS to enter relevant information while outside of the HRCCs the counsellors at third party delivery sites use CATS. The CATS data contains information downloaded periodically from the NESS system, so that counsellors at these third party sites have access to client history similar to that available in the HRCCs.

Data from both NESS and CATS are used in the construction of the results data set, which uses a combination of EI data and the action plan information and follow-up to determine which clients have successfully returned to work, and to calculate any resulting EI savings. The results data set is then processed by the regional HQ to produce regional level reports on achievement of targets.

The current system provides a reasonable basis for tracking clients who are participants in EBSMs. In combination with EI data files a good client contact list can be produced. However there are two serious deficiencies in the current data management system.

• Successes (clients returned to work) recorded in the data do not correspond with actual client experiences as measured via the follow-up survey conducted in this evaluation. The calculation of success using EI information (clients who return to work within their EI period) is substantially more accurate than the calculations that have to rely solely on information entered by counsellors.
• Many clients who have TWS placements are not recorded in the data. This occurs since HRDC makes payments to the employers; not directly to clients so no client contract has to be set up. This problem is most prevalent with employers who have multiple placements. For the province as a whole, data is available for 46% of TWS participants. In the Avalon District data is available for only 32% of TWS participants.

Client Characteristics and Targeting

Under the Agreement, targeting provisions exist for priority sectors and for Social Assistance Recipients (SARs) who satisfy the eligibility provisions of the EI legislation. For the first two years of the Agreement, targets were set at relatively modest levels. These targets have guided the District Management Committees as well as local Project Assessment Committees and staff of HRDC and third party organizations contracted to provide services. However, key informants noted that the modest targets meant that major changes did not occur relative to the past. In particular, it was noted that project applications as well as participant applications were generally assessed based on their merit rather than on targeting provisions.

For the 1999/2000 fiscal year, targets for SARs were doubled relative to those for 1998/1999. Achieving these targets is likely to be a greater challenge and will necessitate co-ordination at the operational level between HRDC and the Department of Human Resources and Employment (HRE). Training has been provided to HRE staff across the province to allow them to fulfill their responsibility to identify and refer eligible SARs to HRDC.

6.3 Success

The success issues identified for the evaluation address:

• the short-term post-program employment experiences of participants;
• the effectiveness of case management;
• the satisfaction of participants and employer/sponsors;
• the extent to which the experiences of SARs differed from those of other participants; and,
• the extent to which targets for employment and savings to EI have been met.

Short-term Post-Program Employment

Based on 1,494 participants surveyed during this evaluation, participants have been employed on average 49% of the time since completing their intervention. This varies substantially by intervention type:

• SEB participants have been operating their business or employed for 89% of the time since HRDC funding support was terminated;
• TWS participants have been employed 64% of the time since their wage subsidy ended; and,
• JCP participants have been employed 46% of the time since the project ended.

Statistical regression models indicate that for SEB and TWS this post-program employment is higher than would otherwise be expected. The models employed adjusted for other measurable characteristics which are known to affect employment success (e.g. age, education, prior labour market experiences, etcetera) but do not adjust for selection bias. Separate models were employed for males and females. These models indicated the following:

• SEB participants have been in business or employed more than 35 percentage points more of the time than would be expected based on their other characteristics (36 percentage points for males and 39 for females).
• TWS male participants have also been employed 26 percentage points more than would be predicted from the models. The post-program employment for female TWS participants is estimated at 6 percentage points higher than would otherwise be expected. This result is not statistically significant.

JCP participants post-program experiences are closer to what would be predicted based on their other characteristics. Male JCP participants have been employed an estimated 12 percentage points more of the post-program period than the model predicts while females have been employed 9 percentage points less that the model predicts.

All the above data is short term in nature with most participants interviewed within a year of their program completion date. It is likely that this short-term orientation explains at least part of these results. In particular SEB participants are likely to continue in business after their subsidy ends and many TWS employers retain participants after termination. Consequently, participants from both interventions can be expected to do well in the short term. Conversely, almost all JCP participants will be unemployed at the end of their project and poor short-term post-project employment is expected.

Effectiveness of Case Management

The evaluation did not examine case management processes and their effectiveness in depth. Based on the evidence obtained, the following conclusions can be drawn:

• 64% of HRCC staff describe the current method of case management as very effective compared to only 32% of staff of third party providers;
• the "Action Plan" which participants must complete is not, in retrospect seen as a valuable process. Only 12% of participants surveyed, recalled preparing an action plan; and,
• participants were generally satisfied with the level of service they received from the HRCC or Outreach office with 80% rating their satisfaction at 8 or higher on a 0 to 10 scale.

Satisfaction of Participants and Employers

Participants were satisfied with HRDC staff and service but less so with the value of their project experiences:

• 80% were very satisfied⁴⁴ with the level of service received; and,
• 72% were very satisfied with the variety of services available.

but, only,

• 47% gave equivalent ratings for how close the work was related to their career objectives; and,
• 28% indicated the program was very useful in helping them find a job.

Employers and sponsors were highly satisfied with all aspects of the program:

• 76% were very satisfied with the participant's skills;
• 89% were very satisfied with HRDC administration; and,
• 81% rated the value of the participant's work very high.

Success of Social Assistance Recipients

The regression models of percentage time employed after the program indicate that SAR participants have somewhat less success in the post-program period than individuals who were otherwise similar. However, these effects were small (13 percentage points for males and 7 percentage points for females).

The evaluation compared the experiences of "reachback" participants to those of other participants in order to address this question. Late in the evaluation, however, the validity of the "reachback" indicator in HRDC's administrative data was determined to be questionable.

Employment and EI Savings Targets

HRDC administrative data indicates that employment and EI savings targets have been met. The evaluation indicates that HRDC's estimate of "returns to employment" is low based on data from participants surveyed. TWS, SEB and JCP do not account for a large share of EI savings attributed in HRDC's administrative data.

Footnotes

1	Defined as those with a current active claim as well as those who satisfy the reachback provisions of the EI legislation.
2	8 or higher on a 0 to 10 scale.
3	Evaluation of the Self-Employment Benefit Program in Western Newfoundland, Goss Gilroy Inc., April 1999.
4	The 227 interviews were obtained from 496 reachback clients who had not been selected in the sample for Cohort 2. There were also 110 reachback clients who were interviewed as part of Cohort 2 for a total of 337 interviews with reachback clients. Reachback clients may or may not be in receipt of Social Assistance. Of the 337 reachback clients interviewed in the survey, 51 had received Social Assistance (or their spouse had) in the same quarter in which their intervention started. An additional 25 had received SA in the previous quarter. Finally, the survey also included 32 interviews with individuals who had received SA in the quarter their intervention started but who were not classified as reachback clients by HRDC. For the post-LMDA participants, reachback clients represented 32% of the survey (compared to 16% of the survey frame and 12% of the population). Individuals who received social assistance (as defined above) represented 10% of the sample compared to 10% of the survey frame and 12% of the population.
5	The detailed matching algorithm is provided in the Methodology Report. Briefly, matching was based on age, sex, EI utilization, EI Benefit rate and Social Assistance utilization. The three best matches were identified for each of 500 (already interviewed) participants. Interviewing initially concentrated on the 500 best matches. After 5 unsuccessful callbacks, attention fell to the second best match, etcetera. This methodology minimizes the differences between the participants and the comparison sample that would otherwise result from non-response bias.
6	At least one individual was interviewed from every HRCC and third party provider where counselling takes place.
7	Note that the case studies involved interviews with 28 individuals as follows: all eight district co-chairs (4 from HRDC and 4 from HRE) two district coordinators two HRCC managers three local HRDC employment program managers one DDRR district representative and one local DDRR representative two local HRE managers one representative of the College of the North Atlantic one representative of a Regional Economic Development Board, and, eight HRDC program officers
8	There are two important transactions that are required to ensure financial accountability. First is the verification that an individual is eligible to participate in TWS. Second is the cessation of EI benefits for active claimants who are hired by an employer who receives a wage subsidy. These transactions are handled locally and apparently do not depend on an individual being on the Interventions file.
9	Note that the resulting distortion of the survey frame has introduced some bias to the sample of TWS participants. In particular, the Avalon district is substantially under-represented.
10	Specifically, the number of SARs given is the number of individuals who received Social Assistance (or whose spouse received Social Assistance) in either the quarter their intervention started or the preceding quarter.
11	Categories are based on age on the day that the intervention was scheduled to start. For example the 20 — 24 group includes those individuals who were scheduled to start their intervention on their 20th birthday up to those who were scheduled to start on the day before their 25th birthday.
12	%'s may add to less than 100 due to 'don't know' or refusal to individual questions.
13	Results are weighted to adjust for sample design. N is the actual number of individuals interviewed relative to each intervention. Note that counts in the Appendices will vary from these due to the effects of weighting.
14	Note that TWS participants from the Avalon district were underrepresented in our survey since they were under-represented on HRDC's Interventions file (see Table 1).
15	Overall, 59% of those who identified themselves as a member of a visible minority also self-identified as an Aboriginal person.
16	Note that the range of hours given varies substantially therefore there is a high standard deviation around the mean. For this reason the median is reported.
17	This question was not asked of SE participants who were presumed to participate in order to establish a business.
18	Averaged over those participants who have had one or more jobs.
19	Based on HRDC administrative data for 1,209 TWS projects and 685 JCP projects.
20	Asked only for one participant projects.
21	Asked only for one participant projects which consists of 26 TWS projects from the survey.
22	For multi-participant projects, the employer was asked this question in regard to a specific participant who had been selected at random.
23	Percentage time employed was calculated as the ratio of # months employed since the program to # months since completing the program. Time attending a post-secondary institution was excluded from both the numerator and denominator. Also, individuals for whom the denominator was less than three months (after excluding time in a post-secondary institution) were dropped from the regression.
24	Defined as the % of total earnings during the 1991-1996 period which were EI benefits
25	A "?" indicates a dummy variable which takes the value 1 if the characteristic applies and 0 otherwise.
26	Such as, for example, econometric models relating to other elements of the Canadian Jobs Strategy and U.S. job training programs.
27	This estimate is significantly different from 0 at the 5% level of significance but not at the 1% level.
28	This estimate is not significantly different from 0.
29	Since many JCP projects are with non-profit organizations, it is unusual for individuals to be retained after the project. This contrasts significantly with TWS and SEB where continuation of the job or the business (at least for the short term) is the norm. See Table 7 where 59% of TWS participants and 82% of SEB participants had a job at the end of the program compared to only 34% of JCP participants.
30	This interpretation is supported by the fact that coefficient of Months since program is double the equivalent coefficient for males.
31	In fact, all key informants noted that improving the skills and experience of participants was typically a lower priority than these other objectives.
32	Outreach Offices are third party delivery organizations that were initially established during the TAGS Program.
33	A 1% random file of people who had received any EI since 1994.
34	Since this data is obtained from income tax data, it is not possible to determine whether this difference is due to EI rate, claim duration or both.
35	That is, the percentage of cases where the survey found people to be employed and the accountability data did not count them as a success, or where the accountability data counted the client as a success when they did not become employed within three months of the end of the intervention.
36	We included both EBSM— LMA — purchase of training and EBSM—CEC purchases in this category.
37	Note that marking clients as reachback started only in April of 1997. Some of the people recorded as being successful in 97/98 may have started TWS in 1996/97 and therefore not be marked as reachback in the administrative data. There may also be other problems with the reachback field but the high number of Unit 46 successes for TWS appears to be impossibly high.
38	In the CATS COUN_SUM file, which contains the action plan results information, the start and end dates of the action plan are entered in only about 30% of cases.
39	An exception is in Labrador where the District Management Committee commissioned a study of EAS expenditures for Outreach offices and will collectively review results to determine possible improvements.
40	Although these EBs do differ from interventions available prior to replacement of the Unemployment Insurance Act with the Employment Insurance Act.
41	Employment Insurance, Part II, Employment Benefits and Measures Tool Kit, March, 1996.
42	As well, HRDC is informing "repeat" sponsors that they are expected to avoid recruiting the same participants as had worked on an earlier project.
43	Defined as those with a current active claim as well as those who satisfy the reachback provisions of the EI legislation.
44	8 or higher on a 0 to 10 scale.