|
|
     |
||
 |
     |
 |
|
4.1 Information managementElectronic information for the EBSMs is maintained in two separate systems. Within the HRCCs, the staff use NESS to enter information on action plans, set up contracts and track client achievement of action plan goals. Outside of the HRCCs the counsellors at third party delivery sites use CATS to perform substantially the same function. The CATS data contains information downloaded periodically from the NESS system, so that counsellors at these third party sites have access to client history similar to that available in the HRCCs. Data from both NESS and CATS are used in the construction of the results data set, which uses a combination of EI data and the action plan information and follow-up to determine which clients have successfully returned to work, and to calculate any resulting EI savings. The results data set is then processed by the regional HQ to produce regional level reports on achievement of targets. The current system provides a reasonable basis for tracking clients who are participants in EBSMs. In combination with EI data files a good client contact list can be produced. However there are two serious deficiencies in the current data management system.
Key informants indicated that both of these issues are attributable to resource shortages. In particular, full-time staff in many offices have not had the time to collect and enter the data. 4.2 Targeting and participationThere are several goals that Employment Benefits are intended to achieve, and the optimum strategy for meeting one goal may not be the best way of achieving another. For example, the desire to reach Accountability Framework targets for "clients returned to work" may encourage creaming; i.e. selection of participants who are most likely to succeed with or without the intervention and do not actually need the assistance provided by HRDC programming. Conversely, the desire to help those most in need may adversely affect overall success rates as measured by the Accountability Framework. A second example relates to the contrasting implications of economic development goals and providing individuals with skills and experience. Projects that target economic development may do little to develop the skills of individual participants and vice versa. This section looks at some of the issues that arise out of these targeting conflicts. 4.2.1 Targeting of individuals, groups and/or business sectorsKey informant interviews indicate that targeting strategy varies by intervention. Self-Employment BenefitApplications for self-employment are assessed based on merit rather than on priority sector match. Individuals who meet eligibility criteria, have a credible business plan (including market research) and who are not expected to harm existing local businesses are accepted into the program (subject to funding availability). Targeted Wage SubsidiesHRDC personnel typically review applications for a TWS involving a single participant and a decision is made based on the merit of the application and funding availability. TWS applications involving multiple participants are reviewed by local assessment committees as per JCP projects (see below) in some jurisdictions (e.g. Stephenville) but are processed solely by HRDC in most locations. Job Creation PartnershipsLocal Project Assessment Committees review applications for JCP projects. Membership of these committees includes:
HRDC program officers bring applications to this committee for consideration. HRDC retains signing authority for individual project approvals but has moved to a highly consultative process with its partners under the Labour Market Development Agreement. This evaluation examined how these committees function in four areas (Stephenville, Gander, St. John's and Happy Valley-Goose Bay). In Stephenville, the local committee has developed as follows:
In the other three locations, the last two points do not apply. The interviews with HRDC personnel in Gander indicated a more flexible set of priorities is in place in that area. While economic development is an important criterion for project assessment, projects that provide only income and/or skill development may also be considered especially in communities where unemployment is very high. In St. John's, we were informed by HRDC personnel that — like in Stephenville — the Project Assessment Committee operates using a consensus model and places significant emphasis on economic development due to the involvement of DDRR and the REDB. An increasing focus on the needs of Social Assistance Recipients was also noted. Projects that focussed on SARs as well as initiatives to improve the access of SARs to Labour Market Information (LMI) were noted. In Happy Valley — Goose Bay, geography has presented some unique challenges. In particular the Happy Valley - Goose Bay Project Assessment Committee considers applications from coastal Labrador, which have been recommended by one of four sub-committees, which involve local officials in smaller communities. These local sub-committees are still in a development phase. The survey of front line staff also provides evidence of a focus on economic development. In total, 87 percent of HRCC staff and staff of third party providers gave examples of EBSM projects that have been tied into local economic development initiatives, indicating that sectors which fit into local economic development plans are being targeted. Some also mentioned that a positive aspect of change with LMDA has been more of a focus on local economic development. According to multiple lines of evidence, HRDCs spending on Employment Benefits is increasingly focused on economic developmentAs noted above, the extent of this increase has varied based on intervention type and locality. Key informants at the regional level noted that the industry sector targets in the regional business plans were set at levels which:
In the survey of front line delivery staff, 28 percent of HRCC staff and 32 percent of third party provider staff indicated that they think the identification of priority sectors has meant that some candidates who would benefit from EBSMs have not had access to them. Key informants, however, indicated that the priority sector targets were low enough so that applications of merit from outside these sectors could typically be funded. The identification of SAR reachback targets has only affected the participation of SAR clients in HRDC Employment Benefits in a limited way to date.Staff at both HRCCs and third party providers commonly commented that the LMDA had broadened their client base with the inclusion of SARs, but other evidence suggests that by the end of 1998-99 progress in this direction was quite limited. Interviews with key informants from HRE and HRDC conducted in the fall of 1999 established that effective steps to truly broaden the client base to include SARs were anticipated but had not yet occurred. Two primary reasons were cited:
HRE and HRDC as co-chairs of the LMDA have recognized and are addressing the first two points. In regard to the first point, SAR reachback targets have been doubled for 1999/2000. On the second point, development of HRE's capacity is proceeding according to its strategic priorities. As well, how to best identify and case manage SAR clients has been considered at provincial, district and local management committees. In the fall of 1999, HRE personnel were receiving training in Service Needs Determination to better equip them to identify appropriate candidates for referral to HRDC. HRE has also acted to reduce financial disincentives by extending drug cards for families with dependents. Nevertheless, based on key informant interviews and the survey of delivery agents conducted late in fiscal 1998-99 the policy issues presented by "shared clients" were not fully resolved at that time. The most common issue raised by counsellors at both HRCCs and third parties is that there are financial disincentives to SARs participating in employment benefit programs. SARs with dependants (who need to arrange child care) and single SARs with health problems in particular may end up to be worse off financially if they give up social assistance to take an employment benefit program. As part of the survey of front line delivery staff, respondents were asked whether they have found that SAR reachback clients have different needs than other clients. Overall, 65 percent of respondents indicate yes and another 16 percent say "yes and no". These latter respondents mainly want to distinguish between two types of social assistance recipients. The longer term SARs have additional needs compared with other clients, but short-term, usually first-time SARs do not. Delivery staff are split on the extent to which SAR needs are met with 36 percent of respondents answering that the needs of SARs are being met fully, 41 percent that they are being met somewhat and 22 percent that they are not being met to any great extent. Those at third party providers were more likely to indicate that the needs of SARs are not being met, with 29 percent giving this answer compared to 13 percent of HRCC staff. Similarly HRCC staff were more likely to indicate that they are being met fully with 44 percent giving this answer compared to 31 percent of third party staff. Counsellors also discussed how SARs have more self-esteem issues, emotional problems and other barriers that require individual counselling and often family counselling. Here HRCC and third party counsellors had a different view on how to deal with these sorts of issues. HRCC counsellors want the provincial Department of Human Resources and Employment (HRE) to continue to play a role with these clients, particularly when there are family related issues. Counsellors at third parties generally feel more able to help these clients, but worry that with increasing workloads they will not be able to continue to do this in the future. Counsellors at both HRCC offices and Outreach Offices32 also worry that they do not have the training to assess the needs of SARs. This was not raised as a concern with other types of third party providers. Both types of counsellors raised a lack of literacy programs as an issue when dealing with SARs. Many also brought up a lack of local economic development initiatives and suggested when there are initiatives there should be some JCP-type positions or quotas for SARs to give them a bit of an edge. Suggestions for improvement made by counsellors include:
Some suggested that guarantees could be made to SARs who take training such as if they successfully complete the training they will be guaranteed that funding would be available for a targeted wage subsidy for the year after their training. To avail of this opportunity, it would be necessary to recruit a willing and suitable employer. 4.2.2 Are program participants representative of the unemployed labour force?Income tax data was used to examine differences between EBSM participants and the Newfoundland population of EI recipients33. The major differences are that:
4.2.3 Is the selection of clients affected by the accountability framework?Respondents to the survey of front line delivery staff noted that the accountability framework has led to an emphasis on helping clients who can be more easily placed in an employment benefit program. These clients are helped as opposed to those who require in-depth counselling or who have greater need such as those with disabilities or those who require literacy training. The accountability framework means that counsellors are primarily judged on short-term success. Many third party counsellors also expressed frustration that the accountability framework does not look at the starting point of clients. 4.2.4 Achievement of targetsEach of the four HRDC districts (Avalon, Central, Western and Labrador) has a target defined for number of clients served, clients returned to work and EI savings. These targets are specified in the Labour Market Development Agreement and are defined for the 1998/99 fiscal year. Clients ServedTable 15 shows the district targets for clients served for fiscal 1998/99. These targets were well beyond activity in fiscal 1997/98 (the first year of the LMDA) since activity was curtailed in 1997/98 by multi-year commitments to training clients in the previous fiscal year.
Return To Work TargetsTable 16 identifies the overall (including training) success targets for returns to work. It was not possible to validate these numbers since the data available was only for participants in JCP, TWS and SEB whose interventions started between July 1, 1996 and December 31, 1998 and whose interventions had ended by February, 1998.
Table 17, however, examines the HRDC results data for those individuals surveyed and compares the HRDC definition of successes with the survey estimates. Using the survey data on jobs after the intervention, for TWS and JCP, anyone who started a job within 3 months of the end of the intervention or who was still working for the same employer was counted as employed. For SEB clients anyone who was still running his/her own business or who got a job within three months of completing the program was counted as a success. As indicated in Table 17, the survey results indicate that the number of successes is greater than the number of successes recorded in the HRDC results database. In addition to this apparent undercounting of successes, the two sources did not agree well. They agree in about 56 percent of cases, and disagree in 44 percent (identified as mismatches in Table 17). Table 17 illustrates the dual nature of the problem for each of the three interventions. Results in Table 17 are based on all individuals surveyed (excluding the extra reachback clients surveyed). In summary this table indicates that:
To illustrate this dual problem, Table 18 illustrates the detailed comparisons for Targeted Wage Subsidies participants. For 221 of the people the survey and the results data agree on the result, including 146 that both indicate were a success. For 152 people (41%), the two sources disagree, the majority (103) being people not counted as a success in the results file, but who are "successes" based on the survey data. However, there are an additional 49 individuals classified as successes in the results data base who did not return to work within three months of completing their subsidized employment.
EI Savings TargetsThe business plans for each district also specify the EI savings targets that are expected from getting people successfully employed. These savings are counted only if the client becomes employed within the period in which they are entitled to collect EI. There are thus no savings attributed to getting reachback clients employed. As well, individuals who are active claimants when they participate but whose claim expires before returning to work do count as "return to work" successes but do not count as "savings to EI account" successes.
Based on Table 19, it is clear that the SEB, JCP and TWS interventions do not account for a significant share of EI savings targets. Table 20 shows the breakdown of the source of successful clients in 1998/99. The information is based on matching the 98/99 successes with the HRI client data for 97/98 and 98/99. Clients often receive several types of assistance from HRDC. Where a client had multiple interventions, the client is assigned to a single intervention in the same order as they are presented in the table. For example, if a client participated in TWS and also attended a counselling group, he/she would be assigned to TWS.
The table confirms the information in Table 19 that SEB, TWS and JCP account for a small proportion of the EI savings calculated by HRDC. Clearly training (both seat purchase36 and enhanced feepayers) account for a large proportion of the estimated savings. The Other EBSM category is also large. The major component in this is Govt. to Govt.—RBA. Note that although clients have been counted as being in group services and counselling group sessions only if they participated in no other interventions, that these two items account for a substantial proportion of savings (almost double the combined contribution of SEB, JCP and TWS). 4.2.5 Validity of Accountability Framework DataThe variations between the Results Data and the survey data raise serious questions as to the accuracy of HRDC's data. We assume that one source of the missed successes are problems in case management. In particular, many of the missed "return to work" successes are likely due to:
The results file differentiates between four groups of clients who "return to work" as illustrated in Table 21.
The results file contains successes for any case-managed client and for people who attended group sessions. For the EBSM evaluation we are primarily interested in the three employment benefit measures (TWS, JCP and SEB). The results file shows successes for 1422 people (of 5567) who participated in an employment benefit after July 1996. The success rate using the data from this file are given in Table 22 along with the total numbers of interventions and reachback clients for each intervention.
There are two major apparent inaccuracies in Table 22. Firstly, the number of Unit 43 successes in TWS is impossibly high. This intervention is generally one year in duration, so in most cases the end-of-intervention action plan indicator of success would be entered after the client's claim ended. The other inaccuracy is the large number of Unit 46 successes. For a Unit 46 success the following must be true:
The Unit 46 code is designed to capture reachback clients, i.e. those who received help even though they had no active EI claim. There are obviously other clients being captured in this category. Note for example that the results file classifies 250 successful TWS placements as Unit 46 successes even though the administrative data identifies only 3 TWS clients as reachback clients37. The likely problem here is that even though a client really did start an intervention while they were on EI, some problem with the data resulted in the match not being made in all cases38. 4.3 Co-managementUnder the Canada-Newfoundland and Labrador Agreement on Labour Market Development (LMDA) the two levels of government agreed on the following:
The governing structures stipulated in the LMDA were as follows:
The LMDA stipulates that decisions of all five committees are to be made by consensus and if consensus cannot be reached than decisions move up to a higher level. The LMDA also includes a commitment to review the design of the EBSMs covered by this evaluation as well as to develop a new training benefit. The LMDA does not stipulate how co-management is to be implemented at local levels. In practice, local Project Assessment Committees have been instituted which examine applications for Job Creation Partnership Projects, Local Labour Market Partnership and in some cases, larger Targeted Wage Subsidies. In general, it is our understanding that local HRDC managers retain responsibility for management of local HRDC operations and that the local Project Assessment Committees are generally not involved in other aspects of HRDC operations39. Front line staff at HRDC and third party delivery organizations were asked how the introduction of federal-provincial co-management of EBSMs has changed their day to day work. About a third (30 percent) of respondents with three or more years of work experience indicate that it has not changed their day to day work. Respondents at third party organisations were more likely to indicate that it has not had an influence, with 37 percent indicating there has been no change compared with 20 percent of HRCC staff. The evaluation included four case studies of co-management (Stephenville, St. John's, Gander and Happy Valley — Goose Bay). Based on interviews with representatives on the Project Assessment Committee, HRDC personnel, and the co-chairs of each of the four District Management Committees, these local committees are functioning effectively without any serious complications. In Happy Valley — Goose Bay, however, processes for considering projects from coastal Labrador are not yet fully effective. 4.3.1 Advantages of co-managementKey informants saw co-management as an effective approach. In particular, they noted that review of projects is much improved under co-management. In the past, they noted:
Under co-management, all parties can consider a project together which offers the potential to eliminate these difficulties. In particular, by having the various government organizations working together, it is much easier to:
In Stephenville, all parties were convinced that these potential advantages are currently being realized. In the other three locations, the full potential advantages from working together has yet to be entirely realized in the view of those interviewed. All expected continuing progress. Key informants also noted that the co-management of EBSMs has also led to several other advantages in other areas of programming:
In terms of positive aspects of the change, both HRCC and third party staff most commonly mentioned that duplication of services is being avoided and that there is more communication and co-operation between the province and HRDC. Respondents discussed how both levels of government now promote each others services more and are generally more aware of what each other has to offer. That they could now help a broader client base was also commonly mentioned by both kinds of staff. Third party staff mentioned several positive aspects of co-management that HRDC staff did not. Some have found that since co-management they have been getting more support from HRDC and that they can do more for specific client groups, including SARs and disabled clients. 4.3.2 Disadvantages of co-managementKey informants saw few disadvantages to co-management. Those difficulties which were identified were considered surmountable or to have compensating advantages. For example, the delays in approval of Regional Business Plans for 1999/2000 was lamented and noted as a barrier to best implementation. However, the perceived solution was commitment of all parties to a timely approval process rather than changes to the co-management process. Also the possible delays in review of projects by the Project Assessment Committees was noted by some. However, this concern did not exist in Stephenville where the PAC functions smoothly and is perceived by all parties as an improvement on the previous process. The biggest disadvantage of co-management identified by key informants from HRDC is that co-management is perceived by many as a temporary situation and this perception has caused serious morale problems for HRDC personnel. In particular, most HRDC personnel believe that either the province will decide to take full responsibility for employment programming under a devolution approach as has been applied in most other provinces or that Ottawa will pressure the province to make such a decision. Respondents to the staff survey tended to give more examples of negative changes than positive ones. HRCC and third party staff gave fairly different responses. The most commonly mentioned negative change by HRCC staff is that making decisions has become much more time consuming because they have to consult and co-ordinate with so many outside agencies. Some expressed frustration that they can no longer make decisions and that they do not have the authority anymore to make decisions about they type of supports that would best help their clients. Some HRCC staff members have also found that under co-management they see a shift in focus as the year goes by. In the first quarter they are encouraged to try to help clients find sustainable employment by whatever means is best and by the last quarter they are just trying to get clients in programs so they can spend the budget. Key informants noted a particular problem in 1998/1999 when substantial additional funding became available late in the year when the Pan-Provincial Budget was not fully spent. Several informants with both HRDC and HRE noted that this damaged HRDC's credibility with its partners as well as undermining the efforts of all parties to communicate that HRDC support is now limited to projects/initiatives providing the potential for sustainable employment. Both HRCC and third party staff noted an increased workload with no additional staff, and in some instances reduced staff, as a problem under co-management. This, together with having to spend more time on data entry to ensure accountability framework measures are tracked, is, according to many of those interviewed, causing a situation where counsellors cannot always get to know a client well enough to know what is best. 4.4 Case Management and Client Counselling4.4.1 Is a tracking system in place?All front line staff surveyed use either CATS or NESS (or both) to track clients. Some counsellors at third party providers commented that while they use CATS to supply the necessary information to HRDC to track for accountability framework purposes, they do their actual case management from their paper files. As Table 23 shows, HRCC counsellors are more positive about the effectiveness of their current method of case management than are counsellors at third party providers.
4.4.2 Participant satisfactionParticipants were asked a series of questions about their satisfaction with the services they received and with their job/placement in the Targeted Wage Subsidies or the Job Creation Partnership programs. As Table 24 shows, there is a high level of satisfaction with the level of service received at the HRCC or Outreach Office. The satisfaction level in Labrador is somewhat lower than in other regions. Satisfaction with the variety of services offered also varies by region, with participants in the Avalon and Central regions being more satisfied than those in the Western Region and Labrador.
Satisfaction with the actual work placement is lower, in particular in terms of the work being related to career objectives and being useful in helping participants to find a job. Provincially, 47 percent of respondents were satisfied with how closely work was related to their career objectives. However, TWS participants were more likely to indicate satisfaction with career-relatedness with 58 percent indicating high satisfaction compared to 45 percent of JCP participants. Those who are no longer working for their project employer were asked how useful the program has been in helping them find a job. Only 28 percent have found the experience to be useful. However, 37 percent of TWS participants have found the program very useful in helping them to find a job compared with 27 percent of JCP participants. 4.4.3 Participant involvementCounsellors were asked about what percentage of EBSM participants develop action plans. As Table 25 shows, the clients of third parties are less likely to develop action plans than the clients of HRCCs.
When asked to rate the extent to which the development of action plans benefits clients, 57 percent of HRCC staff rate the benefit 9 or above on the 0 to 10 scale (0 not at all and 10 very much so) and 40 percent of third party provider staff rate the extent of benefit a 9 or above on the same scale. When participants themselves were asked "with the help of an employment counsellor at an HRCC, did you develop an action plan for the future?" only 12 percent indicated yes. The percentage was the highest in Central where 16 percent of participants said they had developed an action plan and was lowest in Western where only 6 percent said yes. These low percentages show that clients themselves are not necessarily aware of the action plan process and therefore are likely not as involved in this process as they might be. Based on these statistics, the primary benefit of Action Plans to clients appears to be that they are required to access EBSMs. 4.5 Labour Market InformationBoth the survey of front line staff and the participant survey asked questions about the availability and use of labour market information. 4.5.1 Availability of Labour Market Information to front line staffIn the survey of Delivery Staff, 52 percent of HRCC staff and 67 percent of third party staff indicate that they have the labour market information resources and tools that they need to be able to help clients. When counsellors were asked what was missing a range of responses was given. Those at HRCCs tended to give more specific answers than those at third party providers. HRCC staff mentioned they were missing (in order of frequency of mention):
Staff at third party providers indicated that they were missing (in order of frequency of mention):
Most respondents indicated that they would benefit from additional training on labour market information. In particular, 83 percent of those at HRCCs and 93 percent of those at third party providers indicate that they would benefit from LMI training. Staff members at HRCCs are interested in the following training opportunities (in order of frequency of mention):
A few people in HRCCs commented that the same people get all the training and in particular that those in acting positions never get to take any training. Many commented that time was the big issue for LMI. They just don't have the time to stay up to date with all of the other pressures they have. Staff members at third parties are interested in:
Some third party staff at outreach offices commented that they should have access to updates and training provided to HRCC staff. Interestingly counsellors at third party providers are more likely to indicate that they have had useful LMI training than have those at HRCCs with 57 percent of third party staff indicating that they have received useful LMI training compared to 20 percent of HRCC staff. 4.5.2 Client satisfaction with the availability of Labour Market InformationAs Table 26 shows, participants are generally satisfied with the LMI available to them at local offices. A higher percentage of clients are satisfied with information available on the local labour market than they are on the provincial or national job market.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060127004545im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127004545im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127004545im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127004545im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}