|
|
     |
||
 |
     |
 |
|
Formative Evaluation of Employment Benefits and Support Measures Delivered Under the Canada/Newfoundland LMDAPhase 2 - Training BenefitFinal Report
Evaluation and Data Development June 2000
SP-AH-191-06-00E
This evaluation has examined the delivery of training services provided in Newfoundland and Labrador under the Canada/Newfoundland and Labrador Agreement on Labour Market Development (LMDA). Several factors have affected the delivery of training under the LMDA relative to past practice. These include:
The period since July, 1996 is therefore one in which there have been significant changes introduced that affect training, and the services and level of assistance that clients have received may vary by date and also by HRCC district. This study addresses clients who started training after April 1997; i.e. since the signing of the LMDA. It addresses "enhanced feepayers" and seat purchase. Feepayers1 are not addressed in this evaluation. The evaluation examines the immediate results of the training component of the EBSMs in the post-LMDA period. Since many interventions are still ongoing, and many other clients, even from the beginning of the LMDA period, have only just finished their training, this evaluation does not deal with the long-term impact of training on employment. This will be addressed in the summative evaluation. A major focus of the evaluation has been on the decision-making process that clients have engaged in, including contact with counsellors, and with the adequacy of financial assistance and other supports. Client's Training Decisions The evaluation provides strong evidence that the move away from a seat-purchase model has positively affected training decisions. In particular, the evaluation has found a marked increase in the number of individuals undertaking training in business and information technology and a decrease in studies relating to natural resources, applied arts and apprenticeable trades. Interestingly, these trends are consistent with changes noted for all post-secondary students but are much more pronounced among HRDC-funded students. In particular, there has been a growth or more than 200% in the numbers of HRDC-funded students who enrol in information technology programs as compared to a growth of about 25% in overall enrolment in such courses. Other evidence supports the above result:
Negotiated Financial Assistance Negotiated Financial Assistance processes continue to develop and both advantages and disadvantages of this approach have been noted. This evaluation has found conflicting evidence regarding NFA. On the one hand, the evidence indicates that NFA is, at least potentially, a more cost-effective approach. Although, it has reduced the cost of training assistance most clients (75%) remain satisfied with the level of support. And although it has complicated the work of employment counsellors, they are nearly unanimous (93%) in describing it as a better system. However, the evaluation also raised concerns about the implementation of NFA:
Careful monitoring of the effectiveness and appropriateness of NFA processes is warranted.
Joint Management Response - Formative Evaluation of EBSMs under the Canada-Newfoundland and Labrador Agreement on Labour Market Development AgreementThe Canada-Newfoundland/Labrador Agreement on Labour Market Development (LMDA) is an agreement between the Government of Canada and the Government of Newfoundland and Labrador relating to the co-management of federal employment programs and services specifically under EI Part II. Section 12 of the LMDA outlines the requirement of the Labour Market Partners to cooperate in developing and implementing a two-phase evaluation framework. To this end the Management Committee is responsible for the completion of all evaluations related to the employment benefits and support measures. The Management Committee is pleased that the first phase of the evaluation process, namely the formative evaluation, has been completed. The formative evaluation is comprised of three associated reports. The first report (June 2000) covers three Employment Benefits, i.e., Targeted Wage Subsidy (TWS), Self-Employment Benefit (SEB) and Job Creation Partnership (JCP). The second report (June 2000) deals with the Training Benefit (currently Skills Development), and the final report (June 2001) deals with Support Measures, i.e., Local Labour Market Partnerships (LLMP) and Employment Assistance Services (EAS). The evaluation evidence from the three formative evaluation reports completed to date has shown that in the near term, clients served by the employment benefits and support measures have been moderately successful in achieving their employment goals. Labour Market Development Agreement accountability targets for unpaid EI benefits and client returns to employment have been met in each fiscal year since the inception of the agreement. The Management Committee anticipates that the summative evaluation report, which will contain more definitive evidence, will confirm these findings in the longer term. The Management Committee looks forward to the commencement of the summative evaluation in 2002. The formative evaluation reports have identified many specific findings. The Management Committee suggests that we can deal with these findings under general categories: the functioning of the co-management relationship, findings related to programs/services and findings related to client impacts. A joint Management Committee response to each group of findings is reported below: Co-management: The evaluation reports suggest that "under the co-management approach, it is clear that each government has been able to pursue its respective priorities." The Management Committee suggests that this is perhaps one of the most fundamental achievements of this co-managed agreement, given the unique cultures and priorities of both levels of government. The Management Committee also feels that the effectiveness of the co-management approach is primarily attributable to the level of communication and co-operation among partners. The reports suggest that co-management has increased administrative responsibilities in terms of business and financial planning. The Management Committee feels that it has resulted in a more targeted response to the needs and opportunities of unemployed individuals. The Management Committee also believes that these pressures are abating as the partners become more knowledgeable of each other's programs and collective priorities. Programs & Services Finding: The evaluation evidence suggests that eight out of ten individuals who received services from the HRCCs and Employment Assistance Services offices had a high level of satisfaction with that service. The Management Committee feels that this evidence reflects a strong service commitment of HRCC staff. The committee also believes that despite a challenging economic climate in many rural parts of the province, HRCC staff are making a valuable contribution to the lives of individuals in these communities. With respect to the training benefit two findings are noteworthy from the Management Committee's perspective: Ninety percent of the sample of training participants indicated that participation in the training benefit made them more employable. However, it is recognized that a lack of participant information at this early stage of evaluation, particularly with two and three year programs, did not allow a complete analysis of the impact of training on employment. The Management Committee is interested in the long-term impact of training on subsequent employment gains and will examine this issue in the summative evaluation. The Management Committee has also noted that graduation rates are somewhat unclear given the limitations of administrative data. This issue must be addressed so we can more closely track individual impacts resulting from training. Participant surveys highlight that the support measures offered were well received by program participants. The Management Committee feels that the support measures are essential elements of the LMDA. The measures allow the Management Committee to provide support to incremental initiatives that have many positive impacts for individuals and communities throughout the province. The evaluation suggested that Employment Assistance Services participants welcome the convenience of their being served in their geographic area. " The main tangible benefit emerging from the first three years of EAS seems clearly to lie in the improved employability of the client group." However, additional evidence is needed to determine the overall impact on employment gains that these services are providing. This work will be undertaken in the summative stage of the evaluation process. Local Labour Market Partnerships (LLMPs) have also furnished the Management Committee with the capacity to engage numerous community partners in various incremental activities. The evaluation evidence suggests many valuable activities have taken place, which, in the absence of LLMP, might not have come to fruition. The Management Committee feels that the employment benefits of these initiatives will be realised in the longer term. Client Impacts: The evaluation evidence suggests that SEB and TWS participants experience positive employment gains in the post-program period. This incremental impact shows that investment in these benefits is a prudent expenditure of public funds. The Management Committee desires to maximize the potential of these benefits while recognizing their finite growth potential, given the modest employer and market base in the province. The evaluation indicates that JCP participants did not realise significant post-program employment gains, however, JCP earnings did improve the lives of these participants, their families and the communities in which they live. These results must be interpreted with the knowledge that JCP participants had the lowest levels of pre-program earnings and education of all participant groups. Given this information, the management committee submits that the impact of JCPs on longer-term employment gains must be established before a judgement can be made about the overall utility of JCP participation. The engagement of Social Assistance Recipients in LMDA programs was a priority of the LMDA partners. The report outlines that these targets were met and that SARs achieved positive outcomes from the programs offered. However, the disproportionately higher level of SAR participation in JCP (compared with other interventions) and the subsequent lack of post program employment are of concern to the Management Committee. Improvement of assessment, selection and referral of SAR clients within the EBSM model is a priority of the LMDA partners. It is believed that this approach will lead to a more balanced level of participation of SAR clients in all benefits and measures. A determination of the utility and impact of this new approach should be one of the priorities of the summative evaluation. Aspects to monitor and evaluate: While the Management Committee is generally pleased with the findings of the three Formative Evaluation reports, challenges remain. The following is a list of LMDA dimensions that the Management Committee feels is in need of additional monitoring and evaluation: The Training Benefit report suggests that in the early stages of implementation of Negotiated Financial Assistance (NFA), there was a perception of unequal treatment of clients in terms of the levels of funding received. At the time of the evaluation, Negotiated Financial Assistance was a new process for both clients and staff alike. The problems with dissemination of information and the generally low level of understanding of the process lead to concerns on behalf of both clients and staff. The Management Committee agrees with the consultant's recommendation that this area of concern should be closely monitored to ascertain whether NFA principles are used consistently. Nevertheless, the Management Committee senses that with the passage of time, the intent of NFA has been more widely understood and accepted by all parties. The NFA issues that were identified in the formative evaluation should however be re-examined in the summative evaluation to determine the extent to which these concerns persist. The evaluation evidence suggests a negative employment gain for female JCP participants. This is of concern to the Management Committee who believes that JCP has much economic and social merit. Perhaps more careful targeting of JCP must take place to ensure positive impacts for participants in the post-program period. Again, the summative evaluation should provide more definitive evidence as to the long-term employment impact of JCP. In addition, decisions regarding SAR participation should not be made based on point estimates from the initial evaluation. The evaluation report outlines a data concern associated with Targeted Wage Subsidy administrative data. While this issue has not limited the effectiveness of this benefit, the absence of reliable data for administrative and evaluation purposes may bias our assessments of success or failure, so the data must be improved. The consultant also reported that based on the employers' survey responses, a proportion of the participants would have been hired without the subsidy. The Management Committee recognizes that this finding may outline a potential weakness in the current TWS model. The Management Committee does not however feel entirely comfortable with the integrity of this initial finding. Prior to making any potential adjustments to the program, a more precise and rigorous examination of this issue must be explored. Support measures continue to provide excellent employment assistance and partnership opportunities within the local labour market. The Management Committee considers that the increased level of funding to these activities over the initial period of the LMDA will necessitate further detailed analysis and ongoing monitoring of the effectiveness of these support measures, through the summative phase of the LMDA operational and evaluation plan.
This report is the formative evaluation of the current HRDC training model in Newfoundland and Labrador under the Labour Market Development Agreement (LMDA). This introduction describes the current training model, outlines the objectives of the evaluation and describes the methodologies used. Section 2 takes a preliminary look at the impact of training. Section 3 reports the findings on the following formative issues:
1.1 Description of HRDC's Current Training ModelThis section outlines the most important influences and key characteristics of the current HRDC training model. These include:
The period since July, 1996 is therefore one in which there have been significant changes introduced that affect training, and the services and level of assistance that clients have received may vary by date and also by HRCC district. This study addresses clients who started training after April 1997; i.e. since the signing of the LMDA. It addresses "enhanced feepayers" and seat purchase. Feepayers2 are not addressed in this evaluation. 1.2 Objectives of this EvaluationThe main objectives of this evaluation are:
In addition to these issues, this report also takes a preliminary look at the labour market success rate of those who have finished training. As mentioned in the previous section, the group of clients this study focuses on are those who started a training program after April, 1997. Training programs funded by HRDC can last up to three years, and many college programs are two years in duration. Consequently the examination of post-program success is very preliminary and likely underreports the longer-term success rate of the program since it is based on data from graduates of shorter programs. 1.3 Scope and methodology of the evaluationThe evaluation examines the immediate results of the training component of the EBSMs in the post-LMDA period. Since many interventions are still ongoing, and many other clients, even from the beginning of the LMDA period, have only just finished their training, this evaluation does not deal with the long-term impact of training on employment. This will be addressed in the summative evaluation. A major focus of the evaluation has been on the decision-making process that clients have engaged in, including contact with counsellors, and with the adequacy of financial assistance and other supports. The evaluation used several methodologies, as follows:
A more detailed description of these methodologies is contained in the Evaluation of the training model of the Labour Market Development Agreement Methodology Report. A description of the sample selection processes and survey completion rates can be found in Appendix B. 1.4 Numbers of clientsFigure 1 shows the total volume of training clients and the split between seat purchases and enhanced feepayers. Seat purchases declined dramatically from the 1995/96 fiscal year to 1996/97 but have remained at about 1,300 per year for each of the past three fiscal years. Figure 1 Seat purchases and enhanced feepayers by fiscal year
The Regional data in Figure 2 indicate that the move from seat purchase to enhanced feepayers is largely consistent among the regions. Two exceptions are :
Figure 2 Enhanced feepayers as a proportion of total assistance, by fiscal year and region
Table 1 and Figure 3 provide data on the number of placements made by region for each of four fiscal years. These show the uncharacteristically high number of starts in the Central region in 1995/96 and in the Avalon region in 1997/98. These figures are based on the start of the first contract that an individual received.
Figure 3 Funding starts by fiscal year and region
The following tables in this section show some of the basic characteristics of the clients that HRDC funded for training. This includes information from both HRDC administrative data and survey results. As Table 2 shows, seat purchases clients are more likely to be from smaller communities. Reachback5 clients are also slightly more likely to be from communities of less than 1000 people.  View Table 2 Table 3 provides data on the characteristics of all participants using administrative data. Note that about 8 percent of clients received assistance both in the form of a seat purchase and as an enhanced feepayer in the time period examined. View Table 3
As discussed in the methodology section, the current training model has not been in place for long enough to assess fully its impact. Only clients in shorter programs have finished their training, and it is only for these clients that the evaluation has looked at labour market success. As Table 4 shows, 64 percent of survey respondents have successfully completed their training. As the table also shows, 44 percent of these clients were in training programs that were less than one year long. Another 19 percent were in programs that were less than 18 months long. Note that not all of these clients received funding for the full length of their training.
2.1 Current Status of Clients who have finished trainingAs Table 5 shows, those who successfully completed training are more likely to be working in an area related to their training and also working full time than those who did not complete their training. In interpreting these results it is important to keep in mind that they may under-report the longer-term success rate because most of the clients who have finished training are those who were in shorter courses or programs. Longer programs typically provide a greater benefit in terms of labour market success.
Table 5 also shows that while reachback clients are more likely to be out of the labour force (8 percent compared with 3 percent in the whole population), their overall success rate in finding employment is very similar to all clients who are no longer in training. Clients who have had any work since they finished their training were asked to rate the likelihood that they would have got their current or most recent job without the training that they took. Answers were very polarised, with 38 percent rating the likeliness a 0 (not at all likely) and 28 percent rating the likeliness a 10 (very likely). Not surprisingly, those whose job is related to their training are much more likely to indicate that there was little chance that they would have received their job without the training, whereas those whose job is unrelated are very likely to indicate that they would have received the job without the training. Clients who took longer training programs and who are employed in a field related to their training are less likely to indicate that they would have obtained their job without the training. For example 47 percent of those who are employed in a related field and finished their program in less than six months indicate that they would have likely obtained their job without taking the training compared to 28 percent of those who took a training program six to eleven months long. Clients who have had work since they completed their training were also asked what they think they would be doing now if they had not taken the training. Of those who are working in a job related to their training, 30 percent of those working full time and 24 percent of those working part time think they would be currently looking for work if they had not taken training. 2.1.1 Factors related to successThe large majority of respondents (90 percent) indicate that the training they took has made them more employable. The percentage is high across regions, for both men and women and regardless of previous level of education or length of funding. It is also high whether or not respondents indicate that they did labour market research. Clients who have completed longer programs of 10 months or more and used Labour Market Information (LMI) have a higher success rate in finding a related job than clients in longer programs who did not. Of the clients in longer programs, 64 percent who used LMI are employed in related work, compared to 38 percent of those who did not. Whether or not clients conducted labour market research is not related to success for those who completed shorter programs of 9 months duration or less. Enhanced feepayer clients who have completed their training have a slightly higher success rate in terms of finding employment related to their training than do seat purchase clients who have finished their training. (40 percent in full time and 14 percent in part time compared with 32 and 10 percent for direct seat purchase.)6 The percentage of clients who dropped out of training prior to completion does not differ for enhanced feepayers and seat purchase clients. As Table 4 showed, reachback clients are only slightly less likely to complete successfully their training programs.
3.1 The Decision Making Process of ClientsThere are two main issues related to how clients make decisions about training. One is whether clients are choosing different types of programs under the new system. The other is whether they are collecting Labour Market Information as part of their decision-making process. As described in the introduction, under the new system funding is to be used to help clients who are more likely to find jobs once they complete their training. 3.1.1 Motivation for Taking TrainingTable 6 shows what motivated clients to take training. As the table shows, by far the most common motivation for both enhanced feepayers and seat purchase clients is to improve chances of getting the type of job the client wants. This is also the most common reason for reachback clients. There are no noteworthy differences between the motivations of the different types of clients shown in Table 6. View Table 6 As with motivation for taking training, the reasons for choosing the particular course chosen also does not vary much between groups. For all groups, the primary reasons are their interest in the field and their expectation that employment is available in the field in which they studied. 3.1.2 Impact of Negotiated Financial Assistance (NFA) on types of training chosen by clients.The rationale for the move to a Negotiated Financial Assistance (NFA) approach was twofold. First it was expected to involve clients more directly in the choice of a field of study which would benefit them versus the prior approach where the client's decision was whether to accept (or not) a placement in a "seat" which had been purchased by HRDC. The second perceived advantage of NFA is that funding is based on need and clients are expected to make a personal investment in their training decision. This involvement of the client in first making the training decision and second providing some of the resources to implement it is expected to result in greater ownership of the training decision and thus better commitment to completing the training successfully. This formative evaluation is unable to fully assess whether clients have made better decisions but has been able to examine whether they have made different decisions. The evidence strongly suggests that clients have made substantially different training decisions than in the past. Front line delivery staff members were asked7 whether they think clients are choosing different types of programs under the new system than they did in the past. As Table 7 shows, the large majority of respondents find that clients are choosing different types of training programs under the new system.
Administrative data supports the views of front line delivery staff. Two major changes from the pre-LMDA period are discernible in the courses for which students enrolled in the post-LMDA period. The first is the decrease in support for secondary education (adult basic education). This result is somewhat artificial and is based on a change in HRDC's approach to funding ABE. At about the same time as the LMDA came into effect, the HRCCs decided that a less costly method of allowing people to achieve high school equivalency was to fund third parties to provide open access to all who wanted this training. Previously, EI clients wishing to avail of support to complete their secondary education would (if accepted) be funded in much the same way as those approved for post-secondary courses: i.e. they received a living allowance and money for tuition fees (or these were paid on behalf of the student)8. Figure 4 Changes in course type of HRDC-funded clients pre- vs. post- LMDA 
The other - and more significant - change is that three areas of study have greatly increased in the post-LMDA period compared to the pre-LMDA period. These areas are Business studies, Information Technology, and Engineering Technology. The growth in the number of students studying information technology is, of course, partly associated with the growth in job openings in this field. However, for the whole of the college population, the post-LMDA period saw an increase of about 26 percent9 in numbers of students studying information technology, whereas the numbers of HRDC-sponsored students in this field increased by 215%. In business studies, although the increase among HRDC-sponsored students was more modest (69%), this compares to a drop in the whole college population of 8.2%. Included in the business studies group are secretarial, bookkeeping/accounting and business use of computers type courses, as well as entrepreneurship. These changes are shown in Figure 5, which shows the change in type of course taken in the post-LMDA period (after April 1, 1997), compared to the type of course taken in the year and a half prior to the LMDA being implemented. The figure shows the changes for two groups: those students who were HRDC-funded, and for all students in the province's post-secondary institutions. Figure 5 Pre- and post-LMDA course changes for HRDC-funded and all post-secondary students 
While it is most important to look at changes within the HRDC client population from the earlier period to the post-LMDA period, it is also interesting to note that the course preference of HRDC clients changed in quite different ways from the college population as a whole, as shown in Figure 5. These differences include:
3.1.3 Client decision making process and Labour Market ResearchInformation on the impact of labour market research on client decision making comes from direct questions on the client survey, and from some open-ended questions in the survey of front-line staff. Clients were asked about two type of labour market research; the first on the types of jobs available, and the second on the training required to enter a chosen career. Two-thirds of all clients who were funded to take training indicate that before they started their training program they did research or collected information on the types of jobs that are available. Similarly, 67 percent did research or collected information on the type of training they would need to get the type of job they wanted. There are interesting regional differences. Respondents in the Avalon region were about 10 percent more likely to indicate that they had done both kinds of research compared with those in the Central and Western districts. (In the Avalon 72 percent did both sorts of research compared with 61 and 62 percent respectively in the Central and Western districts who did research on jobs available and 63 percent in both districts who did research on the type of training employers are looking for). Level of education completed prior to taking training with HRDC funding is positively related to whether or not clients did LMI research. The higher the level of education the more likely it is that clients did research. The most notable difference is between those without high school graduation and those with - 47 percent of those who had not graduated from high school indicate that they did LMI research compared with 67 percent of those who did graduate from high school. Of college graduates, 69 percent indicate having done LMI research and 88 percent of university graduates say they have done so. Respondents who indicated that they had done either type of research were then asked in an open-ended question what they did to get this information. Note that almost half of those who had done research used at least two sources for information. Table 8 outlines the responses by region. Front line delivery staff talked about LMI in the context of why they thought the new process (which includes NFA) was better than the old method. As discussed earlier, taken as a whole the system has resulted in clients choosing different programs than they did previously. Counsellors believe that the two major factors in bringing about this change are:
It is clear that counsellors believe the financial commitment aspect of NFA is the most important factor in changing decision-making, however the new emphasis on getting clients to do labour market research before entering a training course is also seen as having a significant influence. Table 8 indicates, by region, the sources employed by individuals when they did their pre-training labour market research. Respondents in the Avalon region were more likely to have consulted directly with employers and least likely to have relied on the HRCC. View Table 8 3.1.4 Client employability expectations on entry compared to their current viewIf clients are doing useful labour market information research then they should make better decisions about training. Respondents were asked questions about how they feel now about the training decisions that they made. When asked to rate on a 0 to 10 scale, where 0 is definitely not and 10 is definitely, if they were doing it again would they register for the same program, 81 percent gave a rating of 7 or above. When asked if they now think their expectations were realistic 92 percent give a rating of 7 or above. Respondents who did LMI research were even more likely to be positive. A series of questions was included about the expectations respondents had when they first registered for the training. As Table 9 shows, all except for 4 percent of respondents were confident that they would find a job when they finished the training.10 Not surprisingly, the greatest regional difference is whether respondents expected to find a job in their local area or elsewhere in the province. Those in the Avalon region were the most likely to expect to find a job in their local area, whereas those in the Western district were the least likely to have this expectation. Reachback clients do not differ from the population as a whole. View Table 9 3.2 Drop-Out rate changesSurvey data (see Section 2.0 above) indicates that an estimated 8% of those surveyed had left their program early without completing while 64% had completed. The remainder were still enrolled in their program at the time of the survey. The evaluation also attempted to determine whether drop-out rates had changed under the new training model by examining administrative data. According to HRDC data, clients in the post-LMDA period are very likely to have successfully finished their programs with 97%11 reported to have successfully completed. This compares to 92 percent in the "transition period"12, and 89 percent in the period before the EI legislation was introduced. This apparent improvement may be due to changes in administrative procedures. Certainly, HRDC staff and officials indicated in interviews that keeping administrative data up-to-date has been a challenge since downsizing13. Most of the people who do not successfully complete the course give up studying before the end of the course. The number of people who complete the course, but do not get sufficiently high marks to graduate from the program, is very small (10 people over the course of five years, or less than 0.1 percent). View Table 10 3.3 Client Satisfaction with Counselling and Availability of LMIRespondents were very positive about being able to find the kind of information they needed to choose a training program to meet their goals, with 78 percent of clients funded giving a rating of 8 or above on the 0 to 10 scale. Another 10 percent gave a rating of 7. Respondents were asked about the counselling that they received before making a decision about training and their level of satisfaction with the counselling. They were asked their level of agreement with a series of statements on their counselling experience. As Table 11 shows, clients in the Avalon region are slightly less likely to have discussed training options with a counsellor whereas those in Labrador are the most likely to have done so. View Table 11 As Table 12 shows, satisfaction levels with counselling are generally high. The greatest regional differences are in the level of satisfaction with information available on jobs in the local area and the province. Satisfaction levels on the Avalon are lower on these items than in other regions. View Table 12 3.4 Negotiated Financial Assistance3.4.1 How the NFA process is actually working in practiceBefore discussing satisfaction levels and other aspects of NFA, it is important to understand how NFA is working in practice. Front line delivery staff were asked about the NFA process in their offices. Similarly clients who took training as well as those who considered taking training but did not were asked about their experiences with NFA. This subject was covered in detail in both the survey and the three focus group sessions held with training clients for this evaluation. Negotiated Financial Assistance and HRCCs All HRCC counsellors, except one, involved in deciding how much financial assistance clients will receive, described a process of negotiating with clients. Counsellors described in varying amounts of detail the process they use with clients to negotiate financial assistance. All mentioned the process of filling out budget sheets and assessing what the client could afford to contribute. A minority (about 20 percent) of respondents indicate that they typically use an arbitrary process where all clients are expected to contribute a fixed percentage of their training costs. Most of these individuals commonly required individuals to contribute 20 percent of training costs. Negotiated Financial Assistance and Third Party Providers Two counsellors at third parties indicated that negotiated financial assistance was not done to decide what level of financial assistance their clients would receive to take training. One indicated that seats were still purchased for their clients and the other that there were set allowances. The remaining 94 percent of the third party counsellors describe NFA processes that fall into three main categories.
Clients' Description of the NFA Process Clients who received financial assistance were asked before they received financial assistance whether they discussed the details of their financial situation with a counsellor. In response, 84 percent indicated that they had, 12 percent that they had not, and 4 percent did not know. Respondents were also asked if they had completed a worksheet outlining day to day expenses and income sources - 68 percent indicated they had, 21 percent said they had not and the remaining 11 percent could not remember whether they had or not. The only regional difference is that almost all of the Labrador respondents indicated that they had completed a worksheet.14 Respondents were also asked which of two statements best described the way in which the amount of financial assistance they received was decided. As Table 13 shows, about half the clients said they were told how much money they would receive, and half went through a negotiation process. The reachback clients were slightly more likely than other clients to indicate that they went through a negotiation process. View Table 13 As described in the methodology section, three focus groups were conducted with training clients. Focus group participants also described a full range of experiences with NFA. These ranged from no negotiation to intense negotiation with three or more meetings to determine the amount of assistance. 3.4.2 Satisfaction with NFAAs mentioned earlier, 93 percent of front line staff see NFA as an improvement over the old way of giving assistance to clients who take training. Nevertheless concerns were identified by some counsellors. Some examples are:
Most counsellors at both HRCCs and third parties (93 percent at each) indicate that negotiated financial assistance gives them the flexibility they need to take individual circumstances into consideration. One quarter of respondents gave examples of cases where NFA did not work. When asked what improvements could be made to NFA, counsellors suggested :
Counsellors at third party providers also made some suggestions that were not made by HRCC counsellors. These include that
Focus group participants indicated that there was a large amount of variation in the determination of financial assistance. On the positive side, almost all focus group participants were eventually reasonably satisfied with the amount of assistance they received, and appreciated the opportunity that it gave them to study. There was support for the principle of determining need in setting assistance levels. On a less positive note, there was a consensus in all three focus groups that the determination of assistance levels was quite arbitrary, if not unfair. Major determinants of the amount of money participants received were (from the point of view of focus group participants) the counsellor they were assigned and timing. For example, in Corner Brook three participants said they were presented with contract with no negotiation. Two others came to an agreement after one meeting, while most had three or more meetings. One eventually called the counsellor to say that she could not pursue training because her family could not live on the amount offered, and was almost immediately offered more funding. Some participants reported having to go to their counsellor's supervisors to resolve funding issues. In St. John's, participants believed there were more funds available at both the start and the end of the fiscal year. One participant was offered an increase in funding for her second year of study although she had not requested one. Another was turned down for funding by one counsellor, but approved by another. Examples of unfairness mentioned by focus group participants include being informed that 100% funding for tuition was not available, but later meeting people whose tuition fees had been paid for entirely by HRDC. It is important to note that, ultimately, participants were generally satisfied with the level of support. However, most have doubts about the consistency of the funding process and the subsequent support levels. In the survey, clients were asked a series of questions about their satisfaction with the NFA process and the amount of financial assistance that they received. As Table 14 shows, satisfaction is higher for the amount of assistance received than for the process itself in all regions except Labrador. This fits with the views of focus group participants who, while generally satisfied with the amount of assistance they received, were frustrated with the lack of transparency and seeming inconsistencies with the way decisions about assistance were made.
3.5 Impact of NFA on client selection and participation In order to assess the impact of NFA on what types of clients are approved to take training, a survey was done of clients who expressed an interest in taking training with HRDC help, but who did not.15 This survey was designed to see if NFA was discouraging certain types of people from taking training, by asking clients directly why they did not take training and also asking them specific questions about different elements of the process and their satisfaction with this process. 3.5.1 Reasons why some clients did not take training with HRDC helpThe survey of non-trainees focused on HRDC clients who wanted to take training with HRDC help, but did not. Table 15 outlines the reasons why these clients did not take training with HRDC help. The most common reason given was that HRDC would not agree to fund them for the course they wanted to take.
In turn, the most common reason given as to why HRDC would not fund them, was that the counsellor thought they would not be able to find a job in field they wanted to train for. A number of these clients felt they were discriminated against because of age. That 27 percent did not take training because they could not afford to contribute to the cost supports the concern of front line delivery staff that the current system does not allow staff to help those who cannot contribute. It is also a concern that 10 percent of respondents were told that the reason they were not funded was because the budget was used up for the year. 3.5.2 Reasons for wanting to take trainingAs Table 16 shows, the most common reason given for wanting to take training was to improve chances of getting they type of job wanted. Responses are listed in the order of how commonly mentioned they were by clients who did take training with HRDC help. (See Table 6). Most of those interviewed (91 percent) had picked out the course that they wanted to take. The motivation for taking training and for choosing the specific training program given by those not funded are quite similar to reasons given by clients who were funded. The main difference is that a much higher percentage of clients who were not funded indicated their main motivation was to change fields.16 Reasons for choosing the particular area of study chosen do not vary between the two groups.
Clients who were not funded were asked the same questions about LMI research as those who were funded. An even higher percentage of those who were not funded to take training indicated that they had done LMI research. Of this group, 71 percent indicated they did research on the types of jobs available and 74 percent on the type of training they would need to get the type of job they wanted. Those who were not funded were also almost equally likely to have discussed training options with a counsellor, however they were much less likely to indicate that their counsellor suggested information sources for them to consult. 3.5.3 Experience with NFA and Satisfaction with the overall processClients who were not funded described a very similar NFA process to those who were funded. They were equally likely to indicate that they had discussed their financial situation with a counsellor and that they had completed a worksheet outlining their income sources and expenses. Not surprisingly, clients who did not receive funding tended to give much lower satisfaction ratings on items relating to satisfaction with level of service received at the HRCC or outreach office, and with the process used to determine the level of financial assistance that would be received. For each satisfaction item at least 20 percent fewer clients who were not funded expressed a high level of satisfaction. 3.6 Client income, assistance and expenses while trainingThis section reports data collected from training clients in the survey of trainees as to their actual expenditures related to training. Training clients report expenses while studying of about $1800 per month; tuition being the largest component of that cost, with average tuition and other direct expenses (such as books and equipment) of $1,019. Figure 6 Breakdown of monthly expenditures for clients while in training
After accommodation and food, the next largest expense is child care. Far more clients reported having dependent care expenses than received explicit funding for dependent care from HRDC. From the survey results, an estimated 42% of training clients incur dependent care expenses. Based on HRDC administrative data, only 17% of training clients receive a dependent care allowance. Clients reporting dependent care costs in the survey estimated these at about $255 per month. Averaged across all respondents (i.e. including the 58% who reported no dependent care expenses) results in the average expense of $147 per month reported in Figure 6. Analysis of administrative data indicates that among those receiving dependent care allowances, the average amount received was $530 per month. An analysis was conducted for this evaluation which linked those who were surveyed to their administrative records. For those who received funding for dependent care and were included in the survey, the amount of funding was, on average, almost exactly equal to the costs that people claimed that they had ($507 per month from HRDC vs. claimed costs of $49717). Table 17 indicates the importance of the dependent care allowances (and the disability allowance for the very small number of clients who received it) in the training decision. Among individuals not receiving these allowances, 35% say they would "almost definitely" have undertaken the training as feepayers (i.e. with no assistance from HRDC other than extending their EI) and an additional 23% say they "likely" would have. Among those who received special allowances, however, feepayer status was much less attractive and the equivalent numbers are 16% for "almost definitely" and 12% for "likely".
The survey also collected data of funding sources. Funding from HRDC is by far the most important source of income while studying, on average accounting for about 59 percent18 of the income of clients during their period of study. The other significant sources of income were student loans and personal savings that account for about 13 percent each, and part-time job earnings (5 percent). Figure 7 Breakdown of income sources for clients while in training 
These figures are in agreement with the perception of many of the counsellors involved in the process; that HRDC assistance ends up at about 60% of total need19. In the survey questions were asked about sources of income other than HRDC funding. HRDC funding was determined from administrative data. On average, clients reported having income during their study period that very slightly exceeded their needs, with average monthly costs of $1800 and average income (including HRDC funding) of $1861. However, on an individual level about 50 percent showed income exceeding costs, with the remainder showing a shortfall. This compares with a level of dissatisfaction with the amount of funding received from HRDC of about 24 percent20. It is likely that at most only the 24 percent who specifically indicated dissatisfaction were actually short of money during their period of study. On average, those who said they were dissatisfied with the amount of funding they received from HRDC said that they would require about $450 more per month. Overall this agrees with the shortfall in income calculated based on costs and income sources, of $494 per month21.
Clients with low satisfaction levels on the amounts they received for specific allowances were asked first whether the allowance was adequate to serve its purpose (i.e. for the child care allowance they were asked whether the allowance was enough to allow them to find adequate child care arrangements) and if not they were asked the details of their problems. Of those receiving a child care allowance, 12 percent indicate they did not receive enough to allow them to find adequate child care arrangements for their children. All except one indicated that the problem was that the allowance was not enough to cover their child care expenses. Of those receiving a travel allowance, 18 percent did not receive enough to allow them to get back and forth to school. Again, all but one indicated that amount was not enough to cover the cost of their travel. Of those who receiving a living allowance, 21 percent indicate that the amount was not adequate to all them to pay for necessities while they took training. Again almost all respondents indicated that the problem was that they did not have enough to cover all of their necessities. All respondents were asked whether there was any kind of financial assistance that they needed to be able to concentrate on their studies but that they did not receive. 23 percent of respondents said yes. When asked what other kind of assistance they needed 89 percent said more money for all areas, 3 percent indicated for medical costs and 3 percent for computer purchases. People who received extra money from HRDC for dependent care or disability (the majority of these being for dependent care) were more likely to say that they needed extra money for good child care or for better food than other clients. In general, the 23 percent of clients who said they needed more money would have needed to receive substantially more funding to have been satisfied. As indicated earlier, the average amount more than these clients indicate they need is $450 per month. Only 4 percent of respondents would have been happy with less than $100 more a month, 25 percent with $100 to $199 and 24 percent with $200 to $299. All others gave higher amounts.
The evaluation has focussed on formative issues and, in particular, on the transition from a "seat purchase" approach to funding training to a training allowance approach using negotiated financial assistance (NFA) where HRDC funds eligible clients based on need. The conclusions of the evaluation follow: HRDC has largely moved to a training allowance approach for funding of training undertaken by eligible EI clients. In 1995/96, 100% of training was funded under the seat purchase model. This fell to about 40% in 1996/97 and about 20 - 25% for the past two fiscal years. Negotiated Financial Assistance processes continue to develop and both advantages and disadvantages of this approach have been noted. Careful monitoring of the effectiveness and appropriateness of NFA processes is warranted. This evaluation has found conflicting evidence regarding NFA. On the one hand:
On the other hand:
Since the signing of the Canada/Newfoundland and Labrador Agreement on Labour Market Development and the introduction of Negotiated Financial Assistance, clients training choices have evolved in a way that is consistent with the opportunities being offered by the labour market. In particular, the evaluation has found a marked increase in the number of individuals undertaking training in business and information technology and a decrease in studies relating to natural resources, applied arts and apprenticeable trades. Interestingly, these trends are consistent with changes noted for all post-secondary students but are much more pronounced among HRDC-funded students. In particular, there has been a growth or more than 200% in the numbers of HRDC-funded students who enrol in information technology programs as compared to a growth of about 25% in overall enrolment in such courses.
The analysis is based on three files. One starts with all the training contracts since April, 1995. The file was aggregated to provide one record per person. In this step the period of HRDC funding was defined as being from the start of the first contract to the end of the last contract. This file was then merged with a file of all non-university post-secondary college students maintained by the Department of Education. There is a file for each academic year. In order to match, firstly either the SIN number in both files had to match, or the surname and birth date; secondly, the period of study in the Education file had to coincide with some portion of the time the person was funded by HRDC. The file of trainees has 25,481 people (i.e. people who received funding for training after April 1, 1995). The file of trainees matched with education data has 7157 people in it. Thus only slightly more than one quarter of the people were successfully matched with education information. One major problem is that the education information for 1997/98 is only about 50% complete. There is none at all for the 1998/99 academic year. The 1997/98 data does not include information on students at the Marine Institute; this probably explains that result that there are fewer HRDC clients in the post-LMDA period enrolled in studies related to the primary sector. The poor percentage of people matched makes it quite likely that there is some systematic bias in the matching process in addition to the Marine Institute anomaly mentioned above. There are some clients who would not be expected to match correctly using these sources of data; these include people who were supported in university studies and people who studied outside the province. Some matches will be missed because of data inaccuracies (mis-entered SINs and name spellings in the Department of Education data). The second file represents students in non-university post-secondary education from September 1996 to September 1998. Where students occur more than once in the file, the last entry is taken. This file has 34,891 students. A third file contains the complete set of trainees (25,481, as mentioned above) with the associated financial information. This is used for the examination of retention rates. The areas of study were constructed by modifying the cluster scheme used by the Department of Education to identify 21 types of program, as given in Table 19. These types were then grouped to give 11 groups. View Table 19 The boundaries between some of these categories are obviously a little fuzzy, and in some cases the course descriptions were not really sufficient to determine accurately into which category a course should be placed. The major differences between this classification scheme and the Department of Education's clusters are:
Technical information, Section 3.6The analysis of income and expenses is based on information from the survey. Although for any particular component of expenses and income there are few missing values (people who refused to answer the question, or did not know the answer), overall slightly less than half of the respondents answered all of the questions. Most of the missing responses are on expense items rather than on income items.
Survey of Training ClientsA survey was conducted of HRDC clients who received financial assistance to take training under the current training model22 and who were "enhanced feepayers" or direct seat purchase clients.23 This population of clients was divided into four main strata for the purposes of sample selection and interview completion. These criteria are:
These categories were then subdivided by gender and age group. On this basis, Table 20 outlines the sampling strategy employed. View Table 20 In total, 600 interviews were completed. As Table 21 shows, the most common reason for non-completion was invalid phone numbers (either wrong numbers or numbers not in service). There was a 9 percent refusal rate.
Survey of Non-trainees A survey was conducted of HRDC clients who had expressed an interest in taking training with financial assistance from HRDC but who did not do so. Developing a sample frame for this survey was not straightforward. From available administrative data, it was not possible to identify with certainty which clients had actually expressed an interest in taking training as opposed to a counsellor identifying a training need. A further complication was that some third party clients who are not HRDC clients also appear in these files. Using CATS and NESS files, all individuals were selected who had a skills enhancement need identified, an incomplete action plan and the reason for the action plan being incomplete as either "changed mind", "did not follow through" or "no contact". This produced a list of 1040 names. In order to ensure that the proportion of interviews completed with men and women would be about the same as that in the sample frame, these names were divided into two lists - one of men and one of women. These two lists were then put in a random order. Individuals were called in the order they appeared on the lists. As Table 22 shows, a total of 485 calls were made. 14 percent of individuals contact did not fit the criteria for the survey. The two most common reasons were that they were never interested in taking training or that they were not HRDC clients when they were interested in taking training. 22 percent of these calls resulted in completed questionnaires with relevant clients for a total of 106 completed interviews.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060127011456im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[To Top]](/web/20060127011456im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}