|
|
     |
||
 |
     |
 |
|
In this chapter, evaluation findings related to the success of the PBPMs to date are presented. It is important to note that this formative evaluation provides preliminary evidence on the success of the PBPMs. In-depth analysis and assessment of the success of the PBPMs requires data gathered over a longer period of time and the application of higher level analysis than was possible during the formative evaluation. A summative evaluation, planned for fiscal year 2002-03, will be undertaken to provide a comprehensive and thorough reporting of the success of the PBPMs. Issues addressed in this chapter include promotion of client self-reliance, the attainment of results targets, survey results pertaining to the impact of the PBPMs on clients, the impact of PBPMs on employers and communities, features of PBPMs associated with positive outcomes, and unintended impacts of the PBPMs. The treatment of issues presented in this chapter draws primarily on evidence derived from the participant and comparison group surveys, as well as the key informant interviews and focus groups. In several areas, such as satisfaction with the PBPMs and perceived impacts, a number of sub-group differences by program type, as well as by respondent demographics, were observed. Subgroup differences by program type are reported in the text. Differences according to respondent demographics are not presented in the text, but appear separately in Appendix B30, as no clear pattern of results was apparent.31 5.1 Promotion of Client Self-RelianceMore than half of the respondents indicated that they had spoken with an employment counsellor or facilitator to help them plan their strategy to return-to-work (56 per cent), and 23 per cent of respondents indicated having developed an action plan with an employment counsellor. Fully 78 per cent of respondents who developed an action plan indicated that they had completed the activities in their plan. By far the most common reason for non-completion of action plan activities among the remaining 20 per cent of these participants was that they had found a job (61 per cent) (Exhibit 5.1). Focus group participants (Employment Centre staff and third-party delivery agents) agreed that the design and delivery of the PBPMs is compatible with a focus on client responsibility. Respondents stated that some clients are more self-reliant under the PBPMs and are assuming more responsibility for ensuring their own success. Others commented that the level of responsibility assumed by clients varies with each individual depending on a number of factors, such as their willingness to accept this responsibility, their life skills, and their social and economic conditions. With respect to making decisions about training, preparing and following through on action plans, and job search, staff generally agreed that the clients' responsibility has increased. It was further mentioned that there are better resources in the Employment Centres now for clients to do their own research, but labour market information is limited and is often not up to date. Some Employment Centre staff also stated that the termination of Purchase of Training and the new Skills Loans and Grants/Enhanced Fee Payer approach helps promote client responsibility because clients are asked to make a commitment to their training. 
Third-party delivery agents reported a number of factors associated with client involvement, including: providing a safe and secure environment where trust and confidence can develop between the client and the service provider; client's self-identification of their employment needs; appropriate referrals to services; and proper needs assessments. Opinions varied, however, as to the willingness and capacity of clients to assume responsibility for their employment situation. A barrier for some clients is the poor access to training in rural communities coupled with their inability to re-locate to a larger centre due to family responsibilities. Several third-party agents also observed that many clients lack job search skills and are not necessarily job ready, so there is a limit on how much responsibility they can assume. In addition to developing job search skills these clients would greatly benefit from receiving coaching and developing life skills. Clients themselves indicated that they take on a lot of responsibility and many stated they took almost total responsibility for ensuring their own success. Clients noted they assumed responsibility in identifying their needs, conducting research, and selecting not only the type of program or training they should pursue but also the training provider. For the most part, clients felt they were asked to take on the right amount of responsibility. However, some clients felt they were asked to take on too much responsibility and interpreted staff's promotion of their self-reliance as poor service; they felt that because staff were expecting them to do the research and to find out what was available to them (e.g., training programs, college courses), the staff were not doing their job. Also, for clients who are not computer literate the job bank kiosk was difficult to use. Overall a majority of clients believed that an appropriate balance had been established in terms of client-counsellor responsibilities. With respect to the government's role in ensuring the success of clients involved in various PBPMs, most clients agreed that the government assumed an important role with respect to financing their training. Many participants stated that they would not have been able to access training without this support. In summary, staff and clients agree that clients are assuming more responsibility for their employment situations, although the degree to which they are able to do this differs as a function of skills and personal resources. Additional barriers to self-reliance were felt to be the lack of good labour market information and poor access to training in rural areas. Most clients felt they were asked to assume an appropriate level of responsibility, though some clients interpreted the emphasis on client self-reliance as poor service delivery. 5.2 Primary IndicatorsThe Canada/Manitoba LMDA, like all provincial and territorial labour market development agreements, specifies primary results indicators for Provincial Benefits and Provincial Measures (PBPMs). The targets are set annually in three areas: the percentage of PBPM participants who are active EI claimants; participants returning to work; and unpaid EI benefits resulting from active EI claimants returning to work before the EI claim has ended. It was impossible to fully assess target attainment for the 1999/2000 fiscal year owing to the timing of the evaluation, administrative data being available only up to August 15, 1999. Table 5.1 below presents the results of the computations while Appendix C32 contains the complete analysis.
5.3 Comparison of Evaluation Findings and Accountability MeasuresIn this evaluation, employment outcomes from the participant survey were compared against the return-to-work results in the administrative data systems. Our analysis found that administrative data returns to work were 66 to 80 per cent of returns to work as reported in the survey. See Appendix C33 for the complete results of this analysis. 5.4 Impacts on Participants: Descriptive AnalysisThis section presents a summary of findings relating to the impacts of PBPMs on clients. The focus is on clients' ratings of the importance of the help they received in obtaining employment, as well as objective measures of labour-market outcomes as revealed by clients' labour market status. The client outcomes that are presented in this section include: employment, joblessness, job-search behaviour, attitudes, and utilization of income-support. All sub-groups were compared to the participant total to determine whether they were significantly different from the overall result. The interested reader is referred to Annex 1 of this report34 for the relevant data tables. It is important to bear in mind that a true assessment of program impacts can only be made through the modelling of survey results, whereby various potentially confounding factors can be taken into account to provide accurate estimates of PBPM impacts. For example, the fact that one PBPM appears to be more successful than another may have more to do with the characteristics of the PBPM participant (e.g., greater education or motivation) than the PBPM itself. It is those kinds of factors that multivariate analyses can control for. Additional information is provided in Section 5.5 of this report. Thus, the presentation of bivariate results in this section should be used for descriptive purposes only. Rated Importance of AssistanceSurvey respondents were asked to rate how important their employment program was in helping them to get their current or most recent job. Participant ratings of the importance of their employment program in this respect were moderate (Exhibit 5.2). 
Perceived Program ImpactsSurvey evidence reveals that the PBPMs were perceived to have had at least a moderate impact on a number of employment characteristics for the majority of PBPM respondents (Exhibit 5.3). Qualitative findings suggest, however, that the ability to detect impacts may be limited by inadequate tracking and monitoring systems, and narrow definitions of success related to immediate, rather than long-term, outcomes. Some clients were less happy with the impacts of PBPMs on improving their opportunities for better-paying jobs and their quality of life, and third parties expressed some concern over the limited access to PBPMs in rural areas. 
Interest in Entering the Labour ForceParticipants' motivations to be employed were measured through their rated interest in entering the labour force in the next 12 months. Of those respondents who were jobless at the time of the survey, the vast majority of participants rated themselves as very interested (responded with a 6 or 7 on a 7-point scale) in entering the labour force in the next 12 months. As expected, given their proactive behaviour in accessing a PBPM in order to return to work, participants were more likely to be interested in entering the work force than comparison group respondents. As well, active EI claimants in both groups were more likely than reachbacks to be very interested in entering the work force. Labour Market OutcomesThis section presents survey evidence on several employment measures, including employment rates, employment stability, employment status and retention, for participants and comparison group members by claimant status (i.e., active EI claimant versus reachback). It should be pointed out that a different pattern of results was observed for reachbacks and active EI claimants for many of the employment outcome measures discussed in this section. Specifically, comparison group reachbacks were found to have more positive outcomes than participant reachbacks and comparison group active EI claimants, and similar outcomes to participant active EI claimants. As a result, for many of the employment outcomes presented in this section, while participant active EI claimants were found to have more positive results relative to their comparison group counterparts, the opposite pattern was observed for reachbacks (i.e., participant reachbacks had less positive results relative to comparison group reachbacks). Comparison group reachbacks differ from the other groups in that they would have received no form of assistance, whereas all of the participants and the comparison group active EI claimants would have received some form of assistance, either through employment insurance (EI), employment programming (PBPMs), or both. While existing data do not indicate significant differences in profiles between comparison group reachbacks and to the other groups, it is possible that these data do not capture all differences between the groups that could account for the more positive employment outcomes observed for the reachback comparison group. a) Employment Rates Overall, the survey results suggest that an advantage may exist for participants compared to comparison group members in terms of employment but not in terms of employment stability. Participants were more likely than comparison group members to be employed in the post-program period (first panel of Table D.1 in Annex 1). b) Employment Stability Data on two measures of employment stability were collected in the participant and comparison group surveys: the proportion of respondents who have worked for 12 consecutive weeks following the end of the program or reference date, as well as the number of employers they have had since that time (second and third panel of Table D.1 in Annex 1). While participant active EI claimants were more likely than comparison group active EI claimants to have worked for 12 consecutive weeks, the opposite pattern was observed for reachbacks. As well, comparison group members showed more stable employment than participants in terms of the number of employers they had in the post-program period. c) Employment Status Outcomes Participant survey respondents were asked about their employment status at two points in time following their intervention: at one week following the end of the program and at the time of the survey. Overall, these results show a positive shift in employment between these two times for PBPM participants, with the largest positive shifts in employment occurring for full-time year-round jobs (Table D.2 in Annex 1). These findings may indicate a persistence of positive employment outcomes among PBPM participants. d) Pre-Post Employment Status Another way in which the employment outcomes of LMDA program participants were measured involved a comparison of employment status in the week prior to the intervention or reference date and employment status at the time of the survey. Overall, these findings show positive shifts in employment from the pre-to post-intervention periods for PBPM participants. Relative to the comparison group the magnitude of the positive shifts in employment for PBPM participants was even greater (Table D.3 and D.4 in Annex 1). e) Retention A more direct measure of the contribution of PBPMs to positive employment outcomes is the extent to which job placement program participants were hired on by their host employers following the completion of the program. Only participants who completed their job placement program were asked this question. A majority of job placement program participants were retained in their PBPM program jobs, although the rates of retention, of being hired into the same program job and of being hired into full-time year-round employment were higher among Wage Subsidies participants relative to Employment Partnerships respondents (Table D.5 in Annex 1). Characteristics of Current/Most Recent JobIn this section, the characteristics of respondents' current or most recent job are presented (for those respondents who have been employed at some time in the post-intervention period). In the post-program/reference date period, roughly one in four participants and roughly one in two comparison group members had the same job they had prior to their employment program reference date (Table D.6 in Annex 1). Participant active EI claimants worked longer hours than respondents in the other groups, and were more likely than comparison group active EI claimants to be employed year-round, although the opposite pattern was observed for reachbacks. Comparison group members reported higher weekly earnings. Joblessness and Job Search OutcomesSurvey results for three post-intervention outcomes: number of weeks jobless (duration); number of weeks looking for work while jobless; and job search activity are presented in this section. "Jobless" individuals are defined as people who are officially unemployed (i.e., unemployed and looking for work), plus those who are not in the labour force. a) Duration of Jobless Spells Survey data were collected on the duration of jobless spells following the intervention (or reference period), scaled by the time since the intervention or program reference date, for both participants and comparison group members. The duration of jobless spells in the post-program period was shorter for participant active EI claimants than comparison group active EI claimants, although the opposite pattern was observed for reachbacks (first panel of Table D.7 in Annex 1). b) Duration of Job Search Another potential positive outcome concerns the number of weeks clients searched for work in the post-program period. It should be noted that the analyses conducted on this measure include only those respondents who were unemployed at some time in the post-program period. Overall, participants were much more likely than comparison group members to look for work in the post-program period (second panel of Table D.7 in Annex 1). c) Job Search Activity Those who reported actively searching for work in the post-program period were asked to specify the job search methods they used while looking for work. The most common job search methods were distributing resumes or applications and checking job banks (Table D.8 in Annex 1). Participants were more likely than comparison group respondents to engage in most job search methods. Utilization of Income SupportThis section presents results regarding participants' post-intervention use of two forms of income support: Income Assistance (IA) and Employment Insurance (EI). a) Income Assistance Another indicator of the extent to which PBPMs have positively impacted participants is the extent to which participation in the PBPMs has reduced clients' reliance on income support (IA). Survey evidence suggests that participant active EI claimants may have benefited from the program in terms of a reduced rate of IA use in the post-program period, although they may also have been at a disadvantage concerning the duration of IA use (first and second panel of Table D.9 in Annex 1). b) Employment Insurance Evidence concerning the incidence and rate of Employment Insurance (EI) use in the post-program/reference date period revealed that, overall, both the rate of EI use and the duration of EI use was lower for participant active EI claimants than for comparison group active EI claimants. The opposite pattern was observed for reachbacks. Thus, any advantage observed for participants in this regard may be limited to active EI claimants only. 5.5 Impacts on Participants: Multivariate ModellingThe multivariate analysis conducted in this evaluation examines the preliminary incremental impacts of the PBPM interventions in terms of employment, job search35, earnings and income support use. The multivariate analysis controlled for antecedent differences in socio-demographic and background characteristics as well as the use of services such as counselling or an action plan between participants and the comparison group. It also attempted to control for possible self-selection bias (Heckman Correction Factor) — the possibility that the same unobserved variables determining participation in the programs may contribute to the outcomes. Any remaining advantage for participants could then be attributed to participation in the PBPMs. See Appendix E (under separate cover) for the supporting analysis. As noted previously, this formative evaluation provides preliminary evidence on the success of the PBPMs. In-depth analysis and assessment of the success of the PBPMs requires data gathered over a longer period of time and the application of higher level analysis than was possible during the formative evaluation. 5.6 Impacts on Employers and CommunitiesFocus group participants (employers and community partners) agreed that the PBPMs have had a positive impact on employers, individuals and communities. Some employers stated that Wage Subsidies have enabled them to hire employees. In some instances, the fact that employers were able to hire new employees permitted them to expand their businesses and serve outlying communities. Furthermore, employers commented that they have been able to improve the quality of services they deliver to their own clients as they are able to provide training to new employees, which results in a more skilled workforce and better job performance. 5.7 Program Features of PBPMs Associated with Positive Client OutcomesStaff indicated a number of features associated with positive client outcomes. They included:
Finally, staff suggested that longer-term funding for contracts with third-party delivery agents to develop longer-term programs might be important to ensure the success of clients with multiple barriers to employment who may need more time to complete their training or acquire the necessary skills to find and maintain employment. Third-party agents delivering PBPMs identified many characteristics or program components associated with positive outcomes. They included:
Community partners generally stated a preference for service delivery models that rely on partnerships between the government and the community. These include:
Finally, some employers made suggestions to ensure positive client outcomes:
5.8 Unintended ImpactsFocus group participants (Employment Centre staff, third-party delivery agents, community partners and employers) were asked if they had observed any unintended impacts, positive or negative, related to the employment programs. Most respondents could not comment on this issue. Some staff, however, noted there was a negative impact on non-EI eligible unemployed Manitobans who they feel are being neglected due to their ineligibility for benefits under the LMDA (e.g., benefits such as Enhanced Fee Payer which are only accessible to EI clients as per the EI Act). Staff also felt they do not have the time to adequately address the needs of non-EI eligible unemployed Manitobans who often need more counselling. Another unintended impact mentioned was a consequence of providing training cheques directly to clients, which apparently resulted in some students spending the training money for other purposes than their tuition fees and then leaving their educational program. The educational institutions would thus prefer that the money be sent to the school directly. Many third-party agents noted that the programs have had positive impacts on the lives of people in a way that cannot be reflected in the current accountability framework. Clients develop a different perspective about what work is all about and a new attitude toward roles and responsibilities and these are important changes that affect all aspects of their lives. They suggested that the outcome measures should be broadened to capture all the medium and longer-term impacts on client groups.
| |||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060126234159im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060126234159im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060126234159im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060126234159im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}