|
|
     |
||
 |
     |
 |
|
Targets have been established for the LMDA in terms of the number of participants, returns to work (RTW) by participants, and unpaid EI as a result of participants RTW before they have used up their EI Part I entitlement. HRIB has the mandate to assess achievement against these targets. Participants are identified in a number of ways:
Return to work is assessed in two ways. For those who are active claimants in receipt of EI Part I, a systems-approach is used. A RTW is recorded if the claimant has 12 consecutive weeks (or the remaining weeks on claim if less than 12) where EI benefits paid are 25% or less of entitlement. For those who are reach-back clients or who are active claimants and do not RTW before the end of the EI entitlement, a survey is conducted 12 weeks after the end of the EBSM. If the individual is working at the time of the survey, a RTW is recorded. Unpaid EI is estimated for those active claimants who RTW as total EI Part I entitlement minus EI Part I entitlement paid. The evaluation has identified a number of problems with the way the LMDA's success is being measured and reported. These topics are discussed in this chapter. A. Achievement of Short-Term Targets Exhibit 9 outlines the short-term targets established for the LMDA for fiscal year 1997-98.
HRIB identifies 52,857 participants, 23,656 RTW and $93.8 million in unpaid EI. These represent short-term successes which occurred in 1997-98 but not necessarily for the LMDA. Our study evaluates the LMDA. As a result, we focus on participants under the LMDA and the successes represented by those participants. Exhibit 10 identifies the short-term success that has been achieved by the LMDA between April 25, 1997 and March 31, 1998 according to HRIB data. These data are derived by restricting 1997-98 HRIB participant and results data to the participants and EBSMs after the April 25 start date of the LMDA in B.C. Information on mutual clients was determined by comparing HRIB participant data with MHR IA data. The difference between HRIB's reporting of RTW and unpaid EI success for 1997-98 and success for the LMDA is accounted for by clients who pre-date the LMDA. In future years, annual assessments of success will deal exclusively with LMDA participants. However, there will remain a lag between participants in a given year and the results for those participants. In any one year, meaningful comparisons between reported participants, RTW and unpaid EI will be problematic if there are:
It is unlikely that there will be sufficient stability in these variables across years to make meaningful comparisons within a single year. The following sections assess the results identified by the HRIB data against the findings from the evaluation. 1. Participants Served HRIB has a mandate to track the number of participants. We observed anomalies between various HRIB data files-each intended to represent participants in 1997-98. The differences between the high and low numbers of participants identified was 8,000. Twelve per cent of those surveyed by the evaluation could not be found on HRIB's "official" participant file for 1997-98 despite being on an earlier HRIB participant file used for our sample and despite confirming their participation in our survey. Data from other HRDC data systems (Financial Administration System and Online Information System) identified 35% more participants taking comparable interventions and 130% more apprentices. Thus, we expect HRIB participant numbers to under-estimate actual participants. Another anomaly, we identified 33% of non-apprentice participants as reach-back clients based on their eligibility at the time they began their last intervention. HRIB identifies the percentage of reach-back clients at 19% for 1997-98. Our review of HRIB data for participants after April 25, 1997 identifies 4,809 reach-back clients or 10% for all interventions (15% for long-term interventions). The difference is not attributable to apprentices who are few in number and, depending on the definition used to identify them, average about 45% reach-back. (Note that all apprentices should be active claimants to count toward the LMDA targets.) The difference is attributable to the definition of "active". Once an active claimant the participant remains active so long as they participate in an EBSM. If they exhaust their EI eligibility, the active claim is extended to cover the period of the EBSM. So long as the individual has an active claim, he or she is eligible for further EBSMs and the active claim is extended by taking further EBSMs, to a maximum of 2 years. We found 23% of claims were still active after EI Part I entitlement had been exhausted. The average length of extension was 14 weeks and maximum was 52 weeks. To identify the number of mutual clients, we used MHR data to identify whether participants were in receipt of IA at the start of their intervention. Total mutual clients were 6,927-fourteen per cent of all participants and less than one-third of the target set for mutual clients. The number of mutual clients may be under-estimated given our observations about under-reporting of participants. 2. Returns to Work The HRIB results file identified 9,745 RTW for the 48,427 participants who began their last intervention between April 25, 1997 and March 31, 1998. This is less than the 23,656 RTW reported by HRIB for 1997-98. Almost 60% of the reported RTW were for individuals who participated prior to the signing of the LMDA. The evaluation adopted two measures of RTW involving total and consecutive weeks of work. Within each definition it adopted least and most stringent definitions of these measures. We assessed returns to work using these most and least stringent decision rules for each definition for participants in our survey who had a minimum of three months in the post-EBSM period. Exhibit 11 outlines the percentage of participants returning to work following these definitions and decision rules by EBSM.
The LMDA achieved an average RTW across EBSMs which ranged from 46% to 58%, depending on the methods used to calculate RTW. Our survey covered non-apprentices in long-term interventions (EBSMs) only. Based on our survey results, we extrapolate to the population of non-apprentice clients as of March 31, 1998 expected to complete their EBSM by June 1998. This would provide a minimum of three months post-EBSM experience before our survey date, in keeping with our survey participants. Based on this population and the percentage of survey participants with a RTW, we estimate the RTW as of September 1998 for all participants under the LMDA as of March 31, 1998. Exhibit 12 provides the estimated RTW for non-apprentice participants under EBSMs. The HRIB methodology identified 9,745 RTW by March 31, 1998 for all participants between April 25, 1997 and March 31, 1998. Our estimates of 9,442 to 11,743 bracket this value. However, our measures:
Using HRIB-derived estimates for RTW by apprentices, Group Session and Counselling Session clients with our own estimates for other participants, there may be between 14,377 and 16,678 RTW by all LMDA participants. We also note that participant counts are under-reported. RTW by those participants who have not be counted are not included in this potential total. 3. Unpaid EI Benefits The HRIB data for participants under the LMDA suggest that $42.3 million in EI Part I benefits were not paid. This is less than the $93.8 million estimated by HRIB for 1997-98. This difference is attributable to carry-over clients who did not participate in the LMDA. To assess the unpaid EI for participants, we identify the number of current claimants who have a RTW. We then estimate for each their total EI entitlement minus the actual EI paid under their original claim from administrative sources. The difference is the unpaid EI benefit associated with a RTW but not necessarily attributable to a RTW. (More will be said about attribution in a later section.) This unpaid EI definition conforms to the definition used in LMDA evaluations in all jurisdictions. The total unpaid EI benefit for non-apprentices based on this methodology ranges from $21.1 million to $23.2 million. This is approximately one-half of the unpaid EI for LMDA participants estimated by HRIB. Our estimate will exclude the unpaid EI of apprentices who, after taking the in-school training, return to their employer. The HRIB estimate includes $12.5 million in unpaid EI for apprentices. The evaluation method also does not include the unpaid EI for the 2,814 Group Services participants who return to work according to HRIB or the estimated 567 Counselling Sessions clients who may RTW. Applying the average unpaid EI estimated by the evaluation to these latter groups may add between $6.7 and $7.6 million to the total estimated above. The total across non-apprentice participants in long-term interventions, apprentices, and participants in short-term interventions may be from $40.3 million to $43.3 million B. Case by case analysis of the HRIB and Evaluation assessment of short-term measures Our survey identified 902 participants who had enough post-intervention experience at the time of our survey to assess their RTW status and unpaid benefit values. Only 794 of these participants could be identified on the December 1998 HRIB participant file. The remaining 108 could not be found despite being on an earlier HRIB data file used for our profile and despite confirming their participation in our survey. (This is another indication that HRIB may not capture information on all participants under the LMDA.) To confirm that the 794 matched participants had the same last intervention in both sets of data, we excluded cases with different EBSMs (none removed) or different start or end dates by more than seven days (225 removed) or no dates in HRIB results data (23 removed). We compared cases where HRIB did and the evaluation did not identify a RTW and alternatively cases where the evaluation did and HRIB did not identify a RTW. The proportions are about the same for reach-back clients for both methods. Both methods use a survey and appear to have a similar "error" rate according to the other's result. For active claimants, the evaluation identifies RTW for between 33% to 41% of those HRIB did not. This may be because the evaluation tracks clients over a longer period. Assessments of RTW by HRIB for active claimants are rejected by the evaluation in about 8% to 9% of cases. Note that matched individuals are those identified by HRIB to have participated under the EBSMs. We cannot speak to the RTW experience of those individuals who appear to participate under EBSMs, but who do not appear to be counted by HRIB. Our evaluation methodology identified more RTW for active claimants but a slightly lower unpaid EI ($3,129 versus $3,470 by HRIB) per RTW. Some of the difference may be HRIB's 12-week 25% rule. The unpaid EI formula used by HRIB will ignore up to 25% of EI entitlement paid to the participant during the 12-week period used to calculate RTW. Our evaluation would count this amount producing a lower unpaid EI. More will be said about the 12-week 25% rule in the following section. C. Appropriateness of Short-term Success Indicators and Methods to Calculate Them Short-term success indicators (or targets) were established for the LMDA based on:
Earlier sections have identified the findings of the evaluation relative to these targets. This section discusses the appropriateness of the targets themselves and the methods used to calculate them. 1. Problems With the Measures of Success Targets provide three key measures related to the success of the LMDA. These measures are intended to be indicators only. Their value as indicators should be based on their ability to provide unbiased insights into the likely success of the LMDA. A significant portion of the targets represent the activities of apprentices who are provided income support through EI Part I to attend the in-school training component of their apprenticeship. These individuals may not be unemployed under a strict definition of the eligibility criteria for EBSMs and are not unemployed under a common sense definition of unemployment. When they continue the on-the-job component of their apprenticeship, their return to their employer is not equivalent to an unemployed worker's RTW. Similarly, there is no equivalent to the unpaid EI benefit for an unemployed worker since there is no intention that the remaining EI benefit will ever be paid. As a result, to the extent that the targets include apprentices, they do not represent real impacts of the LMDA. They have a built-in bias to over-state the potential impacts. It should be noted that unpaid EI benefits represent a reduction in the expenditure of EI Part I only. To achieve this reduction, an expenditure under EI Part II is being made. The expenditure under EI Part II may be less, equal to, or more than any entitlement under EI Part I. Viewed from the context of the broader EI Fund and not simply EI Part I, the unpaid EI benefit measure does not necessarily show a reduction in the amount paid to the client. Its name may be a cause of some confusion. This is illustrated in the following example for TWS. Say an individual starts a TWS with 20 weeks remaining on their claim and a benefit rate of $300. The TWS is for 30 weeks at 40 hours per week and at a wage rate of $10 per hour. The program pays a 50% subsidy. (TWS can be for up to one and one-half years and up to a 60% benefit rate.) If the individual works at the end of the TWS or at the time of the 12-week follow-up a RTW is credited to the individual. The unpaid EI is $6,000. However, the expenditure under EI Part II has been $6,000. The benefit paid the individual is the same-the only difference is which Part of the EI fund is used to draw the payment. The measure's stated purposes are to: indicate speed of RTW; to focus resources on those early in their claim period, and to compare current to past performance. (HRIB, Accountability Framework: Methods to Measure, September 23, 1998.) However, as the above example indicates, the method of calculation distorts use of the measure for these purposes. These purposes would be better served by a measure of the remaining weeks of claim when the individual returns to work. In the example noted above a "remaining weeks" indicator would be zero. 2. Problems With the Measurement of Success The Human Resource Investment Branch (HRIB) is responsible for measurement of achievement against these short-term success indicators. HRIB has recently revised its methods of measurement and its algorithms to measure success. However, issues remain regarding the measurement of short term success. Not withstanding the inclusion of apprentices in these measurements we note the following issues related to the measurement against these targets. Current Claimants The method to calculate RTW for active claimants is referred to as the 12-week 25% rule. Under this rule administrative data on EI paid for the individual on claim are used to infer a RTW. A RTW is recorded for an active claimant (with an action plan or group session entered in NES, or if an apprentice entered in Benefits and Over-Payments (BNOP)) with 12 consecutive weeks (or the remaining weeks on claim if less than 12) where EI benefits paid are 25% or less of entitlement. The calculation assumes the "trigger" under BNOP has been pulled. This trigger deducts EI dollar for dollar if earnings exceed 25% of the benefit rate (or exceed $50 when the benefit rate is below $200). However, there is no recognition that a benefit rate of less than 25% does not imply earnings and does not mean that the individual has returned to work. (There are many reasons why a benefit will not be paid other than reported earnings from work. These are discussed in a later section.) The result is a potential over estimation of RTW for current claimants. If a RTW is inferred by this formula, unpaid EI is then calculated as the total EI entitlement minus the EI paid up to this point in time. Although the RTW calculation acknowledges EI paid over the 12 weeks of up to 25% of entitlement, any EI paid in this period is not deducted. The result is a potential overestimate of unpaid EI benefits. Also the calculation includes EI not paid before the RTW, regardless of the reason.The calculation of unpaid EI savings is a point estimate only. The individual could go back on claim. The result is a potential over estimation of unpaid EI benefits. To overcome this potential HRIB adjusts its unpaid EI estimate at a later date based on the complete claim history. However, it continues to count unpaid EI before the RTW and exclude EI paid during the EI period. Reach-Back Clients and Current Claimants After the Claim Period Reach-back clients do not receive support under EI Part I. As a result their return to work cannot be assessed using the method outlined for current claimants. Instead a follow-up survey, 12 weeks after the intervention has ended, is used to capture data on whether the individual is working. A follow-up survey is also used in the case of current claimants who are not identified as having returned to work prior to the end of their claim. To the extent that this follow-up is not made, the method will under-report RTW. However, if a follow-up is conducted, the answer to the survey question "Are you working?" may not be equivalent to "Have you returned to work?" The survey is valid only at a point-in-time. It has the potential to under or over report actual RTW. Because reach-back clients do not receive EI Part I, there is no calculation of unpaid EI benefits. Problems Related to Targeted Wage Subsidy The method formerly used by HRIB identified an individual as having a RTW at a point 12 weeks after the start of their TWS, even though they continue on a subsidized work term. The approach has been changed to record a RTW if the case worker identifies the person being employed at the end of their TWS or if they are employed at the end of the 12-week follow-up. This approach suffers the same problems as the point-in-time estimate outlined above. In the past, the formula HRIB used to calculate unpaid EI benefits to TWS clients could estimate a greater unpaid EI benefit than an individual's EI entitlement. The problem was a result of the individual's entitlement being extended under the TWS. The formula has been changed to eliminate this possibility. For TWS clients identified as having returned to work, it now calculates unpaid EI after the point where the participant starts the TWS. This will avoid counting unpaid EI which is in the period prior to the TWS. However, the potential to capture unpaid EI before the intervention (non-attributable) exists with the formula used for all other EBSMs. D. Alternate Measures of Short-Term Success Earlier sections have identified problems related to the way HRIB success measures are calculated, especially for active claimants. The evaluation explored alternate methods to calculate RTW and unpaid EI for active claimants. (Focusing on active claimants, this reduced our effective sample size to 344 cases in the following analysis.) We refine the calculation of HRIB return to work and unpaid EI measures by using more of the data available from the administrative systems. For the RTW calculation, the alternate measure uses:
The alternate unpaid EI measure looks at EI remaining on the claim and compares it to EI actually paid after the start of the RTW. This is an improvement over HRIB's approach which:
Examples will more clearly demonstrate the difference between the HRIB and alternate unpaid EI measures. Exhibit 13 presents the hypothetical example of four individuals (A, B, C and D), all with 24 weeks of entitlement and a $400 weekly benefit, an intervention in week 1, but with different weekly benefits paid over their claim. Earnings (not shown) are the cause of the reduced weekly benefits resulting in a RTW calculated in week 13 for individual A, B and C and week 1 for individual D under the HRIB and alternate methods.
The unpaid EI calculated for each individual and method is shown in Exhibit 14. The alternate measure's formula subtracts EI benefits paid after the start of the RTW from the EI remaining at the start of the RTW. The alternate measure focuses only on that period over which an attributable impact can occur. This is not the case for the HRIB approach.
The alternate method gives the true unpaid EI in each case. The HRIB method calculates:
Not illustrated in the example, the HRIB method can also calculate too much unpaid EI by including EI which has lapsed. Again, the alternate measure will correctly measure the unpaid component of this benefit. 3. Comparison of HRIB and Alternate Success for Participants Exhibit 15 compares the HRIB and alternate approaches for the 344 active claimants in our sample. Both the HRIB and alternate approaches identify RTW for the same 51 cases. Unpaid EI is similar under both methods.
HRIB identifies 42 RTWs that are not identified by the alternate approach. Here the HRIB approach attributes a period where benefits are not paid as work time. Unpaid benefits are low for this group, likely as a result of benefits continuing to be paid after the RTW was identified. In one example reviewed, an individual had 84 weeks of severance pay. In week 60, the individual started a five-week EAS. After the EAS, the 12-week 25% rule counted benefits not paid (because of severance pay) as work and identified a RTW at the end of 12 weeks. The individual continued to use up severance pay, then went on to receive EI Part I until the claim terminated at week 104. HRIB calculated the lapsed EI as an $18,585 unpaid EI. The alternate method identifies 98 cases as individuals having RTW where HRIB does not. On inspection, this difference is a result of two causes:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Previous Page]](/web/20060127005511im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127005511im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127005511im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}