|
|
     |
||
 |
     |
 |
|
Formative Evaluation of the Canada/Alberta Labour Market Development AgreementFinal Report
Evaluation and Data Development
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060126233815im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif) |
This summary highlights key findings from the formative evaluation of the Canada/ Alberta Labour Market Development Agreement (LMDA). The evaluation focuses on the first year of the LMDA (November 1, 1997 to October 31, 1998) and was conducted between October 1998 and March 1999. It should be noted that findings are for a period shortly after implementation of a substantially new set of Alberta Programs and Services (APSs) and delivery methods.
Results of the formative evaluation are predominately positive. This overall conclusion is significant given the extent and magnitude of changes resulting from the LMDA. The formative evaluation did find and does report some areas for review by management. It is our understanding that these problems were known and are being addressed by management.
Findings are based on data collected through multiple sources:
Data were analysed using a broad range of techniques. Qualitative evidence was synthesized in the form of a series of 11 case studies looking at the overall Alberta LMDA, the LMDA from the perspective of four sites (Edmonton, Calgary, Lethbridge and Slave Lake) and by specific APS. Quantitative data were analysed using univariate, bi-variate and regression methods.
Key findings are presented for the following issues of the evaluation.
This section deals with the relevance of the LMDA to the priorities of the federal and provincial governments. More detail may be found in Chapter 2.
Delivery aspects are discussed below and in Chapter 3.
Process outcomes are discussed below. More detail may be found in Chapter 4.
A number of participant impacts are identified in the evaluation. These are summarized below and detailed in Chapter 5.
Community impacts are beginning to be felt. These are discussed below and in Chapter 6.
Alberta has lived up to the spirit and intent of the LMDA.
The LMDA represents a major undertaking. It has resulted in an infrastructure for the delivery of all APSs, and French language programming and services in particular, which outstrips the tentative plans of the LMDA. Despite the magnitude of the changes, participants continued to receive programs and services and to rate highly their APSs and the staff involved. Positive impacts have occurred for individuals and are beginning to occur for communities. In its first year, the LMDA achieved much of the short-term success targets of the LMDA.
As can be expected with any set of programs and delivery mechanisms as "new" as the LMDA in Alberta, the formative evaluation did find areas for further consideration by management. Many of these are currently under review or revision.
|
The formative evaluation of the Canada/Alberta Labour Market Development Agreement (LMDA) was designed to provide an evaluation of the design, delivery, implementation and outcomes of the provincial programs supported under this Agreement during its first year of operation. The requirement to conduct a formative evaluation is detailed in the Agreement under Section 8.1.
The evaluation focused on several key issues including: a) relevance and design, b) delivery, c) process outcomes, d) impacts on individuals, e) impacts on communities, and f) measures of short-term success. Data were collected through telephone interviews and focus groups with participants, key informant interviews with federal and provincial staff and several third part delivery agents, and reviews of administrative data.
Overall, the results of the formative evaluation were positive. LMDA clients were very satisfied with the Alberta Programs and Services they received and the staff that served them. The evaluation noted, and we concur, that the infrastructure established to deliver the Alberta Programs and Services exceeded the requirements of the Agreement. The evaluation also noted that in its first year the LMDA achieved many of the short-term success targets of the LMDA.
Issues of concern raised during the evaluation which have been, or will be, addressed include:
The evaluation notes significant problems with the measurement of "success" in the calculation of savings data by the Human Resources Investment Branch of HRDC in Ottawa. Since the report, the methodology used by HRIB has been adjusted.
|
This report presents findings from the formative evaluation of the Canada/Alberta Labour Market Development Agreement (LMDA) in the form of an Overview Report. Other findings reports in this series include a Technical Report, an Evaluation Brief, Recommendations and Lessons Learned.
The governments of Alberta and Canada entered into a joint agreement (the first for any jurisdiction) on December 6, 1996 with regard to the provincial management of labour market development programs in Alberta. Each agreed to work together in the implementation of Alberta Programs and Services (APSs) and in the operation of the National Employment Services (NES). The LMDA came into effect on November 1, 1997. Delivery is through staff of Alberta's Advanced Education and Career Development (AECD)2 department or third party service providers, using funding from the EI (Part II) Account of Human Resource Development Canada (HRDC).
The APSs under the LMDA are to assist individuals to prepare for, obtain and maintain employment resulting in savings to the Employment Insurance (EI) Account. In so doing, a return on investment will be achieved through a reduction in dependency on Employment Insurance (EI) and in social assistance (in the form of Support for Independence [SFI] benefits, through Alberta Family and Social Services [AFSS]), and additional tax revenues through increased employment.
The key principles of the Canada/Alberta LMDA include:
There are seven APSs under the LMDA, which are the subject of this evaluation:3
Eligibility under the LMDA is restricted to unemployed individuals pursuant to the Employment Insurance Act (without employment and actively seeking employment). To be eligible for participation in one of the above-mentioned programs, individuals must fall into one of the following groups:
Apprentices who obtain a layoff of convenience from their employers to attend the in-school component of their apprenticeship training, and who qualify as active claimants are also considered participants under the LMDA.
In addition, unemployed individuals who are not eligible for funding support through EI Part II may receive services under employment support measures (CEAS and JP). Uninsured individuals may also benefit indirectly through community partnership activities supported under LLMP. These "uninsured individuals" (including Supports for Independence recipients and "other Albertans" not in receipt of SFI), are not included in the targets for the LMDA nor in the findings of the formative evaluation.
Alberta and Canada have set up a Results-Based Framework to measure achievement of success. Short-term results are based on performance measures. These track:
Targets are established for fiscal years based on these measures. For 1997-98, established targets are:
In the medium and longer term, other measures to be assessed include:
A number of factors, many of which pre-date the implementation of the LMDA, have had a bearing on its development:
As part of the LMDA, Alberta and Canada have agreed to cooperate in the development and implementation of a three-phased evaluation process. The first phase resulted in the document Logic Models and Evaluation Issues, Questions, Indicators and Data Sources.
There are two subsequent phases—a formative phase (the subject of this report) and a summative phase. The formative evaluation began in October 1998 and provides information to enhance the design and delivery of the LMDA and individual APSs. The focus of the summative evaluation will be on measuring the outcomes, impacts and cost-effectiveness of the APSs. Both of these phases use multiple lines of evidence—both qualitative and quantitative.
The formative evaluation includes three telephone surveys involving non-apprentice participants (997), comparison group members (426) and apprentices (200). Other data gathering activities include document review and review of secondary data, 61 key informant interviews (involving management, staff and community partners) and 11 focus groups (involving third party delivery agents, employers and participants). In total, 128 individuals were contacted.
As comprehensive as these data collection methods and the analysis they support have been, readers are reminded that the formative evaluation represents the very early days of the LMDA in Alberta. Data collection and analysis represents a 15 to 17 month window (up to the early part of 1999) after implementation of a substantially new set of programs and delivery methods in Alberta. As such, the formative evaluation covers a period of significant transition. Transitions continue. We report information on activities conducted after this date that are intended to address concerns identified during our data collection period. However, we have no evidence of the efficacy of these activities.
This report is structured around the key issues of the evaluation:
Note that quotations from the qualitative research are italicized.
|
This chapter discusses the relevance of the Labour Market Development Agreement (LMDA) to the priorities of the federal and provincial governments. Further it discusses program design issues which may influence the achievement of these priorities. Information related to these issues is drawn from the qualitative activities.
The Alberta Programs and Services (APSs) were generally considered to be consistent with the intent of the EI legislation by all sources we contacted. For example, they are seen to be compatible with the following guidelines of the Employment Insurance Act:
Managers with Human Resources Development Canada (HRDC) and Alberta Advanced Education and Career Development (AECD) commented that the programs and services being delivered under the LMDA were meeting the basic spirit and intent outlined in Part II of the EI Act. However, most managers suggested that although large strides had been achieved, there were still areas of improvement required in order to reach all of the goals of the LMDA. Priorities set for the LMDA and the activities in support of them are outlined below:
The high degree of commitment and professionalism of those involved was a significant contribution to the success of the LMDA.
The relationship between AECD and the HRDC Regional Office was a strong intergovernmental partnership that many respondents considered to be exemplary. A number of problems and issues were successfully resolved through the continued work of staff in both departments on a daily basis. Successes included the transfer of staff from HRDC to AECD, the resolution of complex accommodation issues and the provision of adequate information to support AECD in meeting audit requirements early in the transition process.
HRDC managers commented on the integrity of AECD staff—how they had lived up to the spirit of the LMDA regarding such issues as language of choice, federal visibility, resources and evaluation. Overall, there was a good deal of faith in the relationship between the two organizations in Alberta and there was a general sense of respect on both sides.
Senior managers praised delivery staff for their commitment to the clients and the professionalism they had displayed despite the uncertainties for them personally created by the LMDA.
2.2 Design
This section explores LMDA design issues, particularly those related to the degree of stakeholder involvement in planning, French-language delivery arrangements, local flexibility in terms of program design and delivery, regional differences, and adjustment made to the LMDA since implementation.
a) Involvement of Internal Groups
The LMDA brought with it monumental changes. Senior managers underestimated the number of implementation tasks involved and the myriad of details associated with them. The senior managers at AECD and HRDC stated that, despite extensive planning and preparation in the six months leading up to implementation, project planning efforts had been inadequate and had resulted in many issues being dealt with after implementation, as they emerged and on an emergency basis.
Overall, senior managers were satisfied that some improvement or resolution had been achieved on a number of issues such as human resources, accommodations, and some systems problems. Interestingly, although a formal mechanism had been set up to resolve differences in the Agreement, issues tended to be addressed informally, and were often resolved in a pragmatic way. Despite a number of difficulties, the partnership between AECD and HRDC appeared to work well at the regional level.
Consistent comments across all sites indicated that staff were not adequately prepared for implementation of the LMDA and there was limited or no consultation with them prior to its initiation. On November 1, 1997 many staff either "crossed the street" or walked in the same door as the day before only to find a new, sometimes confusing and often frustrating work environment where the rules were developed to deal with emerging circumstances and then communicated by e-mail. Of all sites, Calgary had the most difficult adjustment and over a year later, some of the issues associated with the transition were still fresh in the minds of staff.
In spite of these factors, clients were served. We found little evidence of adverse client reactions and generally high ratings of satisfaction with staff and APSs.
b) Involvement of External Groups
There was no evidence of community consultation prior to the LMDA implementation although some consultation occurred afterwards (particularly in Calgary which set up a formal process to obtain input from smaller centres). Some delivery agents were involved in a limited way in terms of consultation prior to the LMDA implementation. No employers were consulted.
Local Labour Market Partnerships (LLMPs) were just beginning to be implemented at the end of the evaluation period. LLMPs help foster local partnerships with businesses, labour and other community representatives to ensure that flexible, innovative and responsive programs are designed and delivered. While it is still early, it appeared that LLMPs were becoming a useful way of ensuring program responsiveness to EI clients. The use of partnership was evident in other ways as well. For example, the SE program was being contracted out to community agencies with strong business and government links, such as Community Futures, Business Development and Economic Development. This appears to be an appropriate way to assist fledgling entrepreneurs by accessing the knowledge base and well-established networks in local communities.
After implementation, community consultations to determine program needs began to occur in a number of Regions. Two results of this process, identified in the case studies, were contracted francophone services in Edmonton and the expansion of bilingual services in Banff.
Linkages were also being developed among third party delivery agents. In some areas, the delivery agents had developed informal networks to discuss shared issues related to client services.
Partnerships, networking and community collaboration were evident in Alberta, even in the early stages of the development of the LMDA. These were already having a positive impact on program design.
c) Involvement of Non-EI Eligible Individuals
Eligibility for EI Part II funds for employment benefit programs is restricted to unemployed individuals who are EI clients at the time they start their LMDA activities: those with a current EI Part I claim (active claimants) and those eligible through reach-back provisions based on their former EI benefit status. Those who are not eligible for EI Part II funds may still benefit from services provided through CEAS and JP. Thus two other categories of unemployed individual have become involved to some extent: Supports for Independence (SFI) recipients and Other Albertans, a category which encompasses everyone else. Early feedback from key informants suggests that these latter two groups are beginning to benefit from the wider spectrum of services now offered as a result of the LMDA. Another potential impact is a lowering of average costs per participant through the sharing of overhead expenses.
LLMPs are also a mechanism through which the impacts of LMDA funding may indirectly impact those who are not eligible for Part II funding. LLMPs provide support to community partners to identify, develop, and implement labour force adjustment and human resource strategies. Non-EI clients may also benefit from these strategies.
The LMDA envisioned a network of delivery arrangements which would best meet LMDA requirements while making use of local infrastructure and available staff. Three types of configurations were envisioned:
Alberta's response to the provision of delivery arrangements far outstripped the tentative plans laid out in the LMDA. It appeared to embody the desire of management and staff to provide a broad, made in Alberta network of services across the province.
a) Co-located Services
The most comprehensive delivery model was the Canada-Alberta Service Centre (CASC) which would provide access to labour market programming, including both National Employment Service functions (assessment/service needs determination, employment counselling, labour exchange services for job vacancy information and labour market information) and APSs. In communities where the infrastructure and staff were available, access to both provincial and (remaining) federal labour market programs and services would be available through a common front end which would provide reception, screening, needs determination and information functions offered by both federal and provincial staff operating under common job descriptions. Supervision of the common front end would be conducted by AECD. Other services in this co-located work environment would be determined by the Region (e.g., in some cases Alberta Family and Social Services was co-located).
Originally, five locations were considered for this configuration. At the time of the evaluation there were a total of 18 CASCs.
Three co-located sites were developed previously as part of a provincial demonstration project. Fifteen more had to be created. The more than three-fold expansion of CASCs, beyond the original scope envisioned in the Agreement, demonstrated the very strong commitment of the Alberta Government to provide LMDA services as widely as possible.
b) Use of Alberta's Infrastructure
In 12 locations where AECD Career Development Centres (CDC) already existed and HRDC did not have a sufficient infrastructure to support the LMDA, the CDCs were used to provide access to federal and provincial programs and services through a single point of access. In addition, on April 1, 1999, the Youth Employment Services Centre in Edmonton (formerly co-located with HRDC) was closed and became instead a contracted service called Youth Connections, which was described as a CDC.
c) Contracted Services
As a third option, when neither Alberta nor Canada had a significant physical infrastructure or available staff, services were contracted with community-based delivery agents. At the time of the evaluation, eight of the ten locations (as identified in the Agreement) were using this service configuration, and at least some contracted services were being provided to an additional 14 communities. As a result, this approach doubled the capacity of the original plan, allowing the Regions to provide LMDA services more broadly.
With the implementation of the LMDA, a number of changes were made to the assessment process formerly used by HRDC.
Generally, the assessment process used for the LMDA was seen as a strength that Alberta had brought to the negotiating table. This process was expanded from its traditional SFI population to cover the assessment of all EI clients. Clients were sent through one of three levels of assessment to the service or program most appropriate for them, depending on the anticipated level of their interaction with the LMDA. These included Level 1—Initial Screening (mainly self-directed); Level 2—Service Needs Determination (limited advice and short-term services); Level 3—Workability Assessments (for all clients seeking funding for training or further education, and included needs assessment, goal setting and a return-to-work action plan, followed by financial eligibility determination and referral to an APS).
All stakeholders indicated that the screening and assessment processes were appropriate in most cases and were felt to be better than those offered in the past. In particular, satisfaction with screening was high on the part of participants in CEAS and TOJ who indicated that they had learned useful information about themselves as a result of the process. Feedback from third party agents indicated that better placements were occurring in SE programs.
A few concerns regarding assessment were identified:
While generally referrals to the APS appeared to be appropriate, delivery agents did report some inappropriate referrals and suggested that there had been an overall decrease in the number of referrals received. Although some employers indicated frustration with the number of participants requesting information interviews as part of the process, most were generally pleased about the match between clients and training positions.
A major issue was emerging just as the evaluation was being conducted. A policy change had been implemented that AECD program staff were to get out of the business of assessment and to contract it instead to third party agents. This change was spearheaded in Calgary where staffing shortages had already caused bottlenecks in the assessment process. It was much too soon to determine the effect of this change, but the initial staff response was negative.
Senior managers reported that providing adequate bilingual services in Alberta had been a concern prior to the LMDA. They also added that AECD had done a lot of work to ensure that these services were available, especially in areas of significant demand (those requiring services in French), according to the Official Languages Act. HRDC managers were unanimous in their praise for the way Alberta had addressed this issue—an example of how they are living up to things.
Francophone service is available in a number of locations across Alberta using a variety of delivery mechanisms:
Based on our review, French language services are available in a number of sites through bilingual staff and contracts involving third party service providers with French-language capabilities. Through our qualitative data collection activities, we identified one individual who had not been able to receive service in the language of their choice, at the time they wanted the service. (The bilingual staff person was already serving another client. Rather than wait, the client chose service in English.) In our quantitative survey, only two individuals identified requesting programs or services in French. Both reported no problem in accessing these programs or services in the language of their choice.
Our overall assessment is that Alberta had more than met its commitment to provide francophone services under the LMDA.
Staff views were mixed about the degree to which local decision making was fostered under the LMDA. On the one hand, administration of the LMDA in Alberta was perceived to cause a more bureaucratic system with more paperwork, more rigid funding and contracting guidelines, and greater centralization. On the other hand, there appeared to be more local authority for programming decisions which resulted in greater program flexibility.
Some senior managers believed that different Regional Directors had different approaches to expenditures and one suggested that field staff had more authority to make decisions than they were currently exercising. This lack of standardization was also evident to contractors who held contracts in more than one region.
We observed differences in the way APSs were delivered across regions. The first area of variation reflects differing treatments for several APSs, notably:
The expected level of financial contribution by participants toward the cost of their interventions was a second observed difference across areas. The requirement for a $1,350 contribution was treated differently by region.
In Edmonton, adherence to the requirement appeared to be strict. However, all SW participants (typically multi-barriered clients) were exempted from it. In Calgary, no APS received exemption but staff appeared to manage by exception through the use of the Special Circumstances category. In such cases, the $1,350 contribution could be waived.
Management at both the senior and site levels believed that regional responsiveness was essential in the implementation of LMDA programs and services. Broad program guidelines and encouragement for a flexible interpretation at the local level may account for these differences or adaptations across regions. On the one hand, these observations may reflect appropriate implementation and accommodations to meet the particular needs of the local level.
On the other hand, these observations may reflect inappropriate implementation and a lack of common vision across sites. Generally, despite the broad program descriptions in the program binder, we found interviewees could not make clear distinctions among APSs and a great deal of confusion existed.
Given the evidence, we are unable to assess whether differences, reflect appropriate implementation and adaptation to meet special needs, or inappropriate implementation and a lack of common vision. (Sample sizes for our survey of participants are insufficient to detect differences in success for clients across different program treatments.) You may wish to re-visit this issue in the summative evaluation.
Regardless of the reason, the marked regional differences create equity concerns. Participants may not have the same access to APSs or funds across regions.
There were relatively few problems identified related to the delivery of the LMDA, despite the significant changes resulting from its implementation. Much of this success is attributable to the strength of the delivery staff and degree of collaboration at all levels.
However, areas of concern do exist. These are documented below.
a) Lack of Clear Administrative Guidelines
Staff in the field were not adequately prepared for the transition—and transitions were significant. To an extent, staff felt that management occurred by e-mail as a management response was developed to deal with each emerging issue. Although it may have been unrealistic to believe that administrative guidelines could be created which anticipated all eventualities, staff felt that too many were created in reaction to events. Staff, who may have been accustomed to referring to a policy manual for direction, found reference to a string of e-mails both time-consuming and more difficult for them. They found keeping up with emerging developments and policies a problem.
b) Increased Contract Management
Implementation of the LMDA resulted in a dramatic increase in the volume of contracts to be managed and a significant increase in the administrative burden. For example, prior to the Agreement, the AECD office in Calgary had managed 15 training contracts a year and served approximately 6,000 clients, most of whom were receiving SFI. With the implementation of the Agreement, the office took over 120 contracts from HRDC and provided service to approximately 32,000 additional EI clients. Over time, the HRDC contracts began to run out and new contracts had to be developed according to both the new APS specifications and the Alberta government's Request for Proposal (RFP) process.
The RFP process was new to many contractors who had formerly worked with HRDC. It proved to be a significant change for them as well as for the staff who managed the process. At all case study sites, we received extensive feedback on the contracting process from both groups. Generally, staff and contractors agreed that the RFP process made the whole skills training system more accountable by making it more transparent, fair and open, as well as more market driven. However, staff saw two disadvantages to the process:
c) Contractor Concerns
Contractors had many concerns over contracting. Most were related to issues of accountability (see Chapter 3), particularly related to the deliverables or the fee-for-outcome method of payment, specified by the Alberta government in their contracts. They voiced concern that the ability to write a winning proposal did not necessarily imply delivery of good service to clients. They also worried that the process forced them to be competitors in a skills training world where collaboration was the norm and considered a professional commitment.
To balance these views, contractors also indicated that they found AECD staff approachable and believed that many of their concerns could be resolved through discussion.
d) Privacy and System Issues
Client confidentiality and privacy were being respected in the field, according to feedback received at the case study sites, but the unanticipated consequence of compliance was that information sharing was hampered and timely service hindered. Once the Agreement had been signed, a key issue related to privacy remained which proved to be a great irritant for staff. This related to access to the list of active claimants. While AECD staff wanted access to the list for early identification and strategic targeting of potential service users, HRDC was unable to provide it because of concerns for client confidentiality. With such a list, AECD staff or their third party service providers could contact eligible individuals and determine potential need for APSs. This proactive approach could potentially:
Third-party delivery agents, like AECD staff, do not have access to active claimant lists. As a result, they do not have information on potential clients for their services. However, the implications for them are potentially more significant as their contracts are based on deliverables. The inability to contact potential users may adversely impact achievement of their targets.
Concerns over confidentiality also prevented AECD staff or contractors providing information to HRDC Investigation and Control Officers to allow them to initiate potential disentitlement actions. Potential overpayments of EI benefits may have been a consequence.
Negotiations on information and data sharing re-opened soon after the LMDA was signed and were not resolved as of March 1999. Specific areas of negotiation include access to and interpretation of data on EI claimants, two-way sharing of results information or outcome data and a mechanism to notify HRDC about client refusals so that follow-up procedures can be implemented.
|
This chapter assesses issues of delivery of the Labour Market Development Agreement (LMDA) and Alberta Programs and Services (APSs).
The LMDA represents a major change in the delivery of programs and services in Alberta. This change had negative consequences for staff:
Despite the many issues experienced by staff, they managed to present a professional image. Feedback from participants, employers, delivery agents and managers about the service provided by Canada-Alberta Service Centre (CASC) staff was overwhelmingly positive. They were praised for their ability to cope with the many changes and uncertainties associated with the LMDA and their professionalism and ability to focus on client needs were commended.
Most Advanced Education and Career Development (AECD) senior managers indicated that the physical co-location of services was positive for clients, providing them with one-stop shopping and easy access to services. It was also suggested that co-location had allowed staff members to more quickly identify and respond to client needs, and had expedited the transfer of knowledge among staff members.
However, the topic of co-location highlighted a number of concerns.
Originally, Human Resources Development Canada (HRDC) and Alberta Family and Social Services (AFSS) had greater involvement in the CASC demonstration project than AECD, which was considered to be a minor partner. With the LMDA, the role of AECD expanded and the original CASC sites changed quickly. Barriers among staff groups began to rise and the new working arrangements were less satisfactory for staff originally involved in the CASC prototypes. These individuals found that the more rigid procedures under the LMDA lessened the easy collaboration they had experienced previously.
Although this perception by staff was wide-spread, the flexibility in the name of facilitation and client service that occurred under the demonstration project could not be continued under the permanent LMDA arrangements.
The development of co-located sites across Alberta proved to be (and continues to be) a very significant administrative challenge involving synchronizing leases and finding appropriate premises for significant numbers of staff—often 100 per site. Negotiations involved a number of different parties—representatives from AECD, HRDC (Alberta), HRDC (Ottawa), two departments of Public Works, and other partners such as AFSS or other government or community groups sharing the same facilities. It appears that insufficient resources had been set aside for this process and that the LMDA was not considered a priority by either Public Works organizations. One manager described co-location and getting all the arrangements in place as the most significant challenge of the last five years. Despite these challenges, another manager pointed out that Alberta's experience was being used to create templates for co-location in other provinces.
An associated issue had to do with the lease-dollar transfer arrangement between HRDC and AECD. As employees transferred to AECD, HRDC was to pay the lease-dollar savings to the province. The LMDA had included a notional dollar amount, as the actual number of staff transferring over had not been known at that time. As a result, a complex tracking process occurred by individual, site and square footage costs to calculate the lease-dollar amount. In retrospect, using a fixed amount per employee per year would have been more simple and straight forward.
Co-location at the CASC sites proved to be a difficult transition and planning activities did not adequately address the complex human resource issues involved with the melding of two distinct corporate cultures. The fact that the two entities were of such disproportionate size, in some sites, contributed to the sense of culture shock. Differences in terminology, including what constituted a client, and procedural differences took time to sort out.
Despite these issues, the growing pains related to co-location were mitigated by the commitment of both management and staff to the ideals embodied by the LMDA. Their commitment, vision, and determination to go that extra mile, at whatever personal cost, are what have made the transition possible. All sites indicated that clients have indeed benefited from the one-stop-shopping concept and that increased collaboration between AECD and HRDC, while not perfect, is definitely happening.
The Agreement specified that Alberta would give Canada, as well as Alberta, public recognition in all LMDA activities through such means as signage, letterhead, brochures, reports and advertising. In addition, the Canadian flag was to be prominently displayed at all CASC sites, and that the automated kiosks which provided labour market information would have appropriate signage to indicate that they were provided by HRDC. In all of the case study CASC sites, joint visibility was very evident and it appeared that Alberta had met this requirement. However, the AECD offices (e.g., Edmonton/Northeast Regional Office, Calgary/Central Regional Office, Calgary Labour Market Information Committee [LMIC]) had limited or no exterior signage.
In other respects, public information was much less evident possibly as a result of the reluctance of the Alberta government to advertise and to be seen to be competing with the private sector. Advertising guidelines were still under development at the time of the evaluation. All sites visited indicated that they had not directly marketed their programs for several years, as a result. A likely consequence, community awareness of programs and services, was deemed low by all groups interviewed as part of our qualitative data collection.
Feedback by key informants on public awareness of LMDA programs and services was uniformly negative. Stakeholders at all sites believed there was limited awareness and it was generally felt that the communities were not well informed of the opportunities available to them. EI clients and employers frequently indicated that they had heard about the program or service they were using through word of mouth. Further, lack of information appeared to result in more inappropriate referrals due to confusion about agency mandates. Delivery agents and clients also indicated that some clients were given the run around due to a general lack of awareness on the part of CASC staff. (We do not know if these perceptions have changed over time or reflect an early view.)
Additional evidence on awareness comes from the survey of participants and comparison group members. The latter are non-participants whose characteristics are statistically similar to participants.
Of the individuals who confirmed participation in an APS, one-half (50 percent) obtained information on their APS from a government office. Five percent obtained such information from a combination of a government office and third party service provider, while 10 percent obtained APS information from a third party service provider only. Seventeen percent heard about APSs by word-of-mouth and 10 percent saw or heard of them through an advertisement. Most participants who found out about an APS seemed to know where to go to get information instead of having information provided to them through advertising or by word-of-mouth.
Forty-six percent of comparison group members had not heard of any of the APSs listed for them in our survey. Comparison group members were selected to be statistically similar to participants in characteristics other than participation. Forty-one percent of those selected to be similar to active claimants and 53 percent of those selected to be similar to reach-back clients were not aware of any APSs. This is a significant finding, especially related to the low level of awareness by active claimants.
Awareness by comparison group members who are aware is highest for the Apprenticeship Training Programs (52 percent), TOJ (50 percent), JP (47 percent), CEAS (41 percent) and SDU (40 percent). Awareness of programs by comparison group members was lower for SE (25 percent), SW (20 percent) and SDS (9 percent). We did not ask questions to determine the source of the information used by those who were aware.
All those interviewed through our qualitative research agreed that visible and consistent program marketing must occur. Issues identified for further work related to signage, the representation of the HRDC-AECD partnership, and third party delivery agents and specific program and service advertising.
A provincial committee is looking into these issues. However, it must be noted that overworked program staff were worried that if it (advertising) happens, they will come and they feared being overwhelmed by the demand.
Program and service accessibility appeared to be adequate at all sites based on qualitative findings. No negative client feedback was received. Issues by those interviewed related more to lack of signage (Calgary) or poor community visibility (Slave Lake). Given the difficulty related to leasing co-location premises, the lack of negative feedback on accessibility was viewed by some as a success.
Most confirmed participants in our surveys (60 percent) report accessing their APSs through a government office. Another 38 percent accessed them through a third party delivery agent such as a community organization or training institution.
Most participants (77 percent) say they have not experienced any difficulties accessing employment programs or services since November 1, 1997. Five percent were unsure whether or not they had had such difficulties.
Eighteen percent of participants experienced difficulties. The most frequently reported barrier was financial (5 percent of all participants)—including lack of funding or cut backs, as well as a potential loss of benefits. An equal number of participants (2 percent each) identified:
Program duration was adequate for most participants in our focus groups but a number mentioned that the material was too condensed or that course length did not match the duration of their EI funding. (Participants could apply for support under EI Part II if needed.) In Lethbridge, staff indicated that the length of exposure courses should be more flexible, depending on community needs. Employers felt that TOJ duration should be more individualized, depending on both client and employer needs. No participant in our survey identified similar issues related to their accessing APSs.
All of the CASC sites which were visited during the evaluation provided adequate access for persons with disabilities. No site reported receiving negative feedback on this topic. Most CASCs had excellent access and parking, although the very busy Argyll CASC in Edmonton had some limitations in terms of access to transportation and congested parking.
As already noted, AECD established a network of supply arrangements for service in French. Access is available through a number of means, although we found little evidence of demand for service in French.
Some comparison group members have participated in an APS. (These would be participants after October 31, 1998.) Most frequent APSs participated in were CEAS (13 percent), TOJ (11 percent), JP (9 percent) and SDS (7 percent). The apprenticeship training program was identified by 12 percent of the comparison group.
Comparison group members who are aware but who did not take up an APS were asked to identify their reasons for not participating. Forty-seven percent report that they did not access the APS because they found employment instead. Fifteen percent report not knowing enough about them, while 10 percent did not feel that the APSs would be of benefit to them. Difficulty accessing and too much cost or not receiving financial help were identified by less than 4 percent each.
One of the principles set out to guide the implementation of the LMDA was excellence in client service. The concept of client service involved a number of components including consistency, one-stop shopping, service quality, timeliness, equitable access and affordability. Findings related to service excellence are presented in this section.
The use of consistent rules across clients and across sites was seen as an important attribute of excellent client service. At the same time, program guidelines were purposively broad and flexibility in interpretation was encouraged. Achieving harmony between these two potentially contradictory principles is a difficult undertaking.
Achieving equity would suggest consistent treatment for those who face similar circumstances. However, the application of consistent rules across two locations with different labour market circumstances or client characteristics may result in inequities for those in the disadvantaged location. Flexibility may be a means to redress inequities created by local circumstances.
Overall consistency appears to be the norm. We found limited examples of inconsistencies. Feedback from participants and delivery agents suggested that some variations in programs and some inconsistencies in practices, particularly in the area of funding, existed across regions.
These inconsistencies do not appear to be directed at reducing inequities caused by local circumstances. As a result, the few cases of inconsistency do not appear to be explained by the goal of increasing equity through local flexibility. You may wish to re-visit this issue in the summative evaluation.
A touchstone for all field staff was the concept of one-stop shopping, sometimes referred to as one window access. The LMDA was seen to offer easy access to services, quick identification of client need, a timely response to that need and a quick return to the workforce. The goal was to provide better service; to stop clients from being referred between AECD and HRDC offices (and in some cases, AFSS offices as well); and to harmonize labour market training in the community. Combined services also offered the possibility of resulting efficiencies and economies.
For the most part, CASC sites did offer one-stop shopping. Operational issues, identified above, worked against the potential outcomes of efficiency and economy (at least in the early stages of implementation). Staff generally felt that the operation was moving in the right direction and that more and better service was being offered to more Albertans than had been possible before.
However, in Calgary, staff were concerned that the entry process for clients was confusing, long and complicated as they had to move among a number of points of contact to obtain approvals and get assessed. They suggested that it was not one-stop shopping. No other site registered this concern. Results from our survey of participants did not support this concern. Participants from Calgary were no more likely to identify difficulties accessing APSs.
High quality is another goal related to service excellence. Findings on service quality are drawn from our qualitative and quantitative data collection methods.
In our survey, we asked participants to rate the service quality of the staff they were in contact with. Those who responded may not have been able to distinguish between government staff and third party delivery agents. As a result, data from the survey may include views on the service quality of both groups.
Participants from our survey, who have ended their APSs, rated the staff members they were in contact with while taking their APSs on the quality of their services. They rated these staff on a scale of one to seven. On this scale, one means "poor" and seven was "excellent." Although there was not much variation across APSs in the rating of staff, SDU participants consistently rated the staff they dealt with highest and TOJ participants rated their staff lowest.
a) Courtesy
Across all sites, participants in our focus groups commented positively on the courtesy with which they had been treated at the CASC sites.
In our survey, participants rated the courtesy of staff high at 6.1 on a seven-point scale. Ratings were high across all regions (the highest in the south at 6.3). Perceptions of courtesy increased slightly the more recent the completion date of the APS—from 6.0 for those who completed more than one year ago, to 6.1 for those completing between six and twelve months earlier, to 6.2 for those who completed within the last six months.
b) Knowledge
The average score related to the knowledge of staff was 5.9. Perceptions of knowledge increased the more recent the completion date of the APS—from 5.6 for those who completed more than one year ago, to 5.7 for those completing between six and twelve months earlier, to 6.1 for those who completed within the last six months. By region, the perceived level of knowledge of staff varied slightly, from 5.8 in Edmonton to 6.0 in Calgary. The average score in Edmonton may potentially reflect the initial impact of job rotations which would reduce the average time of all staff in a position. In the longer run, job rotations may result in a broader base of knowledge of staff and potentially higher average scores.
c) Efficiency
Participants assessed the efficiency of staff at 5.8 on a seven-point scale. Perceived efficiency was slightly lower initially, at 5.6 for those who completed their APS more than one year ago.
d) Empathy
Participant feedback in focus groups indicated that staff were generally empathetic and understanding. Further, contractors indicated that staff were committed, professional, approachable, collaborative and usually prompt in their dealings with them. This was borne out by the fact that despite the many contracting concerns identified by the agents, they still felt that AECD staff had done a good job.
Our survey provides further evidence. The question asked participants to rate staff on their "ability to understand your needs." The average score was 5.4. Staff in the north had the highest score at 5.8. Over time, perceptions related to empathy declined—from 6.1 for those who completed more than one year ago, to 5.5 for those completing between six and twelve months earlier, to 5.3 for those who completed within the last six months.
e) Overall Quality of Service
Our survey asked participants to comment on the overall quality of service received from staff. The average score, on our seven-point scale was 5.8. This value ranged from 5.6 for those in Calgary to 6.0 for those in the north. Perceptions of overall service quality were lower initially, at 5.3 for those whose APS had ended more than one year earlier. Those completing their APS less than six months before the survey rated overall quality at 5.8. Those whose APS ended between these two time periods rated the overall quality of service at 5.9.
Staff members raised a number of concerns about timeliness. These reflect problems which:
Staff concerns related to delays in access appear to be supported by other evidence on the lack of awareness from our focus groups and comparison group survey. However, concerns over delays in service once requested were not supported by evidence from our quantitative research. Only 2 percent of participants identified long delays and wait lists as difficulties experienced in accessing programs.
Approximately 20 percent of participants who responded to our telephone survey identified themselves as being a member of an employment equity group. Six percent identify themselves as being of Aboriginal descent, 8 percent as being a member of a visible minority group and 5 percent identify themselves as having a disability. Only 3 percent of our comparison group identified themselves as being a member of an employment equity group.
Those of Aboriginal descent and those who are members of a visible minority account for 4.6 percent and 10.1 percent of the population in Alberta according to the 1996 Census. We do not know the proportion of persons with disabilities. No data is available on the distribution of these groups among the unemployed population in Alberta.
An equity issue identified by the third party agents had to do with programs and services for clients who were more highly barriered. In their view, the accountability targets which were stipulated in their training contracts worked against provision of services for those less likely to succeed in terms of employment outcomes.
Accountability targets in contracts specify levels of employment outcomes. They do not specify the mix of inputs (percentage of participants who are more highly barriered) nor offer increased remuneration for employment results for those more highly barriered. Creaming (selecting those most likely to achieve results) is a possible consequence of this incentive structure and is consistent with the goal of achieving returns to work. However, it also tends to discriminate against those who are more highly barriered.
On the one hand, the "reward" system for third party delivery agents may cause them to focus on those individuals for whom achieving "measured" success is easiest. The availability of suitable programming may be reduced to the detriment of these "disadvantaged" groups. In our focus groups, third party delivery agents indicated that, in the future, they would be less likely to offer programs for groups with poorer prospects for achieving employment results such as high risk youth, persons with disabilities and workers over the age of 45.
On the other hand there are specialized programs and funds (outside of the LMDA) available for some disadvantaged groups—those with disabilities, youth, and those of Aboriginal descent. However, other groups do not have preferential access to programs or funds—women and those over 45 years of age.
Although evidence is insufficient at this time to infer that a negative impact has occurred for disadvantaged groups, there is sufficient evidence to warrant management review of this issue.
It was important that LMDA programs and services be cost-effective so that they were consistent with both resources available and with client needs. Further, funding allocation had to be flexible enough to fit with provincial priorities. There was no evidence of insufficient funding to provide services; in fact some sites had not fully expended their budget by the end of the first year of implementation.
Most participants in the focus groups were satisfied with their funding; some were very satisfied. Recall that only 5 percent of all participants but 29 percent of those experiencing difficulties accessing APSs, identified a lack of funds as the cause of those difficulties. Less than 40 percent of comparison group members who were aware of APS cited too great a cost or not receiving financial help as reasons why they did not participate.
As a result, APS appeared to be highly affordable for EI clients. For a few participants, the funding duration did not match their training duration and this was a concern for them. (For those who exhaust their EI Part I entitlement, income support may be [based on need] available through EI Part II. There is also an appeal mechanism available to individuals.)
Accountability was a key component of the LMDA. Employment programs and services focused on the achievement of results which were clearly defined as part of the accountability framework. The Agreement listed three measures of success:
AECD was accountable for results against these measures. AECD also held its third party delivery agents accountable through its contracting process which specified payment based on deliverables.
As part of their proposal and, for the successful proponent, as part of their contract, firms had to specify project deliverables along with the proportion of their budget to be expended for each activity. They had to list, for example, the expected number of clients to be served in the development of an action plan and the maximum proportion of the total contract fee which could be billed for this activity. In addition, all contracts were divided between service provision and outcomes. These outcomes identified specifically the minimum number of clients to achieve employment at three and six months subsequent to program completion. Contractors in our focus groups stated repeatedly that the use of deliverables placed too much emphasis on client numbers, ignored the variability of client needs and, as some commented, gave them the feeling of being a discount retailer (i.e., counting heads). While third party deliverers felt it unlikely that the overall focus on deliverables would change, based on its fit with both federal reporting requirements and the business practices of the Alberta government, the contractors raised a number of important issues associated with deliverables which warrant further exploration, including the following:
At no time did third party delivery agents object to being accountable for providing appropriate, quality training. However, their comments clearly point to potential problems associated with a narrow definition of success and a reward system based on a single undifferentiated measure. Potentially some of these concerns may be allayed by soon to be developed medium-term savings measures.
Alberta chose to build upon its own Students Finance System to administer EI Part II funds. The System was originally developed to support post-secondary students who were expected to work summers to contribute to their educational costs. It had been estimated that students could comfortably contribute $1,350 from their summer's wages to this end. When the System was expanded to handle all requests for EI Part II by EI clients, the required contribution of $1,350 was retained. However, the client group was of a different nature. EI recipients were adults with very different lifestyles, responsibilities and expenses from their generally younger and less encumbered student counterparts. Many staff and managers interviewed believed that the Students Finance policies were inadequate to assess the needs of this client group.
The result of applying this requirement to this different population has resulted in a number of bureaucratic problems, increased paperwork, decreased local authority, and an increased use of an exemption category called special circumstances.
Some general differences in the application of the requirement seemed evident across sites. In Edmonton, adherence to the required contribution appeared quite strict but at the same time all SW participants had the requirement waived. On the other hand, Calgary did not make exemptions for certain types of programming but staff instead managed by exception. As one staff member commented, "We live in "special circumstances."
Looking at only those cases where participants in our survey were able to identify costs, all costs were paid in 26 percent of cases in Calgary, compared to 15 percent of cases in Edmonton and 20 percent of cases overall in Alberta. Therefore, there is some justification for the observation that financial assessments are being conducted differently in Calgary, although as noted earlier there may be justification for practices to overcome regional differences.
Staff at several sites had a concern that the $1,350 required contribution led to some unanticipated outcomes. It was the view of some staff members that some programming decisions were being based on the amount of financial support provided rather than on the skill training needs. They also suggested that some clients were opting to take longer programs than they might have otherwise because proportionally, their fixed contribution of $1,350 seemed more appropriate for greater training amounts.
To illustrate this point, for their fixed contribution of $1,350, a participant would derive more value and therefore be more likely to opt for a two semester course for which EI Part II would contribute to a maximum of $15,500 than a 1 month course for which EI Part II might only contribute $1,000. Even though the one month course may better meet the skill training needs of the client, the two semester course represents a better "value" for the fixed $1,350 contribution. As a result, the client may choose the longer more expensive course; and expenditures under EI Part II are likely higher and unpaid EI likely lower.
In addition, the less affluent clients might not take either training option because they could not provide the minimum $1,350 contribution or might not wish to incur the debt.
Senior managers concurred with the above issues and also commented that training staff had proved to be difficult, that regional managers had exercised their authority on financial issues differently (e.g., Edmonton waiving the contribution for SW clients), and that regional staff might have had more authority than they were exercising with regard to financial assessments. Some staff believed decisions were made centrally related to such matters where in fact decisions were made locally and only input centrally.
Section 57 (e) of the EI Act states that clients should share in the costs of the assistance, if appropriate.
Exhibit 1 presents the proportion of those who identified no costs for their intervention, or if they were able to identify costs, paid all, part, or none of the costs by APS and by type of client.
| APS | No Costs Identified % | Costs and: | ||
| Paid All % | Paid Part % | |||
| Paid None % | ||||
| CEAS | 64 | 7 | 7 | 22 |
| SE | 24 | 43 | 4 | 28 |
| TOJ | 72 | 3 | 5 | 21 |
| SW | 46 | 1 | 4 | 50 |
| JP | 42 | 13 | 4 | 42 |
| SDU | 7 | 9 | 10 | 74 |
| SDS | 13 | 27 | 43 | 17 |
| All APS | 41 | 12 | 13 | 33 |
| Active Claimant | 44 | 13 | 16 | 27 |
| Reach-back Client | 38 | 11 | 11 | 40 |
| Apprentices | 1 | 73 | 21 | 6 |
Of confirmed participants in the survey, 33 percent identified that, although there were costs involved in their APS, they had not contributed to these costs. They had received financial support to cover all of these costs. A further one-quarter, paid at least part of the costs of their APS. These include 12 percent who paid all of the costs and 13 percent who paid part of them.
The above analysis suggests that there has been limited sharing in the costs of interventions under the LMDA. Sharing of costs occurred for 13 percent of non-apprentice interventions. Most frequent cases of sharing in costs were for SDS (43 percent), those 35 to 44 years of age (18 percent), those with household income of $20,000 to $40,000 (16 percent), and those with grade 12 education (16 percent). Even if those who stated they paid all costs were included, only 25 percent of non-apprentice participants have contributed to the costs of their back to work activities. The low percentage sharing in the costs of the intervention may not be appropriate.
The proportion identifying that all costs were paid by the APS was higher:
This suggests that the financial assessment is appropriate in its targeting of assistance to those with greatest need.
Apprentices are more likely to identify paying all costs of training. Apprentices are required, in most circumstances, to pay the first $1,350 of the costs of their in-school training component. This amount may exceed the cost charged to apprentices for their training plus other out-of pocket costs incurred by the apprentice.
Those receiving full or partial support were asked if they would have been able to take the intervention without the financial support they received. The proportions claiming they would not have been able to take the intervention without support is identified for select groups below (note that those who did not know are excluded from the total):
This suggests that the financial assessment has been successful in identifying those most in need. Earlier evidence suggests that few have difficulty accessing APSs because of financial limitations, further suggesting that the financial assessment is not unduly restrictive.
The LMDA (Annex 1 Section 5) specified that Canada and Alberta would work on a joint strategy to collaborate on the gathering, production and dissemination of local and provincial labour market information. This collaboration is occurring.
During our site visits the labour market kiosks appeared to be well used at the CASC sites and provided labour market information from across Canada. Some possible overlap was noted between federal (national focus with some local information) and provincial (Alberta-specific) information in the area of Occupational Profiles. Further, it appeared that reciprocal data sharing was hampered by systems issues at the time of our data collection. Requests from AECD staff to obtain feedback on market penetration and job-occupation classification fit were filled very slowly. In six months only one request had been completed.
Labour market information was available at the CASCs in well-stocked Labour Market Information Centres (LMICs or Career Resource Libraries). However, staff felt that these resources were under-utilised. The most likely cause was limited awareness. When third party service providers and clients discovered these resources, they reported that they were very useful for their labour market research. AECD publications were well received but, again, tended to be under-utilised according to staff. Respondents to our participant survey reported that about one-third used printed career and work search information. Access to this information may be available through channels other than on-site use or requests for publications. Over 40 percent of participants use the Internet. As a result, this information may be accessed in other, less traditional ways.
An issue for field staff was that the labour market information provided for them was too technical, not timely enough and not broken down to the local level necessary to assist them in their planning activities. As a result, regions began to contract for their own more detailed labour market research to be conducted through LLMPs. These efforts did not appear to be co-ordinated in any way across regions. Greater co-ordination may improve cost-effectiveness.
In support of the LMDA goal to facilitate operations and make programming clear and easy, systems were to be both functional and efficient. In fact, information systems issues have been difficult to resolve and have been detrimental to the implementation of the LMDA. Although system improvements have been on-going, stakeholders identified many system issues which had resulted in administrative inefficiencies and increased costs due to slow service provision. Specific concerns included the following:
An important principle related to access was that the combination of employment strategies and tools provided should allow a quick return to work. Overall, it appeared that appropriate access was provided at all sites and that they were providing short-term skills training and job placement. In particular, positive feedback was received about the relevance of CEAS exposure courses, the appropriateness of TOJ as a link to the labour force, and the use of SW for a wide variety of job skills and levels of expertise (except in Edmonton where the focus of SW was on multi-barriered clients who might be less likely to become employed on completion).
In most areas, SDP focuses on short courses, promoting expedient return to work. In some areas, such as Lethbridge, SDP is used to support academic up-grading for up to two years. SDP requires the participant to show that the intervention is the most expedient route to employment. Notwithstanding, two-year course lengths may not promote expedient return to work.
This section reviews delivery aspects for each APS and potential delivery concerns identified through the research.
a) Delivery
Career and Employment Services (CEAS) are designed to be short-term courses in work preparation, work search, and work maintenance ranging from one-hour interventions to one- to two-week workshops. Suggested services include: LMI and self-directed services; career planning assistance (advice, workshops, exposure courses); job placement services; and needs assessments. No outcome goals have been defined for this program.
In our case study sites we found two broad categories of activities being conducted under CEAS:
CEAS involved the most participants and the largest number of third party delivery agents. Generally, participants were very positive about CEAS and many reported being satisfied with the service they had received, citing staff and contractors as supportive, friendly, knowledgeable and committed. Many indicated that they had gained transferable skills, developed positive attitudes and increased their self-esteem. However, little information was available on the impact on employment status or reduction of EI.
b) Related Concerns
Across sites, concerns raised about CEAS included:
a) Delivery
Self-Employment (SE) was previously a federal program adopted by the province under the LMDA. Alberta had no prior experience with such a program. SE offered financial assistance, formal instruction, one-on-one business counselling, coaching, guidance and follow-up during the administration of participant business plans. These services were designed to assist unemployed Albertans in need of self-employment training to create additional employment for themselves and other community members by starting a new business. It was expected that 90 percent of participants would complete their SE training and 80 percent would become self-employed on completion for an overall success rate of 72 percent.
Contractors indicated that CASC staff knowledge about the SE program was generally mixed. However, overall, it was felt that the program had improved and had a stronger emphasis on assessment, on training objectives and on the relation between the proposed business and the local economy since transfer to the province.
Contractors were providing a useful and informative training program in the view of most participants who tended to be enthusiastic about their own business prospects and more self-confident as a result of the program.
b) Related Concerns
Concerns expressed by key informants included:
a) Delivery
The Training on the Job (TOJ) program was designed to help clients acquire transferable occupational skills and work experience under prospective employers in jobs that were currently in demand in the labour market. Clients were placed with employers for up to eight months while the employer was reimbursed for the job training and work experience provided. During the TOJ placement, the participant became an employee and no longer drew EI or SFI benefits. The employer could be paid up to $1,200 per participant on a fee-for-service basis to set up TOJ opportunities and to develop training plans. In addition employers were reimbursed up to 70 percent of their training costs per participant. A target of 75 percent of participants are expected to be employed at the conclusion of their TOJ.
Large centres tended to deliver TOJ through "parent" contracts held with an agency which administered training plans. In small centres, contracts were held directly with employers.
The matching and referral process generally appeared to work well and employers felt that TOJ participants were as appropriate as any other new hires. The employers found the training plans useful but suggested that the required status reports took too much of their time to prepare. TOJ participants were very positive about the training they were receiving and felt that it was both appropriate to their needs and likely to improve their success in the labour market. Their employers were satisfied with the compensation they had received for training support. Most planned to hire their TOJ participants but also felt that they had gained transferable skills should they be hired elsewhere. Several commented that without the program, they would not have hired their participants.
b) Related Concern
A concern identified for TOJ was the lack of adequate public awareness. Fifty percent of comparison group members in our survey were aware of TOJ.
Responses by participants in all APSs other than TOJ follow a typical pattern where most responses are clustered around some common value. However, for TOJ responses seem to cluster around not one but two common values. This suggests some factor (possibly delivery options) may be the cause. We have insufficient numbers of TOJ participants in our survey to assess the two delivery options (parent contracts vs. individual contracts) separately. Further research, involving larger sample sizes, would be needed to identify the reason.
a) Delivery
The Skills for Work (SW) provides unemployed Albertans with the skills training in order to facilitate their re-entry into the labour market. Training is aimed at providing clients with the necessary skills to enter high demand occupations which have entry level wages of at least $7.00/hour. Clients are provided with a combination of life management skills, occupation-specific skills training, basic skills upgrading, unpaid work experience, and job maintenance support. Participants spend approximately 50 percent of the normal 26-week program in training, and the other 50 percent in work experience. It is expected that 75 percent of SW clients will obtain full-time employment after successful completion of their program.
Most SW training prepared clients for more than entry-level positions in areas of occupational demand but in small communities, there might not be a significant demand at this more advanced level. For example, in Slave Lake, SW contracts were more likely to focus on entry-level skills simply because of market demand. In Calgary and Lethbridge the program targeted a variety of skill levels ranging from basic academic upgrading to high-end occupational training (including electrical engineering and advanced computer training). However, in the Edmonton Region, most SW contracts targeted multi-barriered clients and many were designed primarily for SFI clients. Most of the EI skill training in the Edmonton Region occurred in SDP instead.
b) Related Concerns
The SW initiative displays a high degree of flexibility in terms of its target group and occupational focus. Further, in Edmonton EI clients in SW were not expected to contribute the $1,350 towards their tuition but there was no evidence that this was the case in other sites. There is a potential concern about the equity impacts of these differences across regions. Another impact is that the variety in SW suggests the telephone survey results should be treated as indicative of the program as a whole in Alberta and not indicative of the SW model in any one region.
In the smaller centres, another concern had to do with the impact of the available training options on timeliness. In order to fit with the inflexible academic year maintained by post-secondary institutions (often the only agencies available to offer SW programs) clients had to wait until a program could be provided or be provided with some other APS option. This suggests that clients may have had to wait for the SW course they wanted or choose another less desirable APS option.
a) Delivery
Job Placement Services (JPS) are services available under the LMDA to unemployed Albertans seeking employment. The main objectives of these services are to assist participants in obtaining full-time employment and enable employers to fill job vacancies and skill shortages in their organizations. JPS have a maximum duration of 12 weeks. Typically, participants spend four weeks receiving support services, and eight weeks conducting job search and job placement. The types of services available include: (1) assessment of the match between participants' employment skills and the job requirements of available positions; (2) job placement; (3) job retention skills; and (4) job maintenance/support for the first 3-6 months on the job. Through these services, participants learn the necessary skills to find, obtain, and keep jobs. These services have a 70 percent target success rate.
In the larger centres reviewed, JPS training emphasized not only job placement but also skill development. In order to increase participants' ability to find employment, they were taught a variety of skills including how to write resumes, market their skills and abilities, conduct labour market research, and successfully complete interviews.
Staff were guarded in their comments about JPS:
However, JPS participants were overwhelmingly positive in their comments about training quality, resources provided, the excellence of instructors, the caring follow-up process and the impact of the program on their skill development and attitudes (more confidence, knowledge and focus for their job search). None of them had yet gained employment, but all seemed to think that they would. They also had positive comments about the help provided by CASC staff who were found to be fair and helpful in general.
b) Related Concerns
In general, participants' comments about JPS were positive. However, several participants in Calgary felt that the course they took should be increased in length. Since the implementation of the LMDA, the course had been cut from a two-week full-day course to a two-week half-day course. At least one participant was in the program for a second time (after an interval of several years) and felt that the shorter time frame was not appropriate for the amount of material covered.
a) Delivery
The Skills Development Program (SDP) offered financial assistance to upgrade skills in order to obtain sustainable employment in the shortest time possible and thus to reduce dependence on government income support. In fact, SDP was not a program in the sense of other APSs, but was instead a type of financial assistance which could be provided to clients with specific training needs. There were two categories in SDP:
While in SDP, active claimants continued to receive their EI benefits (Part I) as determined by HRDC for the full length of their benefit period. Active claimants and reach-back clients may also receive EI (Part II) funding to cover tuition, books and to top up their living expenses. They were expected to contribute $1,350 of their own savings towards their support. In cases where the client was unable to do this, the amount was provided to them as a loan through the Alberta Student Loan program.
It should be noted that no SDU participants were interviewed; only those in SDS took part in the focus groups. While they reported that the application process seemed to be fairly smooth, funding determination was generally unclear to them. Most were delighted with the support they received; a few ran into problems with a mis-match between remaining claim length and length of funding required.
Generally, participants were positive about their training, although some in Edmonton had concerns about both training quality and learning resources provided (e.g., limited access to computers and/or the Internet).
b) Related Concerns
Observations identified regarding SDP included the following:
|
This section presents findings on process-related outcomes of the Labour Market Development Agreement (LMDA). It presents details on the labour market information and community employment resources used by respondents on a self-serve basis. It also discusses client self-sufficiency, their attitudes toward work and social assistance and their satisfaction with the Alberta Programs and Services (APSs). We use findings from our survey of comparison group members to contrast with findings from participants. Comparison group members are chosen to match participants prior to the survey. However, we cannot control which comparison group members or participants ultimately respond to surveys. Those who complete our surveys are always less well matched than the initial samples. Regression analyses are able to adjust for these differences. Note, however, that simple comparison across the two groups may be affected by the differences between the ultimate samples.
The unemployed have a variety of employment resources available to them. In our survey, we asked whether individuals had made use of these resources on their own (i.e., on a self-serve basis). Eighty-three percent had. The most frequent resources used by these self-serve clients were newspapers (87 percent), job banks at government offices (77 percent), job banks at other locations (50 percent) and the Internet (44 percent). Exhibit 2 presents details on the use of employment resources.
Seventy-two percent of comparison group members have made use of employment resources available in their communities on their own. The most frequent resources used by these comparison group members are newspapers (93 percent), job banks at government offices (71 percent) and job banks at other locations (30 percent). Newspapers are used more frequently and job banks less frequently. Compared to the participant group, comparison group members are more likely to use newspapers (93 percent vs. 87 percent) and less likely to use job banks at government offices (71 percent vs. 77 percent) and at other locations (30 percent vs. 50 percent).
Comparison group members more likely to have used employment resources on a self-serve basis are:
| Percentage of Use by Self-Serve Clients | |
| Total - Actual/Weighted | (859/831) % |
| Employment Resource Used: | |
| Newspaper Advertisement | 87 |
| Job Bank Kiosks/Job Board at Government Office | 77 |
| Job Bank Kiosks/Job Board at Other Location | 50 |
| Internet Resources | 44 |
| Provided Technology (computers, facsimiles, telephones) | 35 |
| Printed Career and Work Search Information | 35 |
| Recorded Career Information at the Employment Insurance 1-800 Number | 17 |
| CHOICES | 14 |
| Miscellaneous Others | 3 |
| Don't Know | * |
| *Equals less than one-half of one percent. | |
Greater use by those with higher income, higher education and more youth is not evident in the participant survey data. This fact suggests that participation has resulted in greater access among those groups who do not normally seek out self-serve tools.
a) Internet
Forty-four percent from our participant survey, but only 23 percent of comparison group members, identified using the Internet for labour market information.
The Internet resources most frequently accessed by the participant group are job listing (other than Electronic Labour Exchange) or résumé posting services (41 percent), the Electronic Labour Exchange (22 percent) and Canada WorkInfoNet (22 percent). Comparable percentages for comparison group members are 36 percent, 8 percent and 28 percent.
b) Printed Career and Work Search Information
Thirty-four percent of the participant survey group, but only 11 percent of comparison group members, identified using printed career and work search information.
The principal items used by those individuals from our participant survey using printed career and work search information are the Job Guide (53 percent), Work Futures/Job Futures (30 percent) and Career Paths (29 percent).
Participant group members using printed career and work search information accessed it from a variety of sites. They are most likely to have found this information at a Career Development Centre (CDC) (33 percent), Human Resources Development Canada (HRDC) or Employment Insurance (EI) office (22 percent), Labour Market Information Centre (12 percent) or at a Canada-Alberta Service Centre (CASC) (11 percent).
c) Provided Technology
Participant group members are also more likely to use technology provided for them by government offices or their third party service provider. Thirty-five percent of the participant group and 22 percent of comparison group members said they used provided technology. The technologies used most frequently by participant group members were computers (74 percent), fax machines (53 percent) and photocopiers (53 percent). These technologies are most frequently used at CDCs (29 percent), private training centres (13 percent), HRDC or EI offices (12 percent), Human Resource Centres of Canada (HRCC) (12 percent) or CASCs (11 percent). Exhibit 3 provides detail on the use of technologies, by participant group members.
| Percentage of Participant Group Members Who Used Technology on Their Own | |
| Total - Actual/Weighted | (283/292)% |
| Technology Used: | |
| Computer | 72 |
| Fax Machine | 52 |
| Photocopier | 51 |
| Telephone | 45 |
| Printer | 45 |
| Other | 7 |
| Don't Know | 3 |
| Where Technology Used: | |
| CDC | 29 |
| Private Training Centre | 13 |
| HRDC or EI Office | 12 |
| HRCC | 12 |
| CASC | 10 |
| Community Skills Centre | 7 |
| Home | 6 |
| Non-Profit Community Agency | 6 |
| Other | 21 |
| Don't Know | 7 |
d) Participant Group Members' Attitudes Toward the Labour Market
Information Used
Participant group members rated five statements about the labour market information they used on a scale of one to seven. On this scale, one means "strongly disagree" and seven means "strongly agree." They were most likely to find the labour market information easy to understand (5.9), easy to use (5.8) and/or up to date (5.8). They were slightly less likely to find that it met their needs (5.2) or made them more self-sufficient (5.2).
Participants who were also self-serve clients and comparison group members were asked questions in their respective surveys, related to their use of Labour Market Information (LMI) and provided technology. We assessed potential differences through regression techniques. Regression models controlled for potential differences in the demographic and other characteristics between participants and comparison group individuals. By adjusting for the effects of other known differences, the differences that remain may be due to the intervention. Significant results are listed in Exhibit 4. These results represent changes that are larger than would be normally associated with chance. Using regression techniques, we found a significant difference4 in the use of:
| Significant Result | Estimate | Level of Significance | |
| Outcome Variable: | |||
| All APSs | |||
| Use of Printed Information | YES | 20.5% | <0.05% |
| Use of Internet | YES | 19.2% | <0.05% |
| Use of Provided Technology | YES | 10.5% | 0.05% |
| Active Claimants Only: | |||
| Use of Printed Information | YES | 16.5% | <0.05% |
| Use of Internet | YES | 18.4% | <0.05% |
| Use of Provided Technology | YES | 14.2% | <0.05% |
| Reach-Back Claimants Only: | |||
| Use of Printed Information | YES | 29.9% | 0.3% |
| All APSs Except CEAS: | |||
| Use of Printed Information | YES | 21.6% | <0.05% |
| Use of Internet | YES | 25.7% | <0.05% |
| Use of Provided Technology | YES | 81.5% | 3.4% |
| CEAS: | |||
| Use of Printed Information | YES | 27.3% | <0.05% |
| Use of Internet | YES | 14.4% | 1.6% |
| Use of Provided Technology | YES | 20.6% | <0.05% |
| JP: | |||
| Use of Printed Information | YES | 19.8% | 2.9% |
| Use of Internet | YES | 40.2% | <0.05% |
| SDS: | |||
| Use of Printed Information | YES | 15.0% | 1.8% |
| Use of Internet | YES | 15.4% | 3.4% |
| SDU: | |||
| Use of Printed Information | YES | 30.7% | 0.5% |
| SW: | |||
| Use of Printed Information | YES | 32% | <0.05% |
| Use of Internet | YES | 24% | 0.2% |
| Use of Provided Technology | YES | 27% | <0.05% |
Participation may be responsible for these differences. Alternatively, some other unmeasured factor, such as motivation, may account for both higher use and participation in an APS.
Client self-sufficiency was examined using the following indicators:
Two-thirds of confirmed APS participants (67 percent) have developed action or business plans and 90 percent have completed at least some of the goals set out in these plans. As a result, a majority of participants take part in activities that should lead to their self-sufficiency.
As reported, 43 percent of APS participants who identified costs for their APS contributed to these costs. These include 23 percent who paid part and 20 percent who paid all of their costs. Slightly less than half participate in the costs of their back to work activities.
As reported below, participants rate the importance of becoming independent of Employment Insurance (EI) or Supports for Independence (SFI) very highly (at 6.2 on a seven-point scale). They also rate their willingness to pay for a training course to stay off government support high (at 4.9 on a seven-point scale). As a result, participants appear motivated to becoming self-sufficient.
APS participants were asked whether their attitudes toward work and their futures have become more positive, more negative or stayed the same since their involvement in their APSs. Sixty percent report that their attitudes have become more positive and 30 percent say they have stayed the same. Only 9 percent report that their attitudes have become more negative.
Participants whose attitudes have become more positive are more likely to be:
Those with a more negative attitude are more likely to:
On balance comparison group members are slightly more negative and apprentices slightly more positive than non-apprentice participants in their attitudes to a number of statements about themselves and the labour market they face.
In our survey, statements were assessed on a scale of one to seven. On this scale, one means "strongly disagree" and seven means "strongly agree." Participants agree (5.4) with the statement "I am generally optimistic about my future." They also agree, but less strongly (4.9) that "If it means staying off government support, I would be willing to pay for a training course." Comparison group members are slightly less likely (5.2 and 4.7 respectively) to agree with these statements. Apprentices are more likely (5.8) to agree to the first and equally likely to agree with the second statement.
Non-apprentice participants are less likely to agree that "I would turn down a well paying job if I had to move from my community to get it" (3.3) or that "There are no jobs out there" (2.4). Comparison group members are slightly more in agreement (3.5) that they would turn down a job if they had to move. Apprentices are slightly less in agreement (3.2). Comparison group members agree with non-apprentice participants (both 2.4) in their assessment of the availability of jobs. Apprentices are slightly less likely to agree that there are no jobs out there (2.2).
Participation may also affect attitudes beyond their work and futures. We asked participant and comparison group members: "How important is it to you to be independent of employment insurance benefits or social assistance?" A seven-point scale was used to assess importance where "1" was not very important and "7" was very important.
Statistically significant results were found for participants under all APSs, all APSs excluding CEAS, and under SDS who put higher importance on being independent of EI or SFI benefits. Differences may be attributable to participation since the two groups were drawn to be statistically similar in observable characteristics other than participation.
Note, however, that any differences may be due to other unobservable factors. For example, we cannot rule out the possibility that some hidden factor, such as motivation, is attributable both to participation and to any difference between participants' and comparison group members' responses to these questions. Findings are indicative but not definitive of an attributable impact of the APSs.
Satisfaction was assessed on a scale of one to seven. On this scale, one means "not at all satisfied" and seven means "very satisfied."
a) Satisfaction
In total, 725 participants have ended their APSs. On average, these participants rate their satisfaction with their APSs at 5.2. APS satisfaction is highest among SDU participants (6.2) and lowest among TOJ participants (3.5). Satisfaction for all other APSs are between 5.0 and 5.3 on this scale.
b) Satisfaction by Apprentices with Training
Apprentice participants rated their satisfaction with the training they received. On average, apprentices rated their training at 5.7.
c) Satisfaction with Current Job
In total, 407 participants were working at the time we interviewed them. On average, participants rated their satisfaction with their jobs at 5.0. Comparison group members working at the time we interviewed them rated their job satisfaction slightly lower than this (4.8) on the same scale.
Participants who have ended their APSs also rated the usefulness of their APSs to the achievement of a number of goals on a scale of one to seven. On this scale, one means "not at all useful" and seven means "extremely useful." They found their APSs to have been most useful at helping them increase their confidence and clarify their goals (4.7 for each). They also found that they helped them to gain specific occupational skills, general job skills and full-time employment (4.1 for each). They found them least useful at improving their earning potential (3.9).
APSs ranked highest in terms of areas of specific usefulness were:
About one-quarter of participants whose APSs have ended (27 percent) have taken further training or courses since that time. Seventy-three percent have not. Of those who have taken further training, just over one-half (52 percent) say they are taking it is as a result of their involvement in their APSs. Thus, about 14 percent of participants whose APSs have ended are taking further training as a result of their involvement in their APSs. This is a positive outcome of the APS experience.
Of the participants who confirmed their involvement in the SE program, three-quarters (81 percent) report having started their own businesses. This includes 35 percent who started their own business before starting their SE program5 and 46 percent who started their business after starting their programs. Note that we do not know the extent to which these businesses were operating at the time participants started the SE program. Those started prior to the SE start date may have been operating at a marginal level or not at all when the SE intervention began.
Of the participants who started businesses before they entered the SE program, 61 percent say they made changes to their businesses as a result of their participation in the SE program. Of the participants who started businesses after they entered the SE program, three-quarters agree that their participation in the SE program allowed them to start this business. Therefore, 69 percent report a positive impact (changes to existing business or started a new business) as a result of SE.
At the time of the survey, 74 percent were running their own businesses. Most of the businesses opened by SE participants were started between April and June 1998 (31 percent), July and September 1998 (30 percent) or October and December 1998 (24 percent). Most of the participants currently running their own businesses (58 percent) are still participating in their self-employment programs.
SE participants who have ended their programs give them an overall satisfaction rating of 5.0 (out of seven). They found them most useful in terms of learning how to run their own businesses (4.7), getting more confidence in themselves and their abilities (4.6), clarifying their career goals (4.3), gaining general job skills (4.3) and improving their earning potentials (4.1). They found them somewhat less useful in terms of gaining specific occupational skills (3.2).
|
In our qualitative research, staff felt it to be too soon to determine outcomes of the Labour Market Development Agreement (LMDA). The majority of staff, however, felt that LMDA programs and services were effective in increasing clients' work skills and will reduce Employment Insurance (EI) and Supports for Independence (SFI) dependence in the longer run.
The quantitative research is subject to the same limited time frame. This section begins by assessing short-term achievements in terms of action plan and Alberta Programs and Services (APS) completion rates. It presents survey findings on pre- and (short-term) post-APS employment and earning levels as well as dependence on EI and SFI. Finally the chapter tests for significant impacts in the short-term. We use findings from our survey of comparison group members to contrast with findings from participants. Comparison group members are chosen to match participants prior to the survey. However, we cannot control which comparison group members or participants ultimately respond to surveys. Those who complete our surveys are always less well matched than the initial samples. Regression analyses are able to adjust for these differences. Note, however that simple comparison across the two groups may be affected by the differences between the ultimate samples.
Of confirmed APS participants, two-thirds (67 percent) have developed an action or business plan. Eighty-eight percent of those taking Self-Employment (SE) and 79 percent of those taking Skills Development Upgrading (SDU) had completed an action plan or business plan when interviewed. The percentage was higher for females (70 percent), reach-back clients (71 percent), those between the ages of 18 and 34 (71 percent), those with less than a grade 12 education (73 percent) and with household income of under $20,000 (78 percent). This suggests that there appears to be targeting of action plans to those who might benefit most from them. The percentage with action plans did not vary by region. This suggests that the targeting of action plans, if it does occur, appears to be similar across regions.
Of these 581 participants with an action or business plan, 90 percent have completed at least some of the goals set out in these plans. Only 12 percent have completed all of the goals in their plans, but a further one-third (32 percent) have completed most of them and almost one-half (46 percent) have completed some of the goals. Only 9 percent have not completed any of the goals set out in their plans. Of all confirmed APS participants, 9 percent have completed all, 23 percent have completed most, 34 percent have completed some, while 7 percent have not completed any of the goals set out in their plans.
The most frequent reason that all goals have not yet been achieved is that they are still being completed. Forty-six percent of the participants who had not completed all of their action or business plan goals said this. The highest percentage was for SDU participants (67 percent) whose goals may be longer term in nature. Other reasons for goals not being completed are:
Completion of all goals set out in their action (business) plan is highest among participants in SDU (26 percent), Skills for Work (SW) (23 percent), and Skills Development Skill Training (SDS) (23 percent). Using completion of all or most goals (as assessed by respondents) as a measure of achievement, APSs with the highest percentages were SDS (59 percent), SW (55 percent), and SDU (48 percent). Using all or most as the measure, those most likely to achieve this level of success were those with household income of more than $40,000 (60 percent), those who completed their APS (51 percent), those in Edmonton or the North (50 percent for each) and those 18 to 34 years of age (49 percent). Achieving none of the goals was identified by 27 percent of TOJ participants and 9 percent of respondents overall.
In total, 725 participants had ended an APS by the time we surveyed them. Among these participants, two-thirds (66 percent) completed their programs and 27 percent discontinued them. Most of those who discontinued their APSs (62 percent of those who discontinue) did so because they found (other) employment. Finding work as a reason for discontinuing was most significant for participants of JP (90 percentage of those who had discontinued, 39 percent of those who ended programs). The percentage of those who discontinued for this reason increased with the education level and household income of the respondent.
Eleven percent of those who discontinued their APSs did so because funding ceased. A further 7 percent left because of insufficient funds. These responses were most significant for CEAS participants at 24 percent and 13 percent respectively. Six percent of those who discontinued their APSs did not complete their APSs because they were not satisfied with them.
Exhibit 5 presents a summary of the post-APS6 employment characteristics for non-apprentice participants, comparison group members and apprentices. Seventy-nine percent of non-apprentice participants who have ended their APSs have been employed subsequently. For members of the comparison group, 66 percent have worked over a similar period. For apprentices, 94 percent have worked after the date their training ended.
| Percentage of Respondents Who Are ... | |||
| APS Participants | Comparison Group Members7 | Apprentices | |
| Total - Actual/Weighted | (651/725) % | (393/387) % | (180) % |
| Employed During Post-APS Period | 79 | 66 | 94 |
| Proportion of Weeks Employed in Post-APS Period: | |||
| 0% To 25% | 24 | 38 | 2 |
| 26% To 50% | 11 | 6 | 2 |
| 51% To 75% | 17 | 11 | 10 |
| 76% To 100% | 47 | 45 | 86 |
| Employed 12 or More Consecutive Weeks During Post-APS Period (Return to Work) | 57 | 55 | 89 |
| Currently Employed | 56 | 51 | 69 |
| Average Number of Jobs Held in Post-APS Period | 1.2 | 0.8 | 1.3 |
Almost one-half of both participants (46 percent) and comparison group members (45 percent) worked more than three-quarters of the weeks in their post-APS periods. Among apprentices, 82 percent worked more than three-quarters of these weeks.
At the time of the survey, 56 percent of participants and 69 percent of apprentices who had ended their programs as well as 51 percent of comparison group members were employed. Similarly, 57 percent of participants, 55 percent of comparison group and 89 percent of apprentices were employed for 12 or more consecutive weeks during their post-periods.
Exhibit 6 compares the characteristics of jobs held by participants and comparison group members in the post-APS period. Participants (including those who have not worked) held an average of 1.2 jobs during the post-APS period. Comparison group members held an average of 0.8 jobs and apprentices 1.1 jobs during this period. Comparing only those who have held jobs, the average number of jobs held is 1.5 for non-apprentice participants and 1.3 for both comparison group members and apprentices. Most jobs held by participants (92 percent), comparison group members (93 percent) and apprentices (98 percent) were with employers. For participants under the SE program, 50 percent of jobs were in business for themselves.
| Percentage of Respondents Holding Jobs in Post-APS Period Who Are ... | |||
| APS Participants | ComparisonGroup Members8 | Apprentices | |
| Total - Actual/Weighted | (744/810) % | (388/323) % | (224) % |
| Type of Job: | |||
| With an Employer | 92 | 93 | 98 |
| Self-employed | 8 | 7 | 2 |
| Average Hours per Week | 38 Hours | 39 Hours | 45 Hours |
| Average Earnings per Week | $422 | $555 | $667 |
On average, these jobs were full time. Apprentices worked the highest average number of hours (45 hours per week) and earned the greatest average incomes ($667 per week). Comparison group members who worked, worked an average of 39 hours per week and earned an average of $555 per week. Participants who worked, worked 38 hours per week on average for earnings of $422 per week. However, recall that a higher proportion of participants compared to comparison group members (79 percent versus 66 percent) worked in the post-APS period. (These differences are better assessed through the regression analyses that follow.)
These data hide the differential impacts that are occurring by client type. Exhibit 7 compares the post-APS period employment characteristics of non-apprentice participants and comparison group members.
As the exhibit illustrates, reach-back participants do better than those chosen to compare with them on all measures, except for average salary. Active claimant participants do less well on all measures compared to those chosen to represent them.
| Participants | Comparison Group9 | |||
| Active Claimant | Reach-back Client | Active Claimant | Reach-back Client | |
| Worked Since APS | 78% | 79% | 82% | 44% |
| Worked >75% of Time Since APS | 38% | 56% | 51% | 38% |
| Worked 12 Consecutive Weeks Since APS | 52% | 61% | 67% | 39% |
| Average Hours Worked in Typical Week if Worked | 37 Hours |
39 Hours |
40 Hours |
38 Hours |
| Average Income in Typical Week if Worked | $450 | $388 | $540 | $627 |
| Currently Working | 58% | 54% | 60% | 39% |
The latter is a frequent finding of formative evaluations. The intervention appears to lengthen the re-entry to the labour market of active claimants. When they do re-enter they must catch-up to the comparison group members who did not participate in the intervention. Note that the gap between participants and comparison group members may already be narrowing by the time of the formative evaluation. There is only a 2 percent difference between the employment rate at the time of the survey for participant and comparison group active claimants.
Exhibit 8 compares the pre-APS period10 experience for participants, comparison group members and apprentices. In the year before their APSs started, 81 percent of participants had been employed at least once. Seventy-six percent of comparison group members and 93 percent of apprentices had been employed in the pre-period.
Just over one-third of participants (36 percent) and one-half of comparison group members (50 percent) worked more than three-quarters of the weeks in the pre-period. Among apprentices, 83 percent worked more than three-quarters of these weeks. Participants held an average of 1.4 jobs during the pre-APS period. Comparison group members had held an average of 1.0 jobs and apprentices 1.3 jobs during this period. (These averages include those who held no jobs.) Most jobs held by participants (96 percent), comparison group members (95 percent) and apprentices (99 percent) were with employers. (See Exhibit 9.)
| Percentage of Respondents Who Are ... | |||
| APS Participants | ComparisonGroup Members11 | Apprentices | |
| Total - Actual/Weighted | (651/725) % | (426/426) % | (180) % |
| Employed During Pre-APS Period | 81 | 76 | 93 |
| Proportion of Weeks Employed in Pre-APS Period: | |||
| 0% to 25% | 24 | 24 | 2 |
| 26% to 50% | 14 | 11 | 4 |
| 51% to 75% | 23 | 12 | 5 |
| 76% to 100% | 39 | 53 | 89 |
| Average Number of Jobs Held in Pre- APS Period | 1.4 | 1.0 | 1.3 |
| Percentage of Respondents Holding Jobs in Pre-APS Period Who Are ... | |||
| APS Participants | ComparisonGroup Members12 | Apprentices | |
| Total - Actual/Weighted | (805/944) % | (501/430) % | (231) % |
| Type of Job: | |||
| With an Employer | 96 | 95 | 99 |
| Self-employed | 4 | 5 | 1 |
| Average Hours per Week | 41 Hours | 41 Hours | 46 Hours |
| Average Earnings per Week | $442 | $588 | $579 |
On average, these jobs were also full time. Apprentices worked the highest average number of hours (46 hours per week) and earned the greatest average incomes ($579 per week). Comparison group members worked an average of 41 hours per week and earned an average of $588 per week. Participants worked 41 hours per week on average for earnings of $442 per week. (The statistical matching exercise used to draw the comparison sample uses a number of characteristics, including earnings, to choose the best comparison to the participant sample. In the survey, some but not all members of the comparison and participant samples are interviewed. The achieved samples are always less well matched than the original samples. These differences are "controlled" through the regression analyses that follow.)
Exhibit 10 compares the pre-period employment characteristics of non-apprentice participants and comparison group members by client characteristics.
| Participants | Comparison Group13 | |||
| Active Claimant | Reach-back Client | Active Claimant | Reach-back Client | |
| Worked in Year Pre-APS | 96% | 78% | 96% | 64% |
| Worked >75% of Time in Year Pre-APS | 42% | 34% | 40% | 48% |
| Average Hours Worked in Typical Week if Worked | 42 Hours | 39 Hours | 42 Hours | 38 Hours |
| Average Income in Typical Week if Worked | $516 | $357 | $607 | $547 |
As the exhibit illustrates, participants who are active claimants are similar in most pre-APS employment characteristics, other than average salary, to their comparison group counterparts. Reach-back participants were more likely to be employed but for a shorter duration than their counterparts in the comparison group.
Comparison of the pre- and post-APS period employment data by client type continue to suggest that reach-back participants gain relative to their comparison group in all measures other than earnings. Active claimant participants lose "ground" relative to their comparison group in all measures other than average income for those who work. On this measure, participants lose $66 per week and comparison group members lose $67. (Although the data presented in this section are indicative of a change, the regression analyses, which "control" for pre-period differences across our achieved sample of participants and comparison group members, provide a more definitive treatment of impact.)
The high proportion of reach-back participant and comparison group members who have been employed in the year prior to the APS suggests that employment must be too short to produce enough eligible weeks or to be ineligible for Employment Insurance. Being self-employed does not account for the lack of EI eligibility.
Exhibit 11 provides a summary table of the post-APS period dependence on EI and SFI of participants, comparison group members and apprentices. In the post-APS period, about one-third of each group had received EI—32 percent of participants, 29 percent of comparison group members and 33 percent of apprentices. The figure for participants will include those still on claim when their APS ended and may include those whose EI was "extended" (EI Part II income support) through participation in APSs after the one selected for our survey. The percentage on EI at the time of the survey was the same for participants (14 percent) and comparison group members (14 percent). It was 20 percent among apprentices at this time.
| Percentage of Respondents Who Are ... | |||
| APS Participants | ComparisonGroup Members14 | Apprentices | |
| Total -Actual/Weighted |
(651/725) % |
(393/387) % |
(180) % |
| Income Support Received During Post-APS Period: | |||
| EI | 32 | 29 | 33 |
| SFI | 10 | 11 | - |
| Income Support at Time of Survey: | |||
| EI | 14 | 14 | 20 |
| SFI | 6 | 10 | - |
Ten percent of participants and 11 percent of comparison group members received SFI in the post-APS period. None of the apprentices had received SFI in the post-APS period. The percentage on SFI at the time of the survey was lower for participants (6 percent) than for comparison group members (10 percent).
The principal reason identified for being on EI or SFI among participants is a lack of available work. (See Exhibit 12.) Seasonal layoff and poor health or a disability are the most frequent reasons mentioned by comparison group members. Apprentices most frequently mention being in school and a lack of available work as their reasons for being on EI.
| Percentage of Respondents Receiving EI/SFI During Post-APS Period Who Are ... | |||
| APS Participants | ComparisonGroup Members15 | Apprentices | |
| Total - Actual/Weighted |
(292/292) % |
(185/155) % |
(60) % |
| Reason: | |||
| Lack of Work Available | 44 | 27 | 40 |
| Seasonal Work/Layoff | 15 | 28 | 22 |
| Poor Health/Disability | 13 | 32 | - |
| Only Working Part Time | 10 | 3 | - |
| Waiting for Start of New Job | 7 | 6 | 5 |
| In School/Training Program | 5 | 8 | 42 |
| Lack of Training/Skills | 3 | 1 | - |
| Still Eligible | 3 | 1 | - |
| Maternity Leave | 3 | 1 | - |
| Hours Cut/on Call | 3 | - | 2 |
| Miscellaneous Others | 11 | 4 | 7 |
| Don't Know | * | - | - |
| Refused | 1 | - | - |
|
Data based on sample sizes of less than 100 should be interpreted with caution.
*Equals less than one-half of one percent. |
|||
Exhibit 13 compares the post-APS period dependency for participants and comparison group members by client characteristic. As the exhibit illustrates, participants and comparison group members who are active claimants were almost equally likely to draw EI in the post-APS period. At the time of the survey, only 18 percent of active claimant participants were drawing EI compared to 23 percent for active claimants from the comparison group. Reach-back participants were more likely to be on EI at some time after the APS and at the time of the survey compared to comparison group members with like characteristics. This may be because the APS has resulted in eligible hours of work that might qualify them for EI support.
| Participants | Comparison Group16 | |||
| Active Claimant | Reach-back Client | Active Claimant | Reach-back Client | |
| Received EI Post-APS Period | 48% | 14% | 47% | 4% |
| Receive EI at Time of Survey | 18% | 10% | 23% | 1% |
| Receive SFI Post-APS Period | 6% | 14% | 3% | 23% |
| Receive SFI at Time of Survey | 4% | 9% | 1% | 22% |
Reach-back participants are less likely to draw SFI in the post-APS period (14 percent versus 23 percent) and at the time of the survey (9 percent versus 22 percent) than their comparison group counterparts. Interestingly, active claimant participants are slightly more likely to be on SFI in the post-APS period and at the time of the survey than their matched comparison group members.
We used regression analyses to identify whether participation in an APS had a significant effect on the experience of participants in terms of their:
This analysis uses data for a matched comparison group of non-participants to predict what would have happened to participants had their interventions not occurred. Individuals for the comparison group were chosen to reflect the characteristics of participants (determined through administrative sources) other than participation.
The regression model compared the actual and predicted experiences, the difference reflecting the effect of participation in an APS. This model corrected for selection bias. A number of demographic and other variables were used to control for remaining differences between participants and comparison group members, including pre-APS values of the outcome variables.
Exhibit 14 presents significant findings from the regression analysis related to the above outcome variables. For each outcome variable, we list whether the result is significant. For those that are significant, we list the estimated value for the variable and the level of significance17 of the estimate. Impacts are assessed for all APSs, for all APSs other than CEAS, for active claimants and reach-back clients, and for each separate APS.
| Significant Result | Estimate | Level of Significance | |
| Outcome Variable: | |||
| All APSs | |||
| Hours Worked per Year | YES | 173 | 0.5% |
| Active Claimants Only: | |||
| Earnings per Year | YES | -$2,530 | 0.3% |
| Hours Worked per Year | YES | -158 | 1.0% |
| Reach-Back Clients Only: | |||
| Hours Worked per Year | YES | 755 | 0.1% |
| All APSs Except CEAS: | |||
| Earnings per Year | YES | $2,272 | 1.2% |
| Hours Worked per Year | YES | 409 | < 0.05% |
| CEAS: | |||
| Earnings per Year | YES | $-3,382 | 2.6% |
| JP: | |||
| Hours Worked per Year | YES | 307 | 2.4% |
| SDS: | |||
| Earnings per Year | YES | $5,346 | 0.5% |
| Hours Worked per Year | YES | 318 | 2.6% |
Across all APSs, the annual hours worked per year are estimated to increase by 173. Excluding CEAS, annual hours worked for participants of the remaining APSs are estimated to increase by 409 and earnings per year to increase by $2,272. We hypothesized that activities under CEAS may provide access to other APSs. This, in turn, may result in participants exploring other program options, remaining out of the labour force longer, with consequent lower hours worked and earnings per year. An analysis by Advanced Education and Career Development (AECD) finds that 18.4 percent of CEAS participants go on to take other APSs. This explanation, therefore, cannot explain the poorer post-intervention labour market experience for CEAS participants.
Because fewer observations are included in the models for individual APSs, it is less likely that these models are able to detect a significant change when one exists. Significant impacts by individual program were:
For active claimants, annual earnings decrease by $2,530. Annual hours worked decrease by 158. A loss in earnings is often observed for participants in the short-term. Participation results in their absence from the labour market. During this absence, comparison group members may find a job and begin to increase their earnings.
When participants return to the labour market, their earnings may be lower than comparison group members. Participant earnings must "catch up" to earnings of those who have remained in the labour market. However, over the longer term, the rate of increase in earnings for participants may be greater than for comparison group members. The longer time frames afforded by the summative evaluation will provide more definitive proof on attributable impacts.
For reach-back clients, hours worked per year increased by 755.18
There was no impact on EI or SFI dependence in the short-term in any model.
|
Although not a primary focus of the formative evaluation, this chapter identifies impacts of the Labour Market Development Agreement (LMDA) at a community level. It should be noted that the formative evaluation addresses impacts during the LMDA's first year of implementation. In that year, considerable effort was devoted to the devolution of programs and services to the province and to ensuring that clients were served. A consequence is that less emphasis has been directed at developing the community-level linkages and infrastructure that may help promote community impacts.
Community representatives were not involved to any degree in the planning process related to the LMDA. Third party service providers have been involved to some degree. Some of these service providers are not-for-profit organizations who represent various interest groups within the community. However, it remains the case that broad community based input did not occur related to the implementation of the LMDA. Community input did occur in rural areas around Calgary after the LMDA was launched.
The LMDA has resulted in labour market impacts that will bring effects at the community level. Our findings identify statistically significant increases in the hours of work for participants under all Alberta Program and Services (APSs), under Job Placement (JP) and Skills Development Skill Training (SDS), all programs excluding Career and Employment Assistance Services (CEAS), and for reach-back clients. Earnings per year also increase significantly for all APSs excluding CEAS, and SDS.
These are significant positive findings so early in the life of the LMDA.
Findings of a statistically significant reduction in hours worked for active claimants and reductions in earnings for active claimants and CEAS participants are likely short run phenomena. Longer term impacts are likely to be more positive.
Our qualitative assessment did not find evidence of unfair competition resulting from implementation of the APSs. There were a limited number of negative comments about Training on the Job (TOJ) and Skills for Work (SW) projects. These programs may create unfair advantage if the training inputs by the partnering firm are insufficient to justify the compensation to employers for training.
The Self-Employment (SE) program did result in new business formations and in improvements to some existing self-employment enterprises. However, there was no suggestion that these created unfair disadvantage to existing firms.
The APSs may affect the mobility of participants. This effect may either be positive or negative for the community where the interventions occur. Our regression results identified no significant difference between non-apprentice participants and comparison group members with regard to the trade-off they would make if obtaining a good job meant moving from their community. As a result, no impact is expected for the mobility of clients.
The vast majority of apprentice and non-apprentice clients have remained within the communities they were living in at the time of their APS. Seventy-nine percent of apprentices have not moved and a further 12 percent have moved within the same community. Nine percent have moved from their community—4 percent to another community within a one hour drive and 5 percent to another community more than a one hour drive from the first community. Eighty-eight percent of non-apprentices have not moved since their APS. Seven percent have moved within the same community. Four percent are now in another community within a one hour drive and 1 percent have moved to another community more than a one hour drive from the first community.
|
Targets have been established for the Labour Market Development Agreement (LMDA). Human Resources Investment Branch (HRIB) has the mandate to assess achievement against these targets. The evaluation has identified a number of problems with the way the LMDA's success was being measured and reported during its first year. These topics are discussed in this chapter.
Exhibit 15 outlines the short-term targets established for the LMDA for fiscal year 1997-98.
| Active Claimants | Reach-back Clients | Total | |
| # | # | # | |
| 1. Priority EI Claimant Access | At least 65% | NA | NA |
| 2. EI Clients Return to Work (RTW) | Priority | NA | 14,481 |
| 3. Unpaid EI | $99.8 M | 0 | $99.8 M |
Although no specific targets were set for the number of participants, active claimants were to make up a minimum of 65 percent of the total number of participants. A Return to Work (RTW) target of 14,481 insured participants has been established. No breakdown is given by type of client although active claimants are to be given priority. Unpaid Employment Insurance (EI) (the difference between the individual's entitlement to regular benefits (EI Part I) and the actual pay out of regular benefits) targets are $99.8 million. This amount is identified against active claimants since EI Part I is paid to this group only.
Our study evaluates the LMDA, and focuses on its first year of implementation. Exhibit 16 identifies the short-term success that has been achieved under the LMDA between November 1, 1997 and October 31, 1998 according to data we have gathered in the study. These data are not directly comparable to the fiscal year targets identified in Exhibit 15 nor to the tracking of the calendar year success by HRIB.
| Active Claimants | Reach-back Clients | Total | ||
| Participants | ||||
| Non-apprentice | ( # ) | 9,194 | 8,189 | 17,383 |
| Apprentice (active claimant only) | ( # ) | 6,444 | - | 6,444 |
| Minus Participants Found in Both Groups | ( # ) | 68 | - | 68 |
| Total | ( # ) | 15,570 | 8,189 | 23,759 |
| (%) | 66 | 34 | 100 | |
| Return to Work | ||||
| Non-apprentice | ( # ) | 4,158 | 4,115 | 8,273 |
| Apprentice (active claimant only) | ( # ) | 5,252 | - | 5,252 |
| Total | ( # ) | 9,410 | 4,115 | 13,525 |
| (%) | 70 | 30 | 100 | |
| Unpaid EI | ||||
| Non-apprentice | ( $ million ) | 12.7 | - | 12.7 |
| Apprentice | ( $ million ) | 39.1 | - | 39.1 |
| Total | ( $ million ) | 51.8 | - | 51.8 |
Our evaluation identifies 17,383 non-apprentices and 6,444 apprentices who are active claimants in the first year of the LMDA. Sixty-eight participants appear in both groups. (We adjust totals to avoid double-counting.) Of the 23,759 participants, 66 percent are active claimants. Accordingly, the LMDA achieves its target of at least 65 percent active claimants.
Data on participants come from the Alberta Advanced Education and Career Development (AECD) data extract of non-apprentice participants, and an extract of apprentice participants who are active claimants under Human Resources Development Canada (HRDC's) Online Information System (OLIS). Client status was determined from HRDC's Longitudinal Data Base (Long File). Beneficiaries of Support for Independence (SFI) were determined using data from Alberta Family and Social Services (AFSS).
Exhibit 17 identifies the number of participants, their last intervention in the period between November 1, 1997 and October 31, 1998 inclusive, and their EI and SFI status at the time of the last intervention.
| Active Claimants | Reach-back Clients | Total | SFI Recipients of Total | |
| APS: | ||||
| CEAS | 3,381 | 2,796 | 6,177 | 100 |
| SDS | 2,314 | 904 | 3,218 | 48 |
| SDU | 728 | 572 | 1,300 | 22 |
| SE | 383 | 129 | 512 | 3 |
| SW | 623 | 472 | 1,095 | 34 |
| TOJ | 255 | 296 | 551 | 4 |
| JP | 1,510 | 3,020 | 4,530 | 100 |
| Non-apprentice Total | 9,194 | 8,189 | 17,383 | 311 |
| Apprentices | 6,444 | 6,444 | ||
| Minus Overlap | 68 | 68 | ||
| All Participants | 15,570 | 8,189 | 23,759 | 311 |
| Percent of Total | 66% | 34% | 1% |
Exhibit 18 extrapolates RTW results19 from our survey to the population of all participants between November 1, 1997 and October 31, 1998. It identifies 13,525 participants (5,252 of whom are apprentices) as having an RTW by January to early February 1998.
| Active Clients | Reach-back Clients | All Participants | |
| # | # | # | |
| APS: | |||
| CEAS | 1,161 | 1,202 | 2,363 |
| SDS | 1,395 | 331 | 1,726 |
| SDU | 269 | 40 | 309 |
| SE | 104 | 17 | 121 |
| SW | 279 | 150 | 429 |
| TOJ | 54 | 67 | 121 |
| JP | 896 | 2,308 | 3,204 |
| Non-apprentice | 4,158 | 4,115 | 8,273 |
| Apprentice | 5,252 | - | 5,252 |
| Total | 9,410 | 4,115 | 13,525 |
Restricting the HRIB December 1998 results file to those participants between November 1, 1997 and October 31, 1998 identifies less than half (6,651) this number of RTWs. Our measure:
The 1997-98 target set by the LMDA for EI clients returning to work was 14,481. HRIB identified 14,815 who returned to work in 1997/98. However, only 45 percent of them were participants under the LMDA. (The rest are carry-overs from before the LMDA.) Our results focus on participants in the first year of implementation of the LMDA and identify 13,525 as having returned to work, or 93 percent of the fiscal year target for the LMDA.
Approximately one-half of non-apprentice RTWs calculated by the evaluation are accounted for by active claimants. Including apprentices, 70 percent of the RTWs are by active claimants. The LMDA calls for active claimant RTWs to be a priority in Annex 3 to the LMDA, "Annual Results Targets". We are unable to assess whether this guideline has been adhered to since "priority" is not defined.
The HRIB data for participants under the LMDA between November 1, 1997 and October 31, 1998 suggest that $13.6 million in EI Part I benefits were not paid for these participants up to the end of December 1998. This is less than the $64.6 million estimated by HRIB in the results file to this date. The difference is the result of carry-over clients who participated prior to the implementation of the LMDA (before November 1, 1997). As noted earlier, the December results file excludes the unpaid EI for 2,000 apprentices from an earlier file, some of whom may be participants under the LMDA.
To assess the unpaid EI for participants, we identify the number of current claimants who have an RTW. We then estimate for each their total EI entitlement minus the actual EI paid under their original claim from administrative sources. The difference is the unpaid EI benefit associated with an RTW but not necessarily attributable to an RTW. (More will be said about attribution in later sections.) This unpaid EI definition conforms to the definition used in LMDA evaluations in all jurisdictions.
The total unpaid EI benefit for non-apprentices based on this methodology is $12.7 million and for apprentices is $39.1 million. This total ($51.8 million) is 52 percent of the target for 1997-98. Results are given in Exhibit 19.
| $000 | |
| Non-apprentice Participants by: | |
| CEAS | 4,171 |
| SDS | 3,041 |
| SDU | 431 |
| SE | 282 |
| SW | 725 |
| TOJ | 165 |
| JP | 3,876 |
| Total Non-apprentice | 12,690 |
| Apprentice | 39,107 |
| Total All Participants | 51,797 |
Our survey identified 631 non-apprentice participants who had enough post-intervention experience at the time of our survey to assess their RTW status and unpaid benefit values. Only 610 of these participants could be compared against the December 1998 HRIB results file, after we had excluded cases with different intervention start weeks and different Benefit Period Commencement (BPC) weeks for active claimants.
Of this sample, HRIB identified 40 percent of the non-apprentice active claimants and 46 percent of the reach back clients as having an RTW. Comparable numbers from our survey are 69 percent and 66 percent, respectively. (Note that the timing of our survey included up to one and one-half extra months in the post-APS period. Some of the difference between the evaluation and HRIB results may be as a result of this time difference.)
Exhibit 20 provides a more detailed comparison of the HRIB and evaluation RTW results. The two agree in only 57 percent of active claimant cases—33 percent when both identify an RTW and 24 percent when both do not. For reach-back clients both measures agree in 44 percent of cases—28 percent when both identify an RTW and 16 percent when both do not.
| RTW Recorded by: | Active Claimants | Reach-back Clients | ||
| Measures: | HRIB | Evaluation | % | % |
| Agree | Yes | Yes | 33 | 28 |
| No | No | 24 | 16 | |
| Total | 57 | 44 | ||
| Disagree | Yes | No | 8 | 18 |
| No | Yes | 36 | 38 | |
| Total | 43 | 56 | ||
The evaluation is more likely to identify an RTW when HRIB does not. It declares 36 percent of active claimants and 38 percent of reach back clients as having an RTW when HRIB does not. HRIB identifies 8 percent of active claimants and 18 percent of reach back clients as having an RTW when the evaluation does not. Assuming the evaluation accurately measures RTW, the HRIB method is not a good proxy for RTW.
Once an RTW is identified, the HRIB methodology (used for the December results file) calculates unpaid EI using a formula which considers the benefit rate, entitlement weeks and weeks of non-zero benefits paid, treating the latter differently depending if it is in or out of the period used to assess the RTW. The evaluation assesses total entitlement minus total benefits paid in keeping with the methodology used in the evaluation studies in all jurisdictions. For HRIB the average unpaid EI is $2,871 and for the evaluation $2,823. The evaluation finds a much higher proportion with an RTW resulting in a total unpaid EI for the group covered by this comparison of $967,417. The corresponding unpaid EI by HRIB is $553,287.
Short-term success indicators (or targets) were established for the LMDA based on:
Earlier sections have identified the findings of the evaluation against these targets. This section discusses the appropriateness of these targets and the methods used to calculate them.
Targets provide key measures related to the success of the LMDA. These measures are intended to be indicators only. They are not and should not be interpreted as true measures of success. However, their value as indicators should be based on their ability to provide unbiased insights into the likely success of the LMDA.
a) Apprentices Cause Bias
A significant portion of the achievement of targets is attributable to the activities of apprentices who are provided income support through EI Part I to attend the in-school training component of their apprenticeship. These individuals may not be unemployed under a strict definition of the eligibility criteria for APSs and are not unemployed under a common sense definition of unemployment. When they continue the on-the-job component of their apprenticeship, their return to their employer is not equivalent to an unemployed worker's RTW. Similarly, there is no equivalent to the unpaid EI benefit for an unemployed worker since there is no intention that the remaining EI benefit will ever be paid. Further, less than one-third of apprentices taking training receive support under the LMDA for that training. It is not clear that success should be attributed to the LMDA in the case of apprentices.
As a result, to the extent that the targets include apprentices, they do not represent real impacts of the LMDA. They have a built-in bias to over-state the potential impacts.
b) Success or Administrative Goal
The proportion of clients served who are active claimants may represent an administrative goal, but this is less likely to represent a useful short-term measure of success for the LMDA. Targets based on the number of clients by EI status may be a more meaningful measure of success, and a more useful measure to compare results over time.
Also, because only apprentices who are active claimants are counted as participants, they tend to distort the success measure related to the proportion of clients who are active. Data for HRDC's Online Information System (OLIS) suggest that 10,547 apprentices receive interventions. However, only 6,444 or 61 percent count toward the LMDA targets because of their active claimant status.
c) Gross Versus Net Success
It should be noted that unpaid EI benefits represent a reduction in the expenditure of EI Part I only. To achieve this reduction, an expenditure under EI Part II is being made. The expenditure under EI Part II may be less, equal to, or more than any entitlement under EI Part I. Viewed from the context of the broader EI Fund and not simply EI Part I, the unpaid EI benefit measure does not show "savings" or "benefits unpaid" to the client. Its name may be a cause of some confusion.
d) Stated Purpose Not Met
The unpaid EI measure's stated purposes are to: indicate speed of RTW; to focus resources on those early in their claim period, and to compare current to past performance. (HRIB, Accountability Framework: Methods to Measure, September 23, 1998.) These purposes may be better served by a measure of the weeks of claim used before the intervention leading to an RTW or, the remaining weeks of claim when the individual returns to work.
The Human Resource Investment Branch (HRIB) is responsible for measurement of achievement against these short-term success indicators. HRIB has continually revised its methods of measurement and its algorithms to measure success.
This section critiques those methods in use for the December 1998 results file. HRIB has subsequently revised its methods to calculate RTW and the unpaid EI measure.
We note the following problems in the methods used for the December 1998 results file.
a) Active Claimants
The method to calculate RTW for active claimants is referred to as the 12-week, 25 percent rule. Under this rule, administrative data on EI paid for the individual on claim are used to infer an RTW. An RTW is recorded for an active claimant (with an action plan or group session entered in NES, or if an apprentice entered in Benefits and OverPayments (BNOP)) with 12 consecutive weeks (or the remaining weeks on claim if less than 12) where EI benefits paid are 25 percent or less of entitlement. The calculation assumes the "trigger" under BNOP has been pulled. This trigger deducts EI dollar-for-dollar, if earnings exceed 25 percent of the benefit rate (or exceed $50 when the benefit rate is below $200). The problem is that there is no recognition that a benefit rate of less than 25 percent does not imply earnings and does not mean that the individual has actually returned to work. (There are many reasons why a benefit will not be paid, other than reported earnings from work.) The result is a potential over estimation of RTW for active claimants.
If an RTW is inferred by this formula, unpaid EI is then calculated using a formula which implicitly ignores partial benefits paid. Although the RTW calculation acknowledges EI paid over the 12 weeks of up to 25 percent of entitlement, any EI paid in this period is not deducted. The result is a potential over-estimate of unpaid EI benefits. Also, the calculation includes full benefits not paid before the RTW, regardless of the reason. And, unpaid EI does not focus on why the EI was not paid. For example, severance pay will result in unpaid EI. This is counted by HRIB although unpaid EI due to severance pay is not related to the RTW (not attributable).
b) Reach-back Clients and Active Claimants After the Claim Period
Reach-back clients do not receive support under EI Part I. As a result, their return to work cannot be assessed using the method outlined for active claimants. Instead, a follow-up survey, 12 weeks after the intervention has ended, is used to capture data on whether the individual is working. A follow-up survey is also used in the case of active claimants who are not identified as having returned to work prior to the end of their claim.
To the extent that this follow-up is not made, the method will under-report RTW. However, if a follow-up is conducted, the answer to the survey question "Are you working?" may not be equivalent to "Have you returned to work?" The survey is valid only at a specific point. It has the potential to under or over report actual RTW.
Because reach-back clients do not receive EI Part I, there is no calculation of unpaid EI benefits.
The evaluation explored alternate short-term measures that may be used in the place of HRIB measures used for the December results file. The system-derived alternate measures focus on active claimants which represent the bulk of the LMDA participants and exclude apprentices because of their absence from the December results file. (This reduced our effective sample size to 494 non-apprentice active claimant cases.)
The alternate measures refine the HRIB return to work and unpaid EI measures by using more of the data available from the administrative systems. For the RTW calculation, the alternate measure uses reported earnings or reasons for non-reporting to define work. This contrasts with HRIBs focus on benefits paid, which can be tricked by non-work reasons for reduced benefits (such as severance pay). The alternate unpaid EI measure looks at EI remaining on the claim and compares it to EI actually paid after the start of the RTW. This is an improvement over HRIB's approach which:
Using the alternate method, RTW is calculated if the claim has a termination code which identifies the remaining benefit as not paid (i.e., the claimant stopped reporting before exhausting his entitlement), or, if for 12 consecutive weeks (or the remaining weeks in the claim) the individual:
If an RTW is identified using the alternate method, unpaid EI is calculated as total EI remaining at the time of the start of the return to work, minus actual EI paid to the end of the claim. The EI remaining excludes EI which lapses (EI which could not be used within 52 weeks of the start of the claim).
Examples will demonstrate the difference between the HRIB and alternate unpaid EI measures.
Exhibit 21 presents the hypothetical example of four individuals (A, B, C and D), all with 24 weeks of entitlement and a $400 weekly benefit, an intervention in week 1, but with different weekly benefits paid over their claim. Earnings (not shown) are the cause of the reduced weekly benefits in the RTW period, resulting in an RTW calculated in week 13 for individuals A, B and C and week 1 for individual D under the HRIB and alternate methods.
| Participants | EI Benefit Paid per Week | |||
| 1 - 4 | 5 - 8 | 9 - 12 | 13 -24 | |
| A | $400 | $400 | $400 | $0 |
| B | $400 | $400 | $400 | $50 |
| C | $400 | $0 | $400 | $50 |
| D | $50 | $50 | $50 | $50 |
The formula for unpaid benefits used by HRIB for the December results file is:
BENEFIT RATE X (ENTITLEMENT WEEKS - (WEEKS OF EI PAID WHERE THE AMOUNT EXCEEDS $0 - WEEKS PAID IN THE RTW PERIOD WHERE THE AMOUNT EXCEEDS $0))
The alternate measures' formula subtracts EI benefits paid after the start of the RTW from the EI remaining, at the start of the RTW. The alternate measure focuses only on that period over which attributable benefits can occur.
The unpaid EI calculated for each individual and methods is shown in Exhibit 22.
| Participants | Unpaid EI | |
| HRIB | Alternate | |
| A | $4,800 | $4,800 |
| B | $4,800 | $4,200 |
| C | $6,400 | $4,200 |
| D | $4,800 | $8,400 |
The alternate method gives the true unpaid EI in each case. The HRIB method calculates the true value for the most simple case illustrated by individual A. The HRIB method calculates:
Not illustrated in the example, the HRIB method can also calculate too much unpaid EI by including EI which has lapsed. Again, the alternate measure will correctly measure the unpaid component of this benefit.
Exhibit 23 compares the results for the HRIB and alternate approaches for 494 non-apprentice cases. As the exhibit illustrates, HRIB identifies many more RTW in the sample but a much lower average and total unpaid EI, potentially because those identified as having RTW have not.
The alternate method identifies $860,171 as unpaid EI. This figure reasonably approximates the evaluation findings of $967,417 which:
| Unpaid EI | |||
| RTW | Average | Total | |
| (#) | ($) | ($) | |
| Non-apprentice Clients | |||
| HRIB | 254 | 2,415 | 613,516 |
| Alternate | 130 | 6,617 | 860,171 |
HRIB recently changed its methods for estimating short-term success indicators. The RTW calculations now use earnings to verify work and a period of non-reporting to infer work in keeping with the alternate methodology described in this report. This revised RTW method should provide a better assessment of RTW.
HRIB has also adopted a new formula for unpaid EI:
BENEFIT RATE X (ENTITLEMENT WEEKS - WEEKS PAID WHERE THE AMOUNT EXCEEDS $0)
Using this new formula, unpaid EI for individuals A, B, C and D in the above examples would be $4,800, $0, $1,600 and $0, respectively. The new formula implicitly considers EI unpaid only if benefits paid are at $0. The result is that the new HRIB approach calculates the correct value for the simple case provided by individual A. The HRIB method calculates:
The new HRIB method can also calculate too much unpaid EI by including EI which has lapsed. Again, the alternate measure will correctly measure the unpaid component of this benefit.
Calculation of unpaid EI would be more accurate and equally simple if HRIB adopted the alternate formula which uses actual benefits paid instead of benefits exceeding $0 and focuses on EI remaining at the start of the RTW and EI paid after this date.
|
|
| Last Modified: 2002-11-29 | Important Notices |
![[To Top]](/web/20060126233815im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}