Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

The Formative Evaluation Results of the Assessment, Counselling and Referral Strategic Initiative

Final Report

Evaluation and Data Development Branch
Strategic Policy
Human Resources Development Canada

May 1997

SP-AH041E-05-97

The formative evaluation of the Assessment, Counselling and Referral (ACR) Strategic Initiative was prepared under contract by KPMG for the Evaluation and Accountability Branch, Ministry of Education, Skills and Training in partnership with Human Resources Development Canada. This project was co-funded by Canada and the Province of British Columbia as part of the Strategic Initiatives Agreement to test new approaches to social security reform in British Columbia.

The research for this project was conducted in the summer of 1996. Since that time, a number of program elements have continued to be improved and modified.

The ACR Steering Committee would like to thank the time and contribution of local ACR committees, representatives from both levels of government and various service providers in British Columbia. This evaluation has been a long and complex process, and could not have been accomplished without the good will and assistance of the fore-mentioned.

Introduction and Background

This report sets out the findings of a formal, comprehensive, formative evaluation of the Assessment, Counselling and Referral Strategic Initiative (ACR/SI) under the Canada/British Columbia Agreement concerning the labour market components of the Canada/British Columbia Strategic Initiatives.

The objective of the ACR/SI was to test and demonstrate an enhanced assessment, counselling and referral system for people on income support. It was expected to improve linkages within and between employment programs, the individual on income support and the labour market.

Integration of service design and delivery was a key aspect of the innovation.

Ultimately, the pilot projects were intended to address the goal of moving individuals on income support (IISs) from dependence to independence and improve their long term attachment to the labour force.

For the period May 1995 to July 1996, the period covered by this review, there were at least 8,497 participants in ACR/SI components. Of this, there were at least 3,426 participants in Starting Points, the one common component. From 2% to 37% of all individuals on income support (IISs) in the 11 pilot project areas attended one or more ACR components.

The ACR/SI was conceived within a broad framework of partnership between the Government of Canada and the Province of British Columbia. The partnership components included the ACR Steering Group and nine local ACR Committees responsible for assessing client needs and service offerings in their communities. On this basis, the local committees prepared "service delivery plans" for local ACR pilot projects which were intended to enhance the locally available assessment, counselling and referral services for recipients of income assistance.

The purpose of the formative evaluation was to make an assessment of how well the ACR/SI was designed and managed, and what factors were likely to affect the efficiency, economy and effectiveness of the program. Also, the ACR Evaluation Steering Committee wanted answers to a list of specific questions about the design and operation of the overall Strategic Initiative and the outcomes of the individual pilot projects. (The list of questions is attached as Appendix A).

The evaluation issues described above were addressed through an examination of documents, on-site interviews, follow-up telephone interviews and the administration of a formative questionnaire to members of the nine local ACR Committees and the service providers involved in the pilot projects.

Based on the results of the data gathering and research methodology described above, detailed case study reports were prepared and circulated to the local ACR committees for comment. In this report, the results of the detailed case studies are summarized.

One of the primary outcomes of the ACR components was a movement to more group counselling and assessment, whether by government staff or contracted service provider. This move facilitated a broader and more immediate provision of service to IISs and, according to some of the commentaries received, enabled better services to be provided because of the group peer support effects that occur in group sessions.

The review of the pilot projects indicated that the components which were developed by the local committees followed the criteria of the ACR/SI very closely. The more significant challenge for the local ACR Committees was to develop enough components to address as many of the ACR criteria as possible, for the range of client needs.

It seems clear that the community-by-community needs assessment and implementation strategy was very effective for ensuring more integrated and coordinated approach to service delivery.

Involvement of the individual IIS in the action planning process was more intense and comprehensive than had been the case in previous programs at individual partner levels. It appeared that both the nature and level of involvement of clients in action planning processes were appropriate.

The ACR/SI gave rise to significant innovation in program delivery for recipients of income assistance, particularly because of the delegation of responsibility for design and development to the local community level, on a partnership basis.

Starting Points was the one ACR component common to all ACR pilot projects. Staff and clients rated the Starting Points program highly, as a helpful program in establishing client motivation.

It is unclear whether the motivational effects of Starting Points are lasting the sustainability of the outcomes for clients appears to depend on whether the expectations for follow-up programs and other outcomes are actually met. In cases where the seven month waiting period under Youth Works led to a reduction in the number of referrals, class sizes were reduced and some classes were cancelled.

A key finding was that many of the referrals into the initial round of ACR components were clients who were most likely to succeed in their training plans and employment efforts. It was therefore unclear to what extent the ACR pilot projects were meeting the needs of less motivated and more challenging recipients of income support. The question to be considered is whether special efforts should be made to seek out and involve IISs who have traditionally avoided interventions and contact with service providers in the communities.

Most of the tracking described in connection with the ACR components was short term in nature, focusing on time frames like six weeks, six months or one year after participants have completed their programs. These time frames are useful, but must be extended to make it possible to assess lasting outcomes given the fact that many individuals on income support "cycle" in and out of the welfare system over a period of months and years.

The overall set of challenges to the local ACR committees and the staff of the partner organizations were extreme, due to organizational and policy changes occurring within the partner organizations. The seven month waiting period rule under the Youth Works component of the new BC Benefits program was applied in some pilots and referral processes were changed. In others, the rule was not applied and the original ACR referral processes remained intact. This caused a strain in some of the partnership relationships for some ACR pilot projects. The most significant effect was the breakdown in the referral process which in turn resulted in fewer participants in ACR components and higher costs per participant for those components.

A lack of precision in the definition of what was expected, particularly in terms of employment outcomes in the short term was evident in the first group of pilots. If there are different outcomes expected over different time periods, (i.e., short, medium and long term) and if there are significant differences in the objectives for specific client sub-groups, (i.e., youth, single parents, new intakes, etc.) then these should be explicitly and clearly stated at the outset.

Breakdowns in the referral process were a regrettable feature of some of the initial pilots and efforts must be made to avoid a repetition of this. Clearly assigning responsibility or "ownership" for referral coupled with ensuring that the necessary resources (staff, time) are in place, is a vital first step in ensuring the success of a province-wide roll-out of ACR.

Due to the existence of formal organizations that extend beyond the ACR partnership, it should be realized that the partnership itself will necessarily be imperfect. In order to deal with the limitations of partnership successfully, it will be important to set an appropriate level of expectation for all of the parties to the ACR/SI, and particularly the local ACR committees.

In the final analysis, partnership at the local ACR committee level of the partner government organizations plus the efforts of the Federal/Provincial Steering Committee served to overcome extreme levels of diversity and produced important pilot projects in expanded services to individuals on income support. The effectiveness of the projects will be evaluated in a subsequent summative evaluation of the Assessment, Counselling and Referral Strategic Initiative.

Introduction

Since the completion of the research for this report some significant changes have taken place in regards to labour market training in BC. On April 25, 1997 a Canada-British Columbia Agreement of Labour Market Development was signed, providing for a shared responsibility for the employment benefits and support measures available under Part II of the Employment Insurance Act (EI). Joint decision-making will occur in the areas of planning, design and setting program and client priorities. The agreement contains a re-opening clause to allow for transfer of program and delivery responsibility to the Province.

Therefore many of the recommendations of the consultant must be viewed in a different context than had originally been assumed. As a result many of the management responses to recommendations may either change over time as the implementation of the agreement evolves, or may be deferred to the Management Committee established to administer the agreement.

Recommendations at a Business Direction Level REF:

1. Clear and specific goals and expectations for individual client sub-groups at the local level must be established for the short, medium and long term.

• The overall goal of programming under BC Benefits is the movement off government support, this is mirrored by HRDC's requirement to plan and implement a "return to work action plan".
• At the same time the case management process is part of an integrated service delivery approach which focuses on individual client needs and accountability for results through various interventions. This continues, based upon individual client needs, through the short, medium and long term until the client has found work.
  

2. The goals and expectations must be defined in terms of valid, final output measures so that changes in long term attachment to the labour force can be assessed in quantitative terms.

• An evaluation framework of the LMDA agreement is currently being developed to address final output measures. Targets have already been established for 1997/98.
• HRDC's accountability framework "From Planning to Results" for Employment Benefits and Support Measures addresses final output measures, with an emphasis on quantitative measures.
• The MoEST's STAR system is currently being developed to address quantitative measures.
  

3. Detailed employment outcomes for the ACR/SI be specified for the short, medium and long term in all of the documentation for the ACR/SI, including the Resource Material Kit for Local ACR Committees.

• The Steering Committee recognizes the potential value in developing measures or guideposts for the incremental steps to employability and independence. The work priorities for 97/98 include developing the measures for the employability skills. These measures will be generic as success will need to be client centered, building upon the individual needs and abilities.
  

4. Planned employment outcomes should also be set out in the Service Delivery Plans of the local ACR committees, in quantitative terms by sub-group.

• MoEST has a requirement for Area Referral/Placement plans and targets. HRDC currently has a requirement for local office Business Plans to be developed which identifies outcomes.
• With the advent of the LMDA planning for outcomes will be done jointly involving the Ministries of Education, Skills and Training and Human Resources and Human Resource Development Canada.
  

5. Individual plans of ACR clients should identify their planned employment outcomes within specific time frames and in relation to the ACR/SI components and other programs of assistance in which they will be participants.

• The individual "return to work actions plan" of HRDC and the "employment agreement" of MoEST identify the planned employment outcomes within specific time frames and in relation to the ACR/SI components and other interventions they may be eligible to participate in. Having the goals and objectives as measurable as specific and as measurable as possible assists both the clients and worker to track progress.

Recommendations at the Planning Level REF:

6. Include an assessment of significant local economic factors and projections which will likely affect the opportunities available to IISs in the area.

• Labour market information is seen as an integral part of the planning process, and is included in local HRCC business plans and MoEST planning processes. Future revisions to the ACR Service Delivery Plan will reflect this priority.
  

7.Involve local community economic planning organizations and stakeholders in the local economic assessment.

• Joint planning at the HRDC, MoEST, MHR and community level is required and in a number of pilots, and in HRDC business planning processes, this practice is currently being followed. This recommendation will be included in future revisions to the ACR Service Delivery Plan.
  

8. Develop a logical strategy for producing optimum results for IISs in the area taking into account the economic assessment described above.

• This is the goal of the ACR Service Delivery Plan, as noted above economic assessment will form part of the local strategic planning process.
  

9. Develop locally specific course and counselling contents for the ACR components which reflect the local economic conditions and the strategy described above.

• Overall design of programming will take place at the Canada/BC level, however it is intended that local needs and economic conditions are reflected in local courses. Specific ACR components or elements, such as Starting Points, are flexible to take into account the local as well as provincial and national economic conditions.

Recommendations at the Referral Level ref:

10. That specific and detailed eligibility criteria for each client sub-group be identified, based upon input and direction from the headquarters level of the partners to the agreement as well as assessments from the local level to take into account locally specific client and economic factors.

• Within the context of national and provincial priorities, the local planning process, and ACR Service Delivery Plan, will reflect specific local client priorities and economic conditions, and are monitored by the LMDA Management Committee.
• Eligibility for Youth Works and Welfare to Work programming is contained out in the legislation.
• Eligibility for EI benefits and measures is contained in the legislation.
• Eligibility for programming under the LMDA is defined in that agreement. The establishment of overall client priorities will be set by the LMDA Management Committee.
  

11. That all eligibility criteria be approved by the Federal Provincial Steering Committee to ensure consistency of purpose and equitable treatment of individuals and client sub-groups from area to area.

• As noted in 10, eligibility is carried out within the context of the relevant legislative authority, priorities may then be set at the LMDA Management Committee level. Local input as to client priorities are essential to meet local needs and must be identified within the local context as noted in recommendations 7, 8 & 9.
  

12. All questions concerning consistency of policy regarding eligibility and all other aspects of administration be resolved at the Federal/Provincial Steering Committee level to avoid conflicts between local ACR steering committees and their immediate partner supervisors.

• Local ACR committees were formed as the focal point for partnership development between HRDC, MoEST and MHR. Planning and delivery issues are carried out through these local committees based upon local circumstances. The Steering Committee is not a decision making body, it will continue to be involved in identifying and resolving conflicting policies, and developing strategies to deal with administration issues.
  

13. Partnership agreements be used as a mechanism to formalize roles and responsibilities in local referral processes.

• Roles and responsibilities are delineated for various functions including referral of clients, at the local level to meet local needs. Reviewing and adjusting these roles and responsibilities will be important as we move to the new arrangement under the LMDA.

Recommendations Regarding Tracking and Monitoring REF:

14. Appropriate data required to monitor achievements of the goals and objectives of the ACR/SI and the employment outcomes of clients over the short, medium and long term be specifically identified with respect to each ACR component and client sub-group, by the Federal/Provincial Steering Committee.

• Since the time of the formative evaluation, accountability frameworks have been established to address appropriate data collection. Employment outcomes are tracked by HRDC through the accountability framework for the delivery of the Employment Benefits and Measures. The evaluation and accountability framework for BC Benefits includes both MoEST and MHR. MoEST is currently developing a data collection system (STAR) that will combine all Skills Development Division tracking and monitoring systems.
• The partners will continue working together towards connectivity initiatives under the LMDA to ensure appropriate data collection. An information sharing agreement has been developed.

15. Each local ACR steering committee be responsible for designing, documenting and implementing a Service Outcome Monitoring System which collects, summarizes and reports the data required by the Federal/Provincial Steering Committee to monitor the achievement of ACR goals and objectives in the area.

• Having locally designed systems would make the current overlap and duplication of effort even more complex. As part of the LMDA, HRDC and MHR and MoEST will be involved in joint planning which will include connectivity to monitor the achievement client outcomes.
  

16. Contractors' roles and responsibilities for facilitating aspects of each local Service Outcome Monitoring System be specifically identified, simplifies, streamlined and set out in the terms of their contract.

• Contractors roles and responsibilities are set out in the contracts written for the delivery of ACR services, including data collection and reporting requirements.
• In addition HRDCV uses the contractor based Contact IV system while the Province uses the current CTAC and CMAR systems ( these will be replaced by STAR) to track information.

17. If tracking and monitoring systems break down or cannot be implemented to monitor the achievement of ACR/SI goals and objectives in an area, then consideration should be given to limiting the resources committed to those components and re-allocating those resources to other components which can be monitored and appear to be producing positive outcomes.

• It remains important that systems improvements be continued in order that tracking and monitoring can be supported for all types of investments of public funds. The basis of a client centered approach is to provide support for what is needed, type of intervention, to return/access work.

18. Barriers to client tracking and monitoring which have been identified in relation to the Freedom of Information and Protection of Privacy Act be identified and steps taken to overcome them. Otherwise, a careful re-assessment of the resources going to components which cannot be effectively monitored as a result of the legislation should be made with a view to possible re-allocating those resources.

• A new information sharing agreement is now being negotiated between the two levels of government that will enhance our ability to monitor and track client outcomes.

Streamlining the Service Delivery Mechanism REF:

19. Efforts to streamline and simplify the services on offer be made so that clients can move through their assessment, counselling, referral and related components with a minimum of duplication, travel to various sites, overlap of service and overall effort.

• The recent signing of the LMDA sets out arrangements for shared responsibility approach. One of the main objectives is to streamline and simplify the services offered to clients and reduce overlap and duplication.
• The client path should be simple but also be flexible to allow for a client centered approach to ensure the most appropriate services for clients are provided.

20. Individual program components be consolidated under one service provider at one location wherever possible to maximize service quality and efficiency from a client perspective and minimize cost from the ACR/SI partnership perspective.

• Although this approach is being tested in a number of locations consolidating service delivery is not always possible or desirable. The approach and mix of service providers is best decided at the local level, ensuring that value for funds is the focus in every contract no matter the size.

Recommendations for Contract Administration REF:

21. Fixed payment schedules in contracts be amended to include unit price components so that payments to contractors are made for services actually delivered to IISs.

• Both HRDC and MoEST contracts set out employment related outcomes for clients. Monitoring processed exist to ensure the activities are carried out and outcomes achieved in accordance with the contract.
• Unit prices, calendarized expenditures by service and payment on reports of service are all options being considered by MoEST.

22. Longer term contracts be used to enable contractors to maintain client relationships over a period which is reasonable in relation to the time required for clients to achieve their objective.

• It is agreed that longer term contracts can be used to enable contractors to maintain client relationships over a period which is reasonable in relation to the time required for clients to achieve their objectives.
• The Provincial contract review process is currently piloting the use of longer term contracts.

23. Other contract terms be established which create incentives for contractors to establish and maintain appropriate client relationships through to the achievement of the ACR/SI objectives by those clients.

• Contractors are paid to assist clients with completing the steps necessary to return or access work. Establishing and maintaining appropriate client relationships is a critical part of this process, but not the outcome.

Incentive and Reward Systems for ACR Clients REF:

24. Various incentive and reward systems for IIS clients be examined in subsequent pilot projects to determine whether significant improvements in long term attachment to the labour force can be anticipated as the result of such systems.

• Experiments related to such incentives are being conducted in other pilots, such as the study entitled " When Work Pays Better that Welfare". At this point this research needs to be reviewed in terms of the effectiveness with different clients and interventions before this would be adopted overall.

Best Practices REF:

25. A process for identifying "best practices" be defined and implemented by the Federal/Provincial ACR Steering Committee.

• A lessons learned publication will be developed from the formative evaluation report.
• The sharing of best practices is being facilitated through the co-management of LMDA, and occurs between governments and local offices, and between field and headquarters offices.
• HRDC has developed a shared practices forum on the Intranet for staff.
• ACR workshops and video conferences are being used to promote sharing.
• A video conference will be used to disseminate the research and learning from this study.
• Training in "Making Career Sense of LMI" for service providers and staff is being delivered by the ACR/LMI subcommittee in order to promote this best practice in career counselling.
• As best practices are developed, e.g. Starting Points, the Steering Committee will work to ensure that these are supported and promoted.

26. The process include methods for benchmarking and comparing the costs and quality of outcomes of competing practices.

• This can be accomplished as part of the information sharing noted in #25.

27. The process include ongoing efforts to compare successful practices from other jurisdictions.

• There is currently a practice of sharing the evaluations and studies from all Strategic Initiatives across Canada.
• HRDC is planning to use their Intranet to promote and share best practices.
• A number of websites have been established through Strategic Initiatives to promote sharing of information, an example is the resources to support "Making Career Sense of LMI".

28. Best practices identified by the process be implemented throughout the pilot projects so that the benefits from testing diverse approaches are realized and sub-optimal methods are eliminated.

• Starting Points is an excellent example of this being done. This process will be followed as best practices are identified and developed.

1.1 Background

In 1994, the Government of Canada announced its intention to renew and revitalize the country's social security system to create an environment that better rewards effort and offers incentives to work.

To this end, Human Resources Development Canada (HRDC) launched the Strategic Initiatives (SI) program to provide a funding mechanism for the federal government to work in partnership with provincial and territorial governments to test new and innovative approaches in high priority areas of employment, education and income security. Projects supported by SI are funded on a 50/50 basis between the federal and provincial/territorial governments.

Federal and provincial staff conducted informal discussions on achieving greater coordination for the delivery of services and supports to individuals on income support (IISs) to assist them in achieving independence.

A federal/provincial ACR Steering Committee was formed with Human Resources Development Canada (HRDC), the BC Ministry of Skills, Training and Labour (MSTL) and the BC Ministry of Social Services (MSS). The goals and objectives set out above were finalized and activities were identified that would coordinate and enhance assessment, counselling and referral processes and services at the local level.

The formal agreement enabling the ACR Strategic Initiative was signed in July 1995. Funding for the Initiative continues and pilot projects under the Initiative are being managed cooperatively by HRDC, the new BC Ministry of Education, Skills and Training and the new BC Ministry of Human Resources.¹

According to the original Cooperation Agreement between Canada and the Province of British Columbia, the objective of the ACR/SI was to test and demonstrate an enhanced assessment, counselling and referral system for people on income support. It was expected to improve linkages within and between employment programs, the individual on income support and the labour market.

A key aspect of the innovation was the extent of integration of service design and delivery. Enhanced assessment and additional support, as well as integration of services at the community level were intended to ensure that income support recipients are referred to programming that is appropriate, flexible and tailored to meet the needs of the very different groups and individuals on income support, including equity group members, people with low levels of education and youth.

The ACR/SI was intended to provide the necessary support and planning mechanisms to actively involve individuals on income support in the development of realistic and achievable training, as well as employment goals leading to long term attachment to the labour force.

The logic model for the ACR/SI is illustrated in Exhibit 1.2.1 below.

Exhibit 1.2.1 illustrates the underlying ACR strategy of enhancing the assessment, counselling and referral system, improving linkages, integrating service design and delivery and enhancing assessment at the community level. This strategy was intended to ensure that income support recipients are referred to programming that is appropriate, flexible and tailored to individual needs. On this basis the IISs were intended to achieve realistic and achievable training and employment goals. These in turn were intended to lead to an increase in the long-term attachment of the IISs to the labour force.

The ACR/SI is unique because it was conceived within a broad framework of partnership between the Government of Canada and two ministries of the Province of British Columbia. The Initiative is not a straightforward public sector program operated within the conventional organization structure of a single level of government. Instead, it was intended to function as an inter-governmental process that extends well beyond the usual level of joint funding and mutually exclusive responsibilities of separate governments. The ACR/SI featured a delegation of joint planning, design, service delivery and accountability responsibilities to the government partners with equal opportunity for input on the part of each of the partners.

Although a joint committee was established to coordinate the ACR Initiative, it was at the field level where much of the responsibility for the planning, design and delivery of enhanced assessment, counselling and referral programs and processes was intended to take place. Nine local ACR committees were established to create 11 pilot projects in various communities in British Columbia. The committees were not required to deliver the same services in the same way. Instead, the local committees were given the responsibility for assessing client needs and existing services in their communities, and on this basis, designing the components of their pilot projects to address specific service delivery gaps and meet local requirements. In so doing, the local committees were intended to prepare "service delivery plans". The original agreement stated explicitly that "The local ACR Committee will approve and administer local projects".

Joint efforts were required at the local level to extend formal partnership between the three levels of government. Also, a substantial amount of guidance was given to the local committees, ranging from the statement of key elements to be considered in designing their enhanced programs, all the way to specific directions about including the Starting Points program described later in this report. Properly tailored programs, diversity and innovation were intended to be the outcomes of local determination and partnership.

Additional elements to the partnership were intended. The business of providing services to individuals on income support is highly dependent upon both private and institutional service providers in the community, in addition to the levels of government involved. As well, employers and agencies involved with community economic development are critical stakeholders in the success of training and employability programs to individuals on income support. Also, key information inputs such as local labour market data are required. The linkages to these service providers, stakeholders and inputs were intended to be enhanced through ACR and a very broad partnership was therefore to be created out of the formative activities of the local ACR Committees and the federal/provincial Steering Group.

Although not stated explicitly, it is clear that the partnership program described above was based on an underlying plan to bring together more of the collective knowledge, experience and wisdom from the direct service delivery level of all three levels of government and the community, to address the common goal of moving individuals on income support from dependence to independence and long term attachment to the workforce. The ACR Strategic Initiative was intended to produce enhanced results from combined efforts uninhibited by the kind of rigidities which can result from a more conventional assignment of mutually exclusive responsibilities between separate government organizations.

The objectives of the formative evaluation were:

• To address questions set out by the ACR Evaluation Steering Committee at the outset, to the extent possible in the formative evaluation phase (Appendix A).
• To identify and describe the most significant formative evaluation issues.
• To obtain all the information needed to carry out a summative evaluation.

The information in this report is presented at a summary level of detail and may not apply to every pilot project due to the diversity of the projects. Readers should refer to the detailed case study reports on the individual pilot projects for more complete information.

This report is organized into five chapters. Following Chapter 1 Introduction, the second chapter of the report sets out the formative evaluation approach. This chapter includes a discussion of the ACR program evaluation process, the nature of formative and summative evaluation methodologies, the methodology for the ACR formative evaluation and some of the limitations that arose in conducting the formative evaluation.

Chapter 3 sets out key findings of the formative evaluation including a listing of the elements to be used in designing components, a description of a pilot project, a summary of all of the ACR pilot projects, a brief discussion of the sequencing of components and referrals and finally a more detailed review of key findings with respect to the Starting Points component.

Chapter 4 presents a detailed discussion of formative evaluation issues. This chapter discusses the extent to which the criteria for ACR were met by the pilot projects, the extent of service provided, the way in which the components met needs in the communities, the planning process, the action planning process for IISs, monitoring and follow-up activities of the ACR components, the extent of innovation in the components, and finally the potential for achievement of the ACR Strategic Initiative objectives.

In Chapter 5, the conclusions from the formative evaluation and the pilot projects are discussed. The conclusions focus on the effectiveness of the ACR partnerships, a consideration of how best practices are identified, and a listing of potential best practices and recommendations.

2.1 The ACR Program Evaluation Process

The Cooperation Agreement which established the ACR Strategic Initiative required that a Management Committee be established to manage the implementation of the Initiative. As well, a federal/provincial Evaluation Steering Committee was to monitor, review and evaluate the effectiveness of the Initiative. A framework for the evaluation was to be developed to determine the effectiveness, efficiency and potential to contribute to social security reform.

Pursuant to the evaluation requirement described above, the ACR Evaluation Steering Committee developed terms of reference for the program evaluation and issued the terms of reference through a Request for Proposal process.

The program evaluation requested by the ACR Steering Committee was to be comprised of a "formative evaluation", a "baseline data study" and, in a separate engagement to be conducted at a later date, a "summative evaluation".

The "formative evaluation" was to address a series of questions determined by the ACR Evaluation Steering Committee. These are set out in Appendix a to this report. This report focuses on responding to these questions and reporting other formative evaluation issues identified in the course of the evaluation.

A formal, comprehensive program evaluation is conducted by carrying out two conceptually different, yet inter-related evaluation exercises known as the "formative" and "summative" evaluations.

The formative evaluation focuses on the conceptual, organizational and operational aspects of a program and, through detailed research, data gathering, study and evaluation, assesses the strengths and weaknesses of those program aspects. In this way, an assessment is made of how well the program has been designed and managed, and what factors will very likely affect the efficiency, economy and effectiveness of the program.

The second evaluation exercise is the summative evaluation. In the summative evaluation, specific, appropriate research methods are used to actually measure the outcomes or impacts of a program in relation to the originally intended impacts, goals and objectives. The summative evaluation does not focus on the operations or processes carried out within a program. Rather, it focuses on the end products, the ultimate effects of the program. The summative evaluation addresses the question of effectiveness directly by attempting to quantify the effects of a program and draw conclusions about the relative value of those effects.

In order to evaluate a program fully, it is necessary to conduct both a formative and a summative evaluation. As well, it is usually advantageous to conduct the formative evaluation first, because the knowledge gained will produce not only the insights described above, but will also play a key role in the design of appropriate summative evaluation methodologies.

In the case of the ACR Strategic Initiative, a detailed formative evaluation was extremely important, because the actual "program" consisted of 11 pilot projects which were designed locally and were therefore different from project to project. Enough information about the individual pilot projects had to be obtained in order to be able to consider the formative issues and design a proper summative methodology.

The research and data collection methodology for the formative evaluation consisted of a literature research, a documents review, on-site interviews, follow-up telephone interviews, and the administration of a formative questionnaire to members of the 11 local ACR Committees and the service providers involved in the pilot projects.

The formative evaluation questionnaire was developed by combining questions designed to obtain fundamental descriptive information about the ACR pilot projects with:

• Questions of the ACR Evaluation Steering Committee that could be addressed in a formative evaluation.
• Questions that were needed to address broader aspects of a formative evaluation.
• Also, the questionnaire was tailored to fit the characteristics of each individual pilot project and was faxed to the interviewees in advance of the on-site visits.
• In each pilot project, the evaluation team met with as many of the members of the local ACR Committee as possible, as well as representatives from the service provider organizations under contract. The questionnaire described above was completed either during these sessions or afterwards by project representatives who then submitted the questionnaire back to the evaluation team. Follow-up telephone interviews were conducted. The follow-up telephone interviews ranged from 1/2 to 1 1/2 hours.
• Based on the results of the data gathering and research methodology described above, detailed case study reports were written for all the pilot projects in a separate volume.

Several significant limitations to the research and data gathering methodology should be understood to fully appreciate what was discovered and what could not be assessed during the formative evaluation.

First, there were limitations to the amount of time and effort the local ACR committee members and others at the pilot project level could devote to the evaluation effort. The limitations stemmed from the fact that case loads and overall workloads at the field level of the three government partners to the Cooperation Agreement were extremely high and the additional capacity to participate in a program evaluation was very limited.

Second, "evaluation exhaustion" may have been a factor at the field level because a number of reviews and evaluations had been conducted and some of the respondents to the formal ACR evaluation had already responded to similar kinds of questions asked for different purposes by other stakeholders.

Third, fundamental changes to the basic business, organizational structures, roles and responsibilities of the government partners were being implemented at the time when the pilot projects were being developed and during the period of the evaluation. The impacts of these changes were unfolding at the field level almost daily in a manner that made pilot project implementation more challenging and evaluation efforts more difficult.

The result of all these factors was that the evaluation team would not be able to administer the questionnaire described above in a completely rigorous manner. First, it would not be possible to obtain responses from all pilot project representatives. Second, it would not be possible to obtain complete responses in every instance. Third, due to the differences in pilot projects, the actual meaning and relevance of the questions to the respondents would vary and as a result, it would be difficult to develop generalizations about the issues. Finally, it was not possible to follow up the responses in detail, or to obtain more detailed explanations or varying views on the same issues.

The ACR/SI was not designed to be a single, standardized approach to assessment, counselling and referral for individuals on income support. It was intended to be a "strategic initiative" in which a number of government and other stakeholders came together at various levels of partnership, to design, develop and test various approaches to delivering certain program elements in 11 pilot projects.

Within the overall ACR logic model described in the previous section, the pilot projects were intended to incorporate up to five elements specified in the Cooperation Agreement, as refined by the ACR Steering Group. The five elements are listed below:

1. Starting Points group assessment process This program element is designed to positively motivate and support people on income assistance in developing a "First Steps" action plan towards employment. It is based on a client-centred approach in which the participants themselves assess their barriers and strengths and begin to formulate their own individualized "First Steps" plan of action towards employment.
2. In-depth group assessment/orientation This program element is designed as an optional first component for a longer term training program. It is intended to be particularly well suited to people who face multiple employment barriers who are likely to require more than one intervention in making the transition from welfare to work.
3. Diagnostic assessment This element is designed for individuals on income assistance where the assessment of the barriers preventing them from participating fully and satisfactorily in the labour market requires expertise that goes beyond the scope of MoEST or HRDC staff and/or funded agencies/institutions. These services will generally be provided by a professional diagnostician and would be provided to people on income assistance with specific physical, psychological and/or learning disabilities.
4. Group career planning This program element is designed to build on the "best practices" in group career planning that have proven effective in assisting income assistance recipients in identifying and building on their career goals.
5. Learner support This program element is designed to provide people on income assistance with the problem solving and crisis management skills, links to community agencies and employers, and practical support that they require to reduce the personal/social barriers preventing them from achieving their employment goals.

The logic models for these elements are set out in Appendix B.

In addition to the elements described above, local committees could test other modifications and innovations in designing the components of their pilot projects.

A description of the Burnaby pilot project is set out below to illustrate how one local ACR committee designed the actual ACR components.

3.2.1 Group Orientation Session

In Burnaby, a Group Orientation session (formerly known as Group Service Needs Determination) was implemented. The Group Orientation Session is a 1.5 hour introductory session held twice a week in which individuals on income support are given comprehensive information on the range of training services available in Burnaby. The objective of these sessions is to enable IIS clients to assess their own needs and self-refer to other ACR services. The intent is to provide the tools and information required.

3.2.2 Starting Points

As in most of the other ACR pilots, the Starting Points component in Burnaby adheres to the curriculum developed by Dr. William Borgen. The program is designed to assist those clients who have experienced barriers in participating in the labour market or who have been on income assistance for some time.

Starting Points is a discriminative assessment carried out by the IIS client with input from the counsellor. It is designed to help the client identify and assess employment barriers/challenges associated with five employability dimensions, to set employment-related goals and action plans.

3.2.3 Career Planning

A key barrier confronting many IIS clients is an inability to focus their efforts on a career or job area. The primary goal of the Career Planning component is to guide clients through the process of self-evaluation and career assessment necessary to research, evaluate and determine appropriate career options.

3.2.4 Career Information Resource Centre

Officials in Burnaby noted that clients require career and labour market information in order to develop realistic educational and career plans. IIS clients also lack many of the material resources and facilities to implement their action plans or conduct job searches.

The Career Resource Centre is designed as a self-help site where a wide array of support services are made available to IIS clients. The inventory of material available includes labour market and career decision-making information. There is also material on job search and resume guides, advice on writing covering letters, brochures and calendars detailing courses available at colleges, universities and private training institutions. Computers, fax machines, photocopiers, access to the Internet, telephones etc., are also available to IISs on a drop-in basis.

3.2.5 Career Preparation for ESL Clients

For a limited number of immigrant clients with professional qualifications, lack of English fluency coupled with a strong accent is a significant employment barrier. Burnaby's ESL program is designed to aid immigrant social assistance recipients who have intermediate to advanced levels of English proficiency and post secondary/professional training. Each intake will assist ESL clients to further develop their English communication skills in order to overcome their employment search barriers and to enter the labour market. Clients work with a speech therapist who helps with accent reduction. There is also assistance with the translation and evaluation of foreign credentials for the Canadian labour market.

The ACR pilot project in each area turned out to be quite different, except for the one common element, Starting Points.

Exhibit 3.3.1 lists the project components that were selected in each pilot project community.

View Exhibit 3.3.1:
ACR Pilot Project Components

3.3.1 Fort St. John

In Fort St. John, the ACR pilot project consisted of four components, Starting Points, Enhancement of Assessment and Referral, College Preparation and a Support Centre.

The Starting Points program in Fort St. John was somewhat unique because it was delivered by a Training Consultant at MoEST and an Employment Counselor from HRDC, rather than by a contracted service provider. Also, the program is delivered in two consecutive half-day sessions whereas most of the pilot projects deliver the Starting Points program in two half-day sessions with a longer break in the middle.

The Enhancement of Assessment and Referral component consisted of a specific training contract to provide Financial Assistance Workers in MSS with additional training in assessing and referring individuals on income support.²

The College Preparation component provided 25 hours of instruction a week for 15 participants who were interested in pursuing further education.

The Support Centre provides two separate services, support for clients with personal barriers to employment and employment services for those individuals who are ready to enter the job market.³

3.3.2 Campbell River

The Campbell River ACR Pilot Project consisted of Starting Points and three other components. The Campbell River pilot seemed to focus more on helping IISs to move towards employment and independence, than into other interventions.

The Starting Points component was administered according to the "Borgen model", with some minor modifications discussed in section 3.5.

The Learner Supports - Taking Next Steps component was an extension of a previously established, non-ACR program, Job Project. IISs who had completed a the former program could move on to Taking Next Steps and review and adapt their skill development plans in a group format.

Campbell River's third program component was Moving Forward. This is a self-directed program in which participants upgrade their academic skills and/or literacy skills over a maximum six month period. The program also includes other life skills and a computer workshop.

The fourth program component was Work Experience. This program focused on planning for work experience, identifying appropriate work experience, developing a training plan and other activities focused on jobs.

3.3.3 Surrey

In Surrey, the first ACR component was a Job Action Workshop. The purpose of the workshop is to assist participants to formulate a very practical and usable action plan. This workshop has three phases and two optional modules. Phase I is similar to Starting Points but focuses more on jobs. Phase II is only for clients who were unable to develop a realistic action plan after the completion of Phase I. It is restricted to IISs who have been on income assistance for ten months. Phase III is a group follow-up session held four weeks later for graduates of the program. The optional modules for Job Action Workshop are Vocational Diagnostics and Job Development.

The second Surrey component was an "English as a Second Language" (ESL) program. In this program, language skills were diagnosed and referrals were made to overcome language and employment barriers often encountered with new immigrant IISs.

The third component was Starting Points. In Surrey, Starting Points was delivered in-house by the Training Consultants of MoEST. Starting Points was not initially chosen as a component by the Surrey Committee because of some difficult early experience with the program but after learning of the success experienced by other pilots, a Starting Points program was initiated.

3.3.4 Prince Rupert

In Prince Rupert, three ACR components were implemented, Starting Points, Career Planning (Career Options), and a Support Worker.

The Starting Points program was delivered in-house, by staff from MoEST and HRDC.

The Career Options program was delivered by North West Community College. The Career Options program was intended to provide additional services to Starting Points graduates. The objective of the course was to assist in the development of a skills plan, to increase job search skills, to decrease barriers, and to ensure adequate skills development to assist clients in successful completion of their skills plan.

The third component in Prince Rupert's ACR pilot project was the Support Worker program. In this component, graduates of the Starting Points and Career Options components could obtain one-to-one counseling or attend group sessions that would assist with a successful transition to the workplace. Through the services of the Support Worker, individuals in receipt of income assistance could improve their personal problem-solving and crisis management skills, and obtain insights into personal barriers and how to overcome them.

3.3.5 Nanaimo

The local ACR committee for Nanaimo implemented Starting Points and four other ACR components.

In the Career Directions component, group sessions were held to assist participants in the creation of a realistic and attainable action plan for employment. This component was designed for IISs experiencing multiple barriers to employment and later focused on youth who have been on income assistance for more than seven months, (i.e., to conform with the provincial BC Benefits program).

In the Job Talk component, employable individuals on income support attended group sessions to improve their communication skills necessary to succeed at work, at home and in the community. The objective was to overcome thinking and perceptual difficulties that constituted significant barriers.

The Job Seekers component focused on employable individuals in receipt of income assistance at the outset but later focused only on youth on income assistance for more than seven months (again, to conform with the provincial BC Benefits program). This component was also conducted in group sessions of three weeks duration in which clients were assisted with their job search and were provided other necessary skills and information needed to look for work.

In the final component, the Labour Market Information Pilot was designed to enable organizations dealing with the unemployed in the Central Vancouver Island area to go online using the Internet, and employ this and other methods to promote communication, cooperation and coordination among agencies that deliver employment-related programs and services to the unemployed.

3.3.6 Abbotsford

The local ACR committee in Abbotsford retained the services of an independent contractor to deliver Starting Points. Two other ACR program components were implemented.

In the Pre-Employment Program, graduates of Starting Points experiencing multiple barriers to employment were assisted to obtain and enhance generic skills that will aid them in successfully participating in training, finding jobs, keeping jobs, and dealing with personal issues that impact on employment. This program was conducted on a group basis over a six week period. It was eventually abandoned and replaced with a shortened version, Career Decision Making described below.

The Career Decision Making component was made available to Starting Points graduates experiencing multiple barriers to employment. In this ACR component, decision-making skills were provided in order to address educational, training and employment needs of clients, and to increase client self esteem.

3.3.7 Nelson

The Nelson ACR Committee implemented two Starting Points components, one in Nelson and one in Nakusp.

3.3.8 Parksville

In Parksville, seven ACR components were implemented including Starting Points.

A Therapeutic counseling component was implemented in which individuals on income support with personal life problems that interfered with their ability to seek employment could attend one-on-one counseling sessions to a maximum of ten sessions. The purpose of these sessions was to assist participants in the resolution of personal life problems interfering with their ability to seek employment.

In the Literacy Services component, individuals on income support in need of literacy and numeracy training could obtain one-on-one assistance from volunteer tutors. Scribe services were provided, as well as assistance with problem-solving, information requests and referrals to other agencies. Diagnostic services and tutoring services were also available.

In the Job Action component, clients who had received income assistance for more than seven months could attend groups sessions of three weeks in duration in which they were assisted with their skills and abilities to find employment.

In the Career Exploration component, Starting Points graduates could attend group sessions lasting one week to identify career and employment goals and develop concrete action plans for achieving those goals.

In the Vocational Counseling component, Starting Points graduates could attend one-on-one sessions as needed to obtain more individualized career and employment counseling services.

In the Labour Market Information pilot, organizations dealing with the unemployed in the Central Vancouver Island area were assisted in using the Internet and were encouraged to communicate, cooperate and coordinate more in the delivery of employment-related services to the unemployed.

3.3.9 Kamloops

The local ACR committee in Kamloops implemented Starting Points and two other ACR components.

In the Compass component, IISs who completed Starting Points could attend group sessions to obtain further assistance for working through pre-employment issues and in developing an employability action plan.

In the Employment Support Worker component, graduates of the Avanti Restaurant Training Program could obtain individual counseling to support their job search and self-marketing efforts. The purpose of this counseling was to assist participants in their transition from the training program to the workplace.

Variations occurred in the way IISs could move through the ACR (and non-ACR) program components of a pilot project. An example of a "non-sequential" approach is described below.

In the Campbell River ACR pilot, the program components were not designed to be sequential (i.e., IISs are not expected to start with Starting Points or move sequentially through the remaining programs). Each program is completely independent, although there may be circumstances where a client will complete one program and move on to another one that seems appropriate. For example, in the action plan that is completed at the end of Starting Points, a client may identify a need to improve his or her academic level and may therefore request to be referred to Moving Forward. Exhibit 3.4.1 depicts this pattern.

View Exhibit 3.4.1:
Sequence of ACR Program Components - Campbell River

In other pilot projects, more of a sequence was developed for IISs to follow. Usually, Starting Points or a preliminary Orientation Session was the first component that an IIS had to attend. Referrals that were tailored to the specific needs of the IISs were then made according to the action plan produced by the IIS in the Orientation Session or Starting Points.

Starting Points was the only common component in all the ACR/SI pilot projects. It is conducted on a group basis. It is an important innovation that is expected to produce more positive results for more persons in receipt of income assistance than previous, conventional, one-on-one counselling programs.

3.5.1 Objectives, Design and Delivery

Starting Points was initially developed by HRDC in collaboration with the design team of Marvin Westwood, Norman Amundson and William Borgen of UBC.⁴ The program was originally designed to serve the needs of HRDC clients but was later adapted to address the specific needs of IISs.

According to its designers, Starting Points was expected to meet the following objectives:

• To assist participants to identify and assess employment barriers/challenges, associated with five employability dimensions.
• To assist participants to understand their emotional reactions associated with unemployment.
• To assist participants to identify assets/strengths to overcome barriers/challenges.
• To set an employment-related goal.
• To develop accompanying action plans to achieve participants' goals.
• To acquaint participants with MoEST and other services available to them.

The Starting Points program was intended to be delivered in two half day sessions of approximately 3.5 hours each.⁵ The recommendation was that there be at least two days between the two sessions in order to allow sufficient time for the participants to complete their homework. A follow-up session was to be held for three hours, one month after completion.

The original agenda for Starting Points is attached in Appendix C. Additional information about each pilot project's Starting Points component is provided in Appendix D.

Eight of the eleven pilot projects chose to contract the delivery of Starting Points to an external service provider. The reasons for this decision varied but tended to focus on a lack of internal resources to facilitate the sessions.

The remaining three pilot projects delivered Starting Points in-house and the responsibility for facilitation was shared between a Training Consultant and a representative from HRDC (Employment Counselors or Program Officers).

Most of the pilot projects adopted the original design for Starting Points. There were, however, a few changes. The West End pilot, for example, chose to deliver life skills workshops on the day between the first Starting Points session and the second. Also, most of the pilots chose to run the two sessions one day after the other, although two others chose to have one full day break (Abbotsford and Burnaby), and one pilot (Nelson) delivered Starting Points all on the same day.

Contrary to the original design, none of the pilots appear to have a two-day or longer break between sessions. One reason given for the minimal breaks between sessions is that IISs are more likely to attend both sessions if the sessions are held close together. Another reason mentioned was that having the sessions back-to-back made it easier for single parents to arrange for day care.

The method and frequency of follow-up sessions also vary across the pilot projects. Three of the pilots deliver group follow-up sessions (Fort St. John, Nelson, and West End); five others provide individual one-to-one phone or in-person interviews (Abbotsford, Burnaby, Campbell River, Nanaimo, and Prince Rupert).⁶ Most of the facilitators suggested that the transient nature of the IISs made it difficult to conduct these follow-up activities. The time intervals between follow-up activities also varied considerably, with the first follow-up contact ranging from two days (in Nelson) to six months after completion (in Nanaimo).

3.5.2 Source of Referrals into Starting Points

The source of referrals into Starting Points varies somewhat, but most referrals originate from FAWs and/or Training Consultants. In pilots that offer an orientation session first, (i.e., Burnaby and West End), IISs voluntarily register for Starting Points after the orientation session. Referrals into the orientation sessions originate from the FAWs and Training Consultants.

View Exhibit 3.5.2.1
Source of Referrals into Starting Points

3.5.3 The Number of Referrals Versus Starting Points Attendance

Attendance rates for Starting Points ranged from 53% to 67%, in relation to the number of IISs initially referred into this component.

Two of the highest attendance rates for Starting Points were found in Burnaby and Vancouver West End where Starting Points participants choose to register at the end of the Orientation Session. In both cases, the Orientation Session and Starting Points are conducted by one contractor. Screening of participants occurs at the Orientation Session and the result is higher attendance rates for the Starting Points component.

Prince Rupert also reported high attendance rates. In this pilot, FAWs refer employable IISs to the Training Consultants who in turn refer a portion of these potential clients into Starting Points.

3.5.4 The Referral Process and the Freedom of Information and Protection of Privacy Act

Contractors in a number of pilot areas voiced their frustration at being unable to contact IISs who were referred to one of their ACR program components before the individuals actually registered. The provincial Freedom of Information and Protection of Privacy Act (the Act) was cited as the prime reason for this restriction.

As a result of the Act's requirements, ministries are unable to share personal information with an external contractor unless the individual in question consents to this disclosure. Under the current referral process, which usually involves ministry letters of referral to IISs, there is no opportunity to obtain this written consent until the IIS takes the initiative to contact the contractor. Consequently, contractors were unable to contact IISs prior to registration to encourage them to attend a Starting Points session.

The impact is that the number of referrals versus the number of IISs who choose to register for Starting Points is outside of the control of the contractors. If the referral process could include consent from the IISs to release their names, telephone numbers and addresses to a service provider contracted to provide Starting Points sessions, the contractor would be able to contact these individuals and encourage them to attend a session. This contact may result in an increased attendance rate, although in at least one pilot (Nanaimo), reminder calls had little affect on attendance.

The pilot projects faced a similar difficulty in their efforts to share information between the contractors and the FAWs/TCs. The action plans provided to the Starting Points contractors were considered personal information, therefore requiring the consent of the individual prior to sharing this information with the FAWs/TCs. To deal with this, a "Release of Information" form was created for the pilot projects. Every ACR participant is required to complete this form so that the action plans and attendance records can be shared between the contractors and the FAWs/TCs.

3.5.5 Comparison of Starting Points Cost Per Participant

Total costs and accurate numbers of participants were not available from every pilot project. However, it is useful to compare the costs of delivering Starting Points for the five pilot projects that were able to provide some information. Exhibit 3.5.5.1 presents the estimated cost per participant for delivering Starting Points, in-house. Although three pilots chose this delivery method, we were only able to obtain information from Fort St. John and Prince Rupert. The dates provided in this exhibit reflect the time period over which the numbers of participants attended Starting Points. The average cost per participant is estimated to be $130.

It should be noted that this is a rough estimate based on salary costs for delivering Starting Points. To be more accurate, other expenses should be included (e.g., office overhead, etc.). These expenses were unavailable and considered to be insignificant compared to the salary costs.

In-house delivery is only practical in pilot areas where representatives from either MoEST or HRDC have sufficient time to facilitate the required sessions.

View Exhibit 3.5.5.1
Estimated Cost Per Participant for In-house Delivery of Starting Points

In Exhibit 3.5.5.2 below, the estimated cost per participant for contracted delivery of Starting Points is presented.

View Exhibit 3.5.5.2
Estimated Cost Per Participant for Contracted Delivery of Starting Points

On average, contracted service delivery appears to be more costly compared to in-house delivery. Without more details on the cost of in-house delivery, and without an assessment of the impact of each of these delivery methods, the value of this comparison is limited but it should be considered in more detail for future planning purposes.

For contracted delivery, the most cost efficient delivery of Starting Points to date appeared to be in Abbotsford ($160 per participant). This pilot had the smallest budget and the largest number of participants. The most costly pilot project appeared to be the Nelson pilot where the referral process essentially collapsed, resulting in few participants.

3.5.6 Achievement of Intended Results

For the most part, the Starting Points component in each of the pilot projects is working as intended and the short term objective of developing action plans for participants is being achieved. The waiting period for Starting Points tended to be significantly shorter than for IISs waiting to meet with their Training Consultant or attend other programs.

It is unclear whether some of the more ambitious objectives (i.e., increased self-reliance, motivation or communication skills) have been met, however, the anecdotal comments made by contractors and ACR Committee members suggest that increased motivation may reasonably be expected of a Starting Points participant, for a short period after completion of the project.

It is important to note that the objectives of Starting Points do not include long-term employment. This "first step" needs assessment program cannot, in most cases, be viewed as the only direct, causal link to employment outcomes for participants. For the vast majority of IISs, a number of other interventions may be required prior to employment. These limitations should be considered in management decisions and during a summative evaluation.

3.5.7 BC Benefits and the Starting Points Referral Process

It is clear from this evaluation that the implementation of BC Benefits caused a great deal of change for clients, program administrators and service providers. Many of the people interviewed for this study suggested that waiting seven months before referring clients to Starting Points affected both the nature and number of referrals. Indeed, whether or not FAWs contributed significantly to the number of referrals appeared to depend on whether or not the IISs could be referred at the point of intake. In pilots where IISs were referred directly from intake (e.g., Fort St. John and Burnaby), the majority of the referrals originated from FAWs. In pilots where BC Benefits was implemented and clients were not referred until after they had been on assistance for seven months or longer, the number of referrals from FAWs tended to drop dramatically and more referrals originated with the TCs.

As an example, at the time of our site visit, the FAWs in Campbell River had not referred any clients to Starting Points and were not likely to do so due to workload pressures. In Nanaimo, the number of referrals, and therefore participants, dropped dramatically after the implementation of the BC Benefits eligibility criteria in May 1996 (from 38 participants in April to 17 in May). The sessions were running well below capacity, averaging six participants per session. In Nelson, the entire referral process collapsed at one point as a result of the implementation of the BC Benefits eligibility criteria.

The result has been a decrease in the cost efficiency of the Starting Points component. Also, with some pilots imposing minimum class sizes, a number of classes had to be cancelled, leaving the IISs waiting for a session to be held once enough participants were identified.

This section of the report sets out key findings in relation to the original questions of the ACR Evaluation Steering Committee, together with other matters identified in the course of the formative evaluation of the ACR/SI.

"In what ways do the ACR projects reflect the criteria established for the strategic initiatives?"

The review of the pilot projects indicated that the components which were developed by the local committees followed the criteria of the ACR/SI very closely. Indeed, the criteria were broad enough that there was room for a substantial amount of variation in components to address the five fundamental elements of ACR described previously. The more significant challenge for the local ACR Committees was to develop enough components to address as many of the ACR criteria as possible, for the range of client needs in the area. This challenge was more significant in some of the rural communities where fewer service providers and less expertise were available to develop the programs required.

"To what extent have the ACR pilot projects met the need for enhanced assessment, counselling and referral services in the communities?"

One of the primary outcomes of the ACR components was a strengthening of the counselling process, from one-on-one counselling at the partner level of government to group counselling and assessment, whether by individual government staff or contracted service provider. This move to group assessment, counselling and referral facilitated a broader and more immediate provision of service to IISs and, according to some of the commentaries received, enabled better services to be provided because of the group peer support effects that occur in group sessions.

The quantitative results for the ACR/SI from inception to July 1996 are somewhat unclear because difficulties were experienced by some of the local ACR Steering Committees and service providers in providing the numbers of participants for the various ACR components. The number of participants that were reported during the formative evaluation are set out in Appendix F. The dates of participation for the participants are set out in Appendix G. The information that was available indicated that between 2% and 32% of all IISs in a given pilot project area attended at least one ACR component.

The participation rates represent early results of the ACR/SI. The period covered by these results, from April 1995 to July 1996, should be regarded as the start-up period for the projects. Accordingly, the participation rates reflect not only the ability of the projects to serve recipients of income assistance but also reflect some of the initial challenges experienced in the start-up period of the program. In any case, the relative proportion of IISs who have received ACR services in the timeframe described above is significant and can serve as the basis for further analysis in a summative evaluation.

It is somewhat difficult to comment on the extent to which the projects have met the need for enhanced assessment, counselling and referral services in the communities. Quantitative assessments were not fully performed in the planning process so that target numbers of placements for individual categories of IISs were not set in every case and are not therefore available for use in measuring actual results.

It is expected that the numbers of participants in the ACR components will increase substantially in the period following June 1996, as referral and service delivery processes are refined and knowledge about the components spreads throughout the IIS target population.

One unresolved issue in the area of extent of service is the question of targeting ACR components to the multiple barriered population that may not attend referrals or may not be identified through self-referral or other processes. The key question is to what extent should special efforts be made to seek out and involve IISs who have traditionally avoided interventions and contact with service providers in the communities. The first round of ACR pilot projects tended to serve those IISs most likely to attend intervention programs and seek employment. Others who traditionally avoid such involvements were not necessarily contacted or served in the first round of pilot projects. During the formative evaluation interviews, some contractors indicated that this was appropriate because it was more cost efficient to serve the needs of motivated and committed participants. It is therefore unclear to what extent the ACR pilot projects are meeting the needs of less motivated and more challenging recipients of income assistance.

As a strategic initiative, ACR/SI did not specify what programming was to be put in place at the local level. Instead, it identified high-level goals and objectives, more detailed program elements and a two-tiered committee structure to develop the actual program components at the field level.

The planning process for the local pilot projects began with the preparation of a "Client and Agency Profile" for the area, in order to determine what kinds of training and employment programs were needed and available in the community. Following this, a "Service Delivery Plan" was to be developed that outlined the program and process elements to be jointly developed by the local ACR Committee.

4.4.1. The Community-by-Community Needs Assessment and Implementation Strategy

"How effective was the ACR community-by-community needs assessment and implementation strategy for ensuring a more integrated and coordinated approach to program delivery?"

During the formative evaluation all of the client and agency profiles and service delivery plans of the 11 pilot projects were reviewed by the evaluation team.

The client and agency profiles appeared to be comprehensive with respect to an analysis of IIS client populations and categories in the local communities. As well, the review of existing services available and service providers was also comprehensive in each pilot project.

The results of these assessments were comprehensive listings of IIS populations and sub-groups, as well as contracts and services available. These were taken into consideration in the development of the service delivery plans.

As a result, it seems clear that the community-by-community needs assessment and implementation strategy was very effective for ensuring more integrated and coordinated approach to service delivery. Since the local ACR Committee represented each of the partners to the strategic initiative, and since they participated in the inventorying of all available services irrespective of which agency provided them, they necessarily developed an understanding of all of the local program delivery components on an integrated and coordinated basis.

4.4.2 The Service Delivery Planning Process

"To what extent have the service delivery plans closed the gaps that exist between the needs for and availability of resources?"

In order to answer this question the evaluation team reviewed the original service delivery plans of each of the local ACR committees in relation to the inventory of service delivery gaps they prepared, to determine whether the design of the individual components would address one or more of the gaps identified. As well, the operation of the ACR components was discussed with local ACR committee representatives and service providers, to further determine which service gaps were being addressed. The evaluation team then prepared a table of service gaps for each pilot project and tabulated which ACR component addressed which gap. The results of this analysis are presented below.

4.4.3 Service Gaps Addressed by ACR

In the Exhibit 4.4.3.1, service gaps which were addressed by one or more of the ACR components in a local pilot project are listed.

With respect to the items in Exhibit 4.4.3.1, progress has been made in addressing many of the service gaps which the local ACR committees identified at the outset of the planning process. However, it is important to understand that the items listed above have not been resolved by the operation of the ACR components or dealt with uniformly from pilot to pilot. Rather, improvements have been made by one or more components in a pilot project with respect to one or more of the items described above.

More progress has been made in some areas than others. For example, under the category "Integration of Service Design", in Exhibit 4.4.3.1, a very substantial amount of progress has been made as a result of the overall partnership of government agencies involved. As well, substantial progress has been achieved in the area of developing client-centred programming.

However, in the areas of "Enhancement of Programs and Services", shown in Exhibit 4.4.3.1, progress has not been consistent from pilot project to pilot project In some cases there has been very little progress. For example, while progress has been made in the area of "better tracking of clients", that tracking does not extend to include post-ACR activities through to employment. Similarly, while some progress has been made in creating links with employers to provide work experience and on-the-job training, there continues to be a substantial lack of capacity in this area.

In overall terms, the design and operation of the ACR components has produced a substantial amount of progress and improvement in the areas set out in Exhibit 4.4.3.1 below.

1.Integration of service design

• Systematic assessment and communication at the beginning linking clients from MSS to MoEST.
• Consistency in contracting with individual clients, i.e., setting plans with individual clients.
• Common action plan for all agencies.
• More communication to coordinate contracts.
• Links with employers to provide work experience and on-the-job training.
• Collaborative planning among MSS, MoEST and HRDC; improve the sharing of information.
• Improve the working relationship between the two levels of government with respect to funding decisions.
• Improve or establish communication of ideas for improvements to management and to ACR service delivery in organizations by line staff. There is a need to cross-share ideas among MSS, MoEST and HRDC.
• "Bouncing around" between various agencies and government offices in order to implement action plans.
• Local meeting/conference on services and programs offered in the community.

• User-friendly reception at MSS; provision of more consistent information to clients.
• In-depth assessment of educational and psychological parameters to cover the needs of clients, earlier in the process.
• Better tracking of clients.
• More follow-up.
• Improve access to Labour Market Information by all government offices.
• Contracts with service providers that build in more ACR components (e.g., liaison/job coach position for clients who require additional support/assistance).
• More group work with one-to-one as an exception.
• More intermediate step between intake and access to training.
• Group-based client assessments and self-assessments; consider a drop-in centre.
• Follow-up or job coaching to help IISs maintain employment.
• More clarity about the role of the TCs.
• More support and follow-up systems.

• Self-assessment before seeing staff.
• Involvement of clients in self-assessment and action planning.
• Quick and easy methods for tracking client outcomes
• Employment opportunities.
• Peer supports.
• Mentoring.
• One-to-one career counselling.
• In-depth diagnostic assessment.

4.4.4 Service Gaps Not Addressed by ACR

In Exhibit 4.4.4.1 below, service gap items identified by the local ACR committees that were not as well addressed by the ACR components are listed.

The items listed in Exhibit 4.4.4.1 are important to analyze and understand in relation to the ACR/SI. They should not be interpreted as shortcomings on the part of the ACR committees or the pilot projects. Consistent messages about them should be communicated to the ACR partners so that appropriate efforts can be employed to deal with them, as described below.

Under the category "Economic factors" in Exhibit 4.4.4.1, high unemployment rates and low levels of income associated with certain jobs are economic factors which are not likely to be directly controllable within the parameters of the strategic initiative. However, the important issue with respect to these is the question of what message service providers in the ACR/SI send with respect to these issues. This is a complex issue because the rates of unemployment and relative levels of income from jobs vary from one geographic area to another. Analysis of the local micro economies will be important so that the ACR pilots can be designed specifically to deal with important regional differences, (e.g., rural, resource-based economy versus urban economy).

The economic factors described above should be addressed through ACR components because service providers counselling IISs will be required to address these issues and the government partners will have to allocate their resources optimally.

In the "Budgetary Factors" category listed in Exhibit 4.4.4.1, those service gaps which might be filled if additional budgetary resources were available are listed.

These items could be addressed by first identifying which ones can reasonably be included within the ACR budgets. Secondly, service providers need to have and present consistent and appropriate messages to IISs with respect to funding of the items. As well, creative attempts at overcoming these items should continue to be made by the ACR partners, especially at the local level.

These items in the "Information System Factors" category set out in Exhibit 4.4.4.1, raise the issue of the design and operation of information systems in support of the objectives of ACR. Partnership at much higher levels in the respective government organizations is required if the necessary information systems for referrals and client tracking and monitoring are to be developed.

1. Economic factors

• The pervasive effects of high unemployment rates.
• Low income levels associated with certain jobs.

2. Budgetary factors

• Lack of placements in work experience projects.
• Lack of capacity in other programs like academic upgrading, etc.
• Need personal counseling therapy for IISs.
• Need resources to provide job shadowing at employer sites.
• Lack of effective wage subsidy programs.
• Lack of affordable and available day care.
• No recognition, rewards or bonuses.
• Lack of support for transportation, snacks, day care and books.
• Lack of programs that contain work placements.

• No job bank at MSS site.
• The inability of the computer system used by MSS to "flag" clients who use many crisis grants or have participated in many programs.

• No identification of priority clients and how they are best served (front-line staff cannot counsel everyone).
• Require planned consultation with employers.
• Lack of involvement with cooperative programs in the local education system.
• A policy requiring clients to go through an agreed-upon ACR process before being sponsored for training.
• Lack of in-depth self-assessment of motivation.
• Lack of involvement with employers, clients, community agencies and business associations in the performance of the overall needs assessment done jointly by the two levels of government.
• Lack of planning process; this needs to happen at the Area Manager level and also at staff and supervisory level.
• No bridges to the regular school system.

• Need more seats for vocational, university and ABE training.
• Need to address the issue of "employer burnout" on work experiences (due to expansion of co-op education programs and the work experience component within the school district.
• Lack of apprenticeship opportunities.
• Lack of vocational training trade.
• Lack of extended work experience.

The "Management Factors" listed in Exhibit 4.4.4.1 lists items identified in the local ACR committees' planning processes that would require, for the most part, more involvement on the part of the management level of one or more of the partners to address. The local ACR committees may not feel empowered to act on these items, because policy changes or broader resource or inter-agency decisions are required.

Some of the items like "No bridges to the regular school system", "Lack of involvement with cooperative programs in the local education system", and "a policy requiring clients to go through an agreed-upon ACR process before being sponsored for training" may represent substantial opportunities for improvement and better outcomes that could reasonably be pursued through a broader partnership with local school systems and other organizations in the community.

The "Capacity Factors" category in Exhibit 4.4.4.1 lists gaps not addressed by ACR components that may not have sufficient capacity to meet the needs of IISs in one or more of the pilot project communities. Decisions to add more resources to these programs and services do not appear to be within the control of the local ACR committees or perhaps the immediate partners to the ACR/SI.

In summary, the ACR pilot projects have made very substantial progress in the integration of service design and the development and enhancement of client-centred programming for IISs. At the same time there are a number of very important and influential factors which do not necessarily fall within the scope of the ACR/SI to change, but nevertheless must be dealt with by front-line staff and service providers on a day-to-day basis in meeting the needs of recipients of income assistance. Some of the key problem areas are:

• The pervasive effects of high unemployment rates.
• Low income levels associated with certain jobs.
• Lack of placements in work experience projects.
• Lack of capacity in other programs like academic upgrading, etc.
• Lack of integrated, well-designed information systems to facilitate referrals and client tracking.

As part of the formative evaluation process, the ACR Evaluation Steering Committee wanted to have the Action Planning process reviewed.

"Did the range of support/services/training opportunities provided assist participants to develop their action plans?"

In summary, virtually all of the ACR program components in the pilot projects led to the development of an action plan, a skills plan, or a training plan.

Two issues that should be considered in connection with the action planning processes of ACR are whether too many plans are created for an individual client or whether separate plans that focus on different issues of the same client are well-integrated and coordinated. IISs clients may participate in more than one component and, as a result may have more than one plan. They may also have plans set up through direct consultation with Training Consultants at MoEST or Employment Counsellors at HRDC. The question arises whether all these plans come together as a well integrated and coordinated "individualization skills plan" as envisioned in the ACR Cooperation Agreement.

"What was the nature of clients' and others' involvement in developing action plans?"

With approximately 38 individual components, it would be impossible to list all of the different ways in which clients and others were involved in the development of action plans. In general, action plans were developed at the end of a program component, after the processes involved in the component were complete. These processes variously included presentation of information, self-assessment processes, assessment of skills, personal and career diagnostics, counselling with respect to barriers to employment, and so on. In every instance, a client-centred approach was taken whereby the individuals were supported in developing their own plans. In many cases, subsequent ACR components were designed to further assist clients who had difficulty with the planning process in the initial ACR component. In the final analysis, involvement of the individual IIS in the action planning process was more intense and comprehensive than had been the case in previous programs at individual partner levels.

"Was the nature of involvement of clients and others appropriate?", and "was the level of involvement of clients and others appropriate?"

No significant findings were identified in connection with the nature or level of client and other involvement in the "action planning process". In general, it appeared that both the nature and level of involvement of clients in action planning processes were appropriate. A related issue was the concern expressed by some Committee members that the action planning process, as conducted in programs like Starting Points should begin at the point of first intake of a IIS and not seven months later under the business rules of the BC Benefits program. This issue could not be resolved in the course of the formative evaluation and would require more detailed study.

"What suggestions/comments for involvement of selves and others would clients make?"

This question could not be answered in the formative evaluation because a client satisfaction survey was not conducted.

"What tracking/monitoring mechanisms are in place to collect information on participants and activities of the ACR Pilot Projects?"

There has been intense activity in the area of tracking and monitoring and it has been a somewhat problematic activity for some of the ACR pilot projects.

4.6.1 Feedback and Evaluation by Participants

In almost all of the projects, IIS participants complete feedback and evaluation forms at the conclusion of their courses, to inform the service providers of their views about the strengths and weaknesses of the courses just completed. In some instances, these reports are summarized for management reporting purposes and the results are made available to the local ACR committee.

4.6.2 Monitoring of IIS Progress

As well, most of the Starting Points programs and many of the other components featured the preparation of Action Plans which the service providers could use to monitor and follow up IIS progress.

Follow-up efforts included facilitator reports on IIS participation, voluntary, one-to-one follow-up sessions and participating in events to commemorate various successes.

Weekly progress reports were required from many service providers. Semi-annual contract summary reports were also required in some cases. Other monitoring methods included voluntary drop in sessions and a bi-annual participant survey.

In general, all of the Starting Points components included monitoring activities to keep in contact with the participants and assist them in completing their Action Plans. Also, some of the pilots arranged to have Action Plans of IISs made available to the Training Consultants of MoEST and the Financial Assistance Workers of MSS. This was done after release of information authorizations were obtained from the IISs.

At least one local ACR committee felt that the reporting and monitoring mechanisms in place were excellent. This view was shared by some of the service providers. Other service providers felt that the paperwork required to satisfy all the partners (ministries) was excessive.

There are some concerns in connection with the monitoring processes. First, there is the question of how to achieve long term tracking which will make it possible to evaluate "long-term attachment to the work force" outcomes. Most of the tracking described in connection with the ACR components was short term in nature, focusing on time frames like six weeks, six months or one year after participants have completed their programs. These time frames are useful, but are not adequate for assessing lasting outcomes given the fact that many individuals on income support "cycle" in and out of the welfare system over a period of months and years.

Second, there is the problem that in many cases it is the contracted service providers who are doing most of the monitoring through call backs described above. The issue here is what will be done after the contract terms have ended. It is very unclear whether long term monitoring of the kind required can actually be achieved by contractors whose contracts last only one or two years. Some sort of succession plan may be required for the ACR partners or subsequent contractors to carry on with client monitoring after the preceding contracts have expired.

Third, there is the problem of information confidentiality discussed in more detail in the Starting Points section of this report.

Efforts are currently under way to implement new and more useful information systems on the part of MoEST for purposes of monitoring the progress of IISs through training and into employment. Through these efforts and in partnership with the other ACR stakeholders, monitoring "long-term attachment to the work force" outcomes needs to be enabled in a manner which produces relevant, complete, consistent and continuous information over the longer term.

"What potential do the ACR/SI projects manifest for innovation?"

In addressing this issue, it is worthwhile to note that an often repeated comment during the evaluation was "there is nothing new in the world of services provided to individuals on income support". This comment serves to illustrate that efforts to provide services to IISs that lead to enhanced employment outcomes have been ongoing for many years and any expectation that it is possible to develop a "silver bullet" solution is unrealistic.

On the other hand, there is significant opportunity for innovation and new ideas in the development of assessment, counselling and referral strategies and program components. The best example of this is the Starting Points program which has been described earlier in this report. This program began as an innovation that was found to produce fruitful results and was subsequently developed further by HRDC and professional experts who refined the program to what it is today.

In the first round of ACR/SI components there were many innovations at the level of the local ACR committee operations and the level of the service providers who ultimately delivered specific program components. One of the key innovations was the support centre concept that was developed and implemented in the Burnaby and Fort St. John ACR pilot projects. With the development of support centres, permanent locations were established where individuals on income support could go to obtain support, use facilities like computers and fax machines, and meet to discuss their progress amongst themselves and with other support workers.

Other innovations took the form of existing programs which were modified to meet the specific needs of individuals on income support. An example of this was the Moving Forward component in Campbell River where a previous program for academic upgrading was modified to meet the specific needs of IISs.

In summary, the ACR/SI gave rise to significant innovation in program delivery for recipients of income assistance, particularly because of the delegation of responsibility for design and development to the local community level, on a partnership basis. Although many of the pilot components were extensions or repetitions of previous programs, the flexibility inherent in local needs assessment and component design established the possibility for innovation and gave rise to a significant number of new initiatives focused on meeting the needs of IISs.

4.8.1 Background

By undertaking the ACR/SI in partnership, the federal/provincial Cooperation Agreement brought together two autonomous governments, the Canadian federal and British Columbia provincial governments, to develop and operate the program components. In so doing, two separate governments with different jurisdictions, operations, organization structures, and decision-making processes had to work together toward a common goal.

At the provincial level, the partnership included two separate ministries, MSS and MoEST. Differences existed between these two ministries. MSS focused on administering income assistance, including establishing and monitoring the eligibility of individuals on income support, while MoEST focused on programs of support and intervention to assist IISs to move from dependence on income assistance to re-joining the workforce.

The partners' organization structures or operations created some challenges for the local ACR committees and the Steering Group. For example, in the case of HRDC, a delay in budget allocation at the regional level hampered the ability of at least one local ACR committee to establish contracts for specific ACR components. In the case of MoEST, this provincial ministry had a highly-centralized contract administration system before ACR. Contracting authority and the authority to make payments in respect of contracts was concentrated in headquarters. This contrasted sharply with the local ACR committees' need to be able to establish contracts at the local level on a timely basis to be able to secure ACR service providers.

4.8.2 Significant Change Dynamics at the Partner Level

A number of changes at the partner level occurred that affected the achievement of the local ACR committees and the Steering Group. First, MSS underwent a series of reorganizations before ACR and the change management dynamics were continuing to be felt. A key outcome of these re-organizations was the separation of the Training Consultants from MSS. Prior to separation, the MSS Rehabilitation Officers who performed this function had worked in close proximity to the MSS FAWs. The Rehabilitation Officers moved to the new Ministry of Skills, Training and Labour and became Training Consultants. The ministry subsequently became the Ministry of Education, Skills and Training.

The Gove Commission of Enquiry also created substantial challenges for MSS as that ministry re-organized and became the Ministry of Human Resources.

At MoEST, the field structure which included the newly transferred Training Consultants described above was in the early stages of development. Even without ACR, the roles and responsibilities of Training Consultants and other MoEST staff at the local level were in the process of being developed.

Changes were also occurring at HRDC, as that organization prepared for a conditional transfer of certain responsibilities to the provinces and the predecessor Unemployment Insurance scheme was being transformed into the new Employment Insurance program.

The local ACR committees would have to deal with the changes described, while at the same time designing and implementing the new ACR components on a partnership basis. As a result, the overall set of challenges to the local ACR committees and the staff of the partner organizations was extreme, due to organizational and policy changes occurring within the partner organizations.

4.8.9 Partnership at the Field Level

Partnerships created at the field level under the ACR Strategic Initiative were very effective at the planning stage of the pilot projects. In the planning phase most of the local ACR committees were assisted by independent consultants retained by the Provincial Steering Committee. In this way, the nature and extent of client needs in the local areas and the types of programs which were being offered by the various levels of government were identified. Also, the local ACR committees identified gaps in the services on offer to the IISs in the areas. Based on this information, the local committees prepared Service Delivery Plans which took into account the existing programs of all the government partners. This process was very effective and the partnership concept worked very well to produce integrated, locally specific plans for meeting the needs of IISs in the areas.

The concept of partnership proved to be less effective once the local ACR committees began to put their Service Delivery Plans into operation. Due to a lack of infrastructure alignment within and between the separate government organizations comprising the partnerships, the operations and administrative processes required to implement the ACR components were not facilitated efficiently or effectively (e.g., contracting, information systems, processing payments, etc.).

As well, the business direction of MoEST changed with the adoption of new policies under the BC Benefits Program and the Youth Works initiative. The result was that the local ACR committees had to reassess their priorities and Service Delivery Plans in relation to the new direction being taken. The impact was that the enthusiasm and commitment of at least some of the local ACR partners was dampened. They felt that their efforts had been wasted and their roles had been compromised because the assessments and decisions which they had made in good faith through the planning processes described above were being overridden without consultation, and indeed in ways which they did not necessarily agree with.

Also, the new rules for BC Benefits and Youth Works were seen to create additional work for MSS staff responsible for referring IISs into the ACR components. This came at a time when workload capacity of MSS staff was already a serious concern. The result was that MSS staff in some areas refused to develop the additional work processes needed to refer IIS clients to ACR components. Referrals into the components ceased and the pilot projects in these areas bogged down without any changes in contractor costs.

A curious feature in the implementation of the ACR pilot projects was that the local committees were not uniformly required to implement the new BC Benefits rules. Some did not, and chose instead to continue with their original Service Delivery Plans.

In other cases, local management at the provincial level required the BC Benefits rules to be implemented irrespective of the wishes of the local committees. This produced uneven outcomes across the pilot projects, so that the pilot projects themselves would not be immediately comparable.

In the final analysis, the results of the way in which the BC Benefits Program was rolled out were that the overall business direction of ACR/SI was made uncertain, the original roles and responsibilities of the ACR partners became confused and the partnerships at the local ACR committee level were weakened. However, in overall terms, although there were many challenges to be overcome, much progress was made by the partners to the ACR/SI in working together to innovate and enhance services available to IISs in the pilot project communities.

4.9 Best Practices and ACR/SI

In the case of the ACR/SI, the flexibility given local ACR committees in designing pilot project components was intended to produce a number of variations in programming which could be observed for the purposes of determining best practices.

In this formative evaluation, it was not possible to identify best practices from one or another pilot projects because performance measurements of the various alternatives were not taken and were not available. Also, a process of developing consensus around a potential "best practice" had not been implemented.

Better information may be available as the result of a summative evaluation of the pilot projects. However, it is important to realize that a summative evaluation may not produce detailed assessments of individual practices because of the cost and level of analysis required for this purpose. Therefore, an ongoing best practice process is required.

5.1 Potential for Achievement of the ACR Strategic Initiative Objectives

During the formative evaluation, the evaluation team reviewed the design and operation of each of the ACR components in the pilot projects in relation to the ACR Strategic Initiative objectives established at the outset, to estimate whether the objectives were likely to be achieved, in whole or in part, or whether they were not likely to be achieved because of the design or operation of the components. The evaluation prepared detailed tables for each pilot project setting out the results. In this section of the report, we present our conclusions.

Exhibit 5.1 presents the ACR Strategic Initiative objectives and sub-objectives that were used to assess the individual pilot project components.

---

Exhibit 5.1: Achievement of ACR Strategic Initiative Objectives

---

1. To increase retention of IISs through the incremental stages of training by:

• actively involving participants in defining realistic and achievable career goals.
• providing the necessary counseling supports and follow-up activities.
• developing better reporting mechanisms between agencies and MoEST.
• improving the tracking of participant through training and into employment

---

2. To increase the potential for success of IISs to move through training and into employment by:

• ensuring that training plans reflect individual and labour market needs.
• ensuring that training plans incorporate employability skills and computer literacy skills.
• ensuring that training plans incorporate training that is portable and transferable.

---

3. To increase the career planning/management skills of IISs enabling them independently to build on career goals by:

• ensuring career planning courses/services include several components (e.g., LMI, career exploration, personal development, vocational assessment) and are based on "best practices".
• ensuring curriculum and teaching methods promote individual skill development.

---

4. To increase access to assessment, counseling and referral services for IISs by:

• increasing availability of career planning services at appropriate stages.
• establishing appropriate referral mechanisms between ministries, agencies and institutions.
• increasing the availability of information on training resources and employment patterns, etc.
• ensuring better integration and coordination of community services.

---

In general, all of the objectives and sub-objectives listed above are being addressed in varying degrees by the ACR components of the pilot projects, although it is extremely difficult to evaluate whether there are strengths and weaknesses in the overall set of components because of the extent of diversity and the varying local requirements.

With respect to the first objective of increasing retention of IISs through incremental stages of training, it was not possible to determine whether retention rates had actually improved in the formative evaluation because this data was not available. However, with respect to the individual sub-objectives that were intended to contribute to this outcome, it is clear that many of the ACR components actively involved participants in defining career goals. The processes that were put in place included various kinds of information and activities intended to ensure these goals are more realistic and achievable. The ACR components did attempt to provide necessary counselling supports and follow-up activities. Extensive efforts were made by the contracted service providers to improve reporting mechanisms and make the necessary information available to other partner agencies and MoEST. Short term tracking mechanisms were implemented by many of the service providers but little progress was made on implementing longer-term tracking mechanisms intended to follow the IIS over a longer timeframe, or in relation to the tendency of many IISs to cycle in and out of the system over several years.

With respect to the second objective of increasing the potential for success of IISs to move through training and into employment, most of the efforts of the local ACR committees and service providers focused on attempting to achieve this objective. Almost every component of the ACR/SI produced plans which reflected individual needs and many also included some assessment of labour market requirements. Training plans generally incorporated employability skills as required and some of the components addressed computer literacy skills where resources were available.

It was not possible to identify ACR components which addressed the sub-objective of ensuring that training plans incorporate training that is portable and transferable. In this respect, it is likely that this sub-objective is being addressed to a greater extent than was evident in the analysis but it is likely that the concepts of "portability" and "transferability" have not been sufficiently defined in operational terms to highlight them in the design of specific components.

Much of the effort of the local ACR committees and service providers focused on the objective of increasing the career planning/management skills of IISs, thus enabling them to independently build on career goals. A number of career planning and decision-making components were implemented and many of these addressed career exploration, personal development, and vocational assessment. Some also included labour market information.

It was unclear to the evaluation team whether "best practices" were incorporated into the components or whether a process existed for identifying best practices based on the experience of individual components. Some interviewees suggested that all of the practices that were incorporated into their ACR pilot projects were "best practices". However, there was little support for this other than the individual opinions of those involved. These practices had not been proven using valid output measures or other processes of consensus building around "best practices" concepts.

Virtually every ACR component focused on ensuring that curriculum and teaching methods promoted individual skill development.

The final objective to be addressed by the ACR pilot projects was to increase access to assessment, counselling and referral services for IISs. Career planning services were made more available as a result of the ACR/SI. However, difficulties were encountered in establishing appropriate referral mechanisms between ministries, agencies and institutions. Key difficulties have been discussed previously in the Starting Points section of this report. The ACR pilot components did increase the availability of information on training resources and employment patterns. Finally, some progress was made in ensuring better information and coordination of community services, although, linkages to local economic development agencies and employers were not improved to the extent that may be possible, subject to resource limitations of local staff.

In conclusion, all of the four main objectives listed in Exhibit 5.1 above are being addressed in a relatively well-balanced manner by the overall set of ACR components. Key issues have arisen in the referral mechanisms and tracking and monitoring processes

Our evaluation has resulted in a preliminary conclusion that the ACR/SI components which may be the most essential and useful to IISs, are as follows:

Starting Points-each person interviewed agreed that a needs assessment is useful to most IISs before they embark on further training or seek employment. Starting Points appears to be a good framework for providing these services.

It should be noted that the Starting Points objectives do not address the long-term goal of assisting IISs in moving from welfare to work. Long-term attachment to the workforce may occur at some future point and Starting Points may contribute to this accomplishment, but, it is quite unlikely that Starting Points will suffice for the majority of IISs in their efforts to become employed.

Orientation/Information Session-two of the pilots (West End and Burnaby) provide a 1.5 hour Orientation Session. This session appears to provide useful information to IISs regarding services and resources available, as well as IIS responsibilities.

Career Planning-some form of career planning is being provided in almost every pilot project. This service may be necessary for IISs who require further assistance in developing a viable career plan.

Support Workers-many IISs require assistance in completing training programs or maintaining employment. Support workers provide this needed service to IISs. Services can include coaching, counselling or employment services.

Career Information Resource Centre-a centre which provides resource information and employment services such as access to facsimiles, telephones, computers, photocopiers, etc. appears to be useful to IISs.

Work experience-work experience is beneficial to IISs who are interested in pursuing a particular career and wish to acquire a practical understanding of such work prior to embarking upon further occupational training commitments. Work experience also provides the IISs with contacts in their chosen field.

Given the overall challenges described above, the local ACR committees and Steering Group have achieved a strong partnership in creating the 11 pilot projects and serving the substantial number of individuals on income support identified in this report.

Several conclusions should be drawn from the experience:

• The ACR partnership worked very well to produce integrated, locally specific plans to meet the needs of IISs in the pilot project communities.
• Due to the existence of formal organizations that extend beyond the ACR partnership, it should be realized that the partnership itself will necessarily be imperfect. Each of the individual partners has their own core missions, priorities, policies and ongoing challenges which will take priority over the requirements of the partnership from time to time.
• In order to deal with the limitations of partnership successfully, it will be important to set an appropriate level of expectation for all of the parties to the ACR/SI, and particularly the local ACR committees.
• The effectiveness of a partnership like the one established for ACR will always be subject to the effectiveness of the individual partner organizations so that if one or more of the partners has difficulty with decision-making processes or effective delegation of responsibility and authority, these difficulties will be reflected in the partnership.
• Insofar as a substantial harmonization of policy, procedure, language and strategy are required in order to have an effective partnership such as ACR, these factors have to be secured through a strong executive sponsorship of the partnership at the individual partner organization level.
• In overall terms, although there were many challenges to be overcome, much progress was made by the partners to the ACR/SI in working together to innovate and enhance services available to IISs in the project communities.

The Client and Agency Profile and Service Delivery Planning processes of the ACR pilot projects appeared to be very effective in ensuring a more integrated and coordinated approach to program delivery in the communities.

The pilot projects reflected the criteria established for ACR quite closely with challenges in rural communities to meet as many of the criteria as possible.

Subject to budget and resource constraints the pilot projects were able to close a number of the gaps that existed between the needs for, and availability of resources in the pilot project communities, although some of the more significant problems like pervasive unemployment were not really addressed.

For purposes of the ACR/SI, the process of identifying best practices should become a standard feature of ACR administration.

Best practices can be established by routinely reviewing alternative processes to determine which process is meeting objectives most efficiently and effectively. For a given process, field level information and insights can be developed at the operational level, followed by further refinement through the application of expertise, research findings and facilitation. Next, through additional piloting, the effectiveness of the particular process or component can be further tested. With this information and the consensus of knowledgeable practitioners, a determination of whether a best practice has been identified can be made.

Evaluation Framework

1. In what ways do the ACR Pilot Projects reflect the criteria established for the Strategic Initiatives:

1. What potential do they manifest for innovation?
2. Do they address a widespread need?
3. How might they influence the social reform process?

2. How and to what extent have the ACR Pilot Projects succeeded in developing effective partnerships among the various levels of government, employers and community groups?

1. Were the roles and responsibilities of the various partners and service providers clearly enunciated and carried out?
2. What structures, processes and/or strategies have enhanced the formation of effective partnerships among the various levels of government, employers and community groups?
3. What structures, processes and/or strategies have detracted from the formation of effective partnerships among the various levels of government, employers and community groups?

3. To what extent have the ACR Pilot projects met the need for enhanced assessment, counselling and referral services in the communities?

4. To what extent have the Service Delivery Plans closed the gaps that exist between the needs for and availability of resources? (e.g. services needed by clients to achieve career goals and services available, availability of labour market resources, of training programs, etc.?)

5. How consistent were the design of the ACR Pilot Project features with the stated objectives of the program?

6. What lessons can be learned from these ACR Pilot Projects?

1. What are the elements of a successful ACR initiative?
2. How effective was the ACR community by community needs assessment and implementation strategy for ensuring a more integrated and coordinated approach to program delivery?

1. To what extent did the ACR Pilot Projects assist clients to make more informed choices regarding employment, training, further education and other actions to enhance their employability?

2. What tracking/monitoring mechanisms are in place to collect information on participants and activities of the ACR Pilot Projects?

1. How many clients participated in various components/activities?
2. How many clients completed various components/activities?

3. How did the range of support/services/training opportunities provided assist participants to develop their Action Plans?

4. How effective were the curriculum materials, self-assessment tools, labour market information and information on program policies as a first step in:

1. assisting clients to create action plans?
2. assisting clients to pursue Action Plans?

5. Overall, what was the level of satisfaction of clients with the ACR Pilot Projects and their perceived level of effectiveness?

1. How useful were specific components? (List variances from pilot-to-pilot)
2. What components were most valued?
3. What components were not useful or negative?
4. What suggestions for improvement would clients make?

6. What was the nature of client and other involvement in developing Action plans?

1. Was the nature of involvement of clients and others appropriate?
2. Was the level of involvement of clients and others appropriate?
3. What suggestions/comments for involvement of selves and others would clients make?

7. To what extent have employment related obstacles and barriers facing individual clients been reduced or eliminated through the operation of ACR pilots (preliminary findings only)?

Exhibit C-1: Starting Points agenda

View Exhibit D-1:
Inventory of Starting Points components

Exhibit E-1: Starting Points objectives

View Exhibit F-1:
Numbers of ACR Participants

View Exhibit G-1:
Dates of Participation

Footnotes

1	These new ministries took over from the previous BC Ministry of Skills, Training and Labour and the BC Ministry of Social Services.
2	As the result of this contract, the sequence of referring IISs to ACR components was changed so that all referrals came from Financial Assistance Workers in MSS and initial referrals all went to the Starting Points component.
3	After this program began, it was learned that there was a greater need for services designed to help IISs select appropriate training programs and obtain employment. These services were therefore added until MoEST could establish a separate contract for this purpose. Afterwards, it was intended that the Support Centre would resume providing support services for individuals with personal barriers and employed persons in need of support to maintain employment.
4	Westwood, Marvin, Norman Amundson and William Borgen. ''Starting Points: Finding Your Route to Employment'', Working Draft, December 1994.
5	Westwood, Marvin, Norman Amundson and William Borgen. ''Starting Points: Finding Your Route to Employment'', Working Draft, December 1994.
6	Details were not available on what follow-up activities are conducted in Surrey.