Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

1. Context of the Evaluation and Terms of Reference

The federal government and the government of British Columbia developed the Strategic Initiatives (SI) program to “... provide a funding mechanism for the federal government to work in partnership with provincial and territorial governments to test new and innovative approaches in high priority areas of employment, education, and income security.” One of a number of such innovative approaches was the establishment of 11 pilot projects providing enhanced Assessment, Counselling and Referral (ACR) services throughout the province.

The ACR pilot projects were designed to “...test and demonstrate an enhanced assessment, counselling and referral system for people on income support. It is expected to improve the linkages within and between employment programs, the individual on income support and the labour market.” They were to be planned and managed by a local Steering Committee of government and community partners.

The ACR/SI pilot projects were to incorporate up to five key program components, depending on their local service needs assessments and assessment of their client profile. The key components from which they could choose were:

1. Starting Points group assessment process to assist clients in developing a “First Steps” action plan towards employment;
2. In-depth group assessment/orientation (especially for multi-barrier clients);
3. Diagnostic assessment for those whose barriers to employment exceed the capacity of standard services provided by the funding partner staff or contracted agencies (especially for clients with specific physical, psychological and/or learning disabilities);
4. Group career planning building on recognized best practices in this process; and
5. Learner support to reduce personal/social barriers for those clients who require this service to increase their capacity to meet their employment goals.

The local Committees could — and did — modify existing programming or develop new (or new to the community) programming that would meet the ACR needs of their local client group as effectively as possible.

Integral to the planning and implementation of the ACR/SI was the requirement that there be “program-wide evaluation at appropriate intervals.” In keeping with this approach there was a formative evaluation completed in 1997. A summative evaluation, including collection of baseline data was to follow, with completion scheduled for the end of the 1998/99 fiscal year.

The overall objective of the summative evaluation was stated in the Request for Proposals as being:

...to examine issues associated with program delivery, effectiveness and outcomes, as well as ongoing performance monitoring. [And it] will evaluate the early outcomes of the Initiative. A baseline data component will be required.

This is the final report of that summative evaluation.

2. Summative Evaluation Methodology

There were two main components of the evaluation research. One was a baseline telephone survey of past participants and a comparison group of non-ACR users. The other was a case study approach to all of the 11 pilots being evaluated. This included on-site visits to most sites and telephone interviews with the remaining ones. The site visits included interviews with the Steering Committee and contracted service deliverers and a review of relevant local documents. These two main research components were supplemented by interviews with key respondents at senior policy and planning levels within the partner governments and a review of related documents at the program-wide level. The data collection strategies and the analytical techniques were matched to the nature of the data and the evaluation issues and questions being addressed. Thus, the findings rely on both content analysis and a variety of statistical analyses.

3.Findings of the Summative Evaluation

Evaluative Findings on Impacts, Effects and Outcome of ACR/SI Programming

The ACR/SI has not been without its difficulties, but overall the impacts and effects have been positive. For the program developers and managers among the government partners, it had benefits for meeting their goals for services to clients, for learning to work together (which has been a useful precursor to their recent move into the Labour Market Development Agreement — LMDA — co-management/co-funding framework), and for learning more about each other’s client group and how to work with them. The workload of management during the planning stages was increased, but once the pilots were fully implemented, this seems to have been reduced. The workload for many of the local MAETT Training Consultants was increased, at least sporadically, but this was not uniformly the case.

For service deliverers the impacts were more mixed. On an individual, local level, the working relationships with government partners tended to be positive, and contractors generally felt the programming models they developed and delivered were good ones. On the other hand, the introduction of the service Phases under BC Benefits (starting mid-1996) tended to have negative effects on service providers (and clients). The flow of potential clients was interrupted and seriously reduced in some cases, resulting in considerable strain on the ongoing business operations of a number of contractors. Clients who were being served before sometimes then had to wait months for service, and many who had just come into the system and completed Starting Points could not access follow-up services for as much as seven more months (if they had not yet found employment by that time). This disruption of the continuity of services was seen by all as detrimental to both clients and service providers.

The survey results are clear that participation in the ACR/SI is beneficial. It enhances the participants’ knowledge of the labour market and increases their confidence in their employability (ability to find, retain or change jobs). When compared to non-participants, it is equally clear that ACR/SI participation increases by a factor of four the likelihood of a successful outcome, defined as finding employment and/or pursuing further education or training.

Evaluative Findings on Objectives Achievement for the ACR/SI as a Whole

The objective of developing partnerships at all government levels to plan for and implement the ACR/SI has largely been met. There has been considerable time, effort and personal dedication invested at all levels in achieving this objective throughout the evolution of the ACR/SI pilot projects. (In several communities these efforts, as embodied in the Steering Committees, have continued under the aegis of the LMDA.) The majority of communities developed positive, collaborative processes for planning and implementation. These efforts were not without their difficulties, as partners with diverse philosophies and modes of operation learned to work together. But most managed quite well and reported that the process in itself enhanced their own understanding of each other’s work and of the clients who each had previously served separately.

The programming delivered indeed enhanced rather than duplicated existing ACR services. The local Committees were careful to identify needs that were not being met and to work with the service-providing community to develop programming that was adapted specifically to those needs. They worked within the framework of the five programming options set out for the ACR/SI, and developed innovative means that met their client profile, budget and local service resource base.

The objective of improving linkages among partners and employment-related services and between clients and these services was not achieved to the same degree. This largely was due to structural constraints. The primary difficulty was that the change to serving Phase II clients on Income Assistance (IA) limited the ability of ACR/SI service providers to refer to these services. Also, when clients were in Phase II they often were less accessible in the first place or less willing to utilize these services. There was a minor issue of multi-service agencies not referring clients to outside services as much as they might have — or at least this was a perception in some cases — but government partners took corrective measures if the concern seemed grounded in fact.

Evaluative Findings on Program Rationale — Is There a Continuing Need for Services and Is the ACR/SI Approach the Optimal One?

There is clearly a need for ACR-type services, given that there is an increasing proportion of individuals who have substantial barriers to planning their moves towards independence from support programming. The economic downturn of the province in several key sectors, the increased competitiveness of the labour market, and the changing profile of the unemployed and those on IA for extended periods all lead respondents to the conclusion that this type of programming will continue to be essential.

Though there have been difficulties in working out the collaborative model of planning and management, respondents in general feel that the principle of drawing on local planning to determine needs and to develop and allocate services should be maintained in future. Local control of significant decision making is seen as an integral part of that approach.

Thus, there is a need for these types of services and every likelihood that the partnership approach to planning and implementation is, overall, quite effective. The approach seems to be fully justified, even if it has its difficulties. Given that respondents learned a great deal about the clientele, about responsive programming, and about working together to plan and implement it, most feel that they are well placed to move ahead along the same lines. Many already are, under the framework of the LMDA and through innovative use of other funding envelopes.

Evaluative Findings on Alternatives/Future Directions for Programming

It is clear that the basic functions of ACR programming should continue to be provided, but there is an increasing need to develop assessment tools that can be more effective with the multi-barrier, long-term IA recipient. If obstacles to an effective process, such as the waiting period arising out of BC Benefits legislation on Phases for receipt of services, are removed, then a number of possible problems for future programming would be reduced or eliminated.

It is evident that the collaborative, co-managed and co-funded approach holds much promise, but respondents point out that programming goals and processes need to be harmonized among each of the government partners. This will enable clarification of programming goals and facilitate the identification of criteria for success, which in turn could lead to a more effective assessment of outcome.

Evaluative Conclusions and Lessons Learned

The ACR/SI was designed to be a strategic approach to enhancing the panoply of assessment, counselling and referral services provided to persons requiring (and required to pursue) assistance in achieving financial independence from government income support programs — through increased attachment to the labour force.

The pilot projects clearly demonstrate that there is a need for such programming and that the enhancement of existing services is both necessary and possible. It is clear that having project planning undertaken by a Steering Committee of local partners at both government levels, acting in concert in many cases with service deliverers, was an effective, if sometimes taxing, means of ensuring that programming met local needs in an innovative manner. The ongoing co-management of programming was generally effective, though it was not without its problems at times.

Work remains to be done to harmonize the sometimes divergent mandates, policies and day-to-day procedures of the two levels of government (and the three partners — MHR, MAETT and HRDC). It is also important to develop additional assessment tools that will be more effective with the various client groups who experience their own distinctive barriers to the move towards independence.

If the ACR/SI is seen as an experiment from which useful lessons can be learned, it has succeeded in being so. The overall lesson learned from the ACR/SI is that it is a model that was effective in itself. The outcomes were positive. It is a demonstration of the value and efficiency of a locally planned and implemented program. The problems that arose often reflected contradictions in the larger policy and legislative framework, rather than a lack of will or skill on the part of those who initially conceived it or of those who then carried it out. The essential elements of its programming content — including the flexibility to modify programming readily — are all worth incorporating into future programming to meet the original policy goals.