|
|
     |
||
 |
     |
 |
|
This section of the report sets out key findings in relation to the original questions of the ACR Evaluation Steering Committee, together with other matters identified in the course of the formative evaluation of the ACR/SI.
"In what ways do the ACR projects reflect the criteria established for the strategic initiatives?" The review of the pilot projects indicated that the components which were developed by the local committees followed the criteria of the ACR/SI very closely. Indeed, the criteria were broad enough that there was room for a substantial amount of variation in components to address the five fundamental elements of ACR described previously. The more significant challenge for the local ACR Committees was to develop enough components to address as many of the ACR criteria as possible, for the range of client needs in the area. This challenge was more significant in some of the rural communities where fewer service providers and less expertise were available to develop the programs required.
"To what extent have the ACR pilot projects met the need for enhanced assessment, counselling and referral services in the communities?" One of the primary outcomes of the ACR components was a strengthening of the counselling process, from one-on-one counselling at the partner level of government to group counselling and assessment, whether by individual government staff or contracted service provider. This move to group assessment, counselling and referral facilitated a broader and more immediate provision of service to IISs and, according to some of the commentaries received, enabled better services to be provided because of the group peer support effects that occur in group sessions. The quantitative results for the ACR/SI from inception to July 1996 are somewhat unclear because difficulties were experienced by some of the local ACR Steering Committees and service providers in providing the numbers of participants for the various ACR components. The number of participants that were reported during the formative evaluation are set out in Appendix F. The dates of participation for the participants are set out in Appendix G. The information that was available indicated that between 2% and 32% of all IISs in a given pilot project area attended at least one ACR component. The participation rates represent early results of the ACR/SI. The period covered by these results, from April 1995 to July 1996, should be regarded as the start-up period for the projects. Accordingly, the participation rates reflect not only the ability of the projects to serve recipients of income assistance but also reflect some of the initial challenges experienced in the start-up period of the program. In any case, the relative proportion of IISs who have received ACR services in the timeframe described above is significant and can serve as the basis for further analysis in a summative evaluation.
It is somewhat difficult to comment on the extent to which the projects have met the need for enhanced assessment, counselling and referral services in the communities. Quantitative assessments were not fully performed in the planning process so that target numbers of placements for individual categories of IISs were not set in every case and are not therefore available for use in measuring actual results. It is expected that the numbers of participants in the ACR components will increase substantially in the period following June 1996, as referral and service delivery processes are refined and knowledge about the components spreads throughout the IIS target population. One unresolved issue in the area of extent of service is the question of targeting ACR components to the multiple barriered population that may not attend referrals or may not be identified through self-referral or other processes. The key question is to what extent should special efforts be made to seek out and involve IISs who have traditionally avoided interventions and contact with service providers in the communities. The first round of ACR pilot projects tended to serve those IISs most likely to attend intervention programs and seek employment. Others who traditionally avoid such involvements were not necessarily contacted or served in the first round of pilot projects. During the formative evaluation interviews, some contractors indicated that this was appropriate because it was more cost efficient to serve the needs of motivated and committed participants. It is therefore unclear to what extent the ACR pilot projects are meeting the needs of less motivated and more challenging recipients of income assistance. 4.4 The Planning Process As a strategic initiative, ACR/SI did not specify what programming was to be put in place at the local level. Instead, it identified high-level goals and objectives, more detailed program elements and a two-tiered committee structure to develop the actual program components at the field level. The planning process for the local pilot projects began with the preparation of a "Client and Agency Profile" for the area, in order to determine what kinds of training and employment programs were needed and available in the community. Following this, a "Service Delivery Plan" was to be developed that outlined the program and process elements to be jointly developed by the local ACR Committee.
4.4.1. The Community-by-Community Needs Assessment and Implementation Strategy "How effective was the ACR community-by-community needs assessment and implementation strategy for ensuring a more integrated and coordinated approach to program delivery?" During the formative evaluation all of the client and agency profiles and service delivery plans of the 11 pilot projects were reviewed by the evaluation team. The client and agency profiles appeared to be comprehensive with respect to an analysis of IIS client populations and categories in the local communities. As well, the review of existing services available and service providers was also comprehensive in each pilot project. The results of these assessments were comprehensive listings of IIS populations and sub-groups, as well as contracts and services available. These were taken into consideration in the development of the service delivery plans. As a result, it seems clear that the community-by-community needs assessment and implementation strategy was very effective for ensuring more integrated and coordinated approach to service delivery. Since the local ACR Committee represented each of the partners to the strategic initiative, and since they participated in the inventorying of all available services irrespective of which agency provided them, they necessarily developed an understanding of all of the local program delivery components on an integrated and coordinated basis.
4.4.2 The Service Delivery Planning Process "To what extent have the service delivery plans closed the gaps that exist between the needs for and availability of resources?" In order to answer this question the evaluation team reviewed the original service delivery plans of each of the local ACR committees in relation to the inventory of service delivery gaps they prepared, to determine whether the design of the individual components would address one or more of the gaps identified. As well, the operation of the ACR components was discussed with local ACR committee representatives and service providers, to further determine which service gaps were being addressed. The evaluation team then prepared a table of service gaps for each pilot project and tabulated which ACR component addressed which gap. The results of this analysis are presented below.
4.4.3 Service Gaps Addressed by ACR In the Exhibit 4.4.3.1, service gaps which were addressed by one or more of the ACR components in a local pilot project are listed. With respect to the items in Exhibit 4.4.3.1, progress has been made in addressing many of the service gaps which the local ACR committees identified at the outset of the planning process. However, it is important to understand that the items listed above have not been resolved by the operation of the ACR components or dealt with uniformly from pilot to pilot. Rather, improvements have been made by one or more components in a pilot project with respect to one or more of the items described above. More progress has been made in some areas than others. For example, under the category "Integration of Service Design", in Exhibit 4.4.3.1, a very substantial amount of progress has been made as a result of the overall partnership of government agencies involved. As well, substantial progress has been achieved in the area of developing client-centred programming. However, in the areas of "Enhancement of Programs and Services", shown in Exhibit 4.4.3.1, progress has not been consistent from pilot project to pilot project In some cases there has been very little progress. For example, while progress has been made in the area of "better tracking of clients", that tracking does not extend to include post-ACR activities through to employment. Similarly, while some progress has been made in creating links with employers to provide work experience and on-the-job training, there continues to be a substantial lack of capacity in this area. In overall terms, the design and operation of the ACR components has produced a substantial amount of progress and improvement in the areas set out in Exhibit 4.4.3.1 below. Exhibit 4.4.3.1: Service gaps addressed by one or more ACR pilot projects (or individual component) 1.Integration of service design
2. Enhancement of programs and services
3. Capacity factors
4.4.4 Service Gaps Not Addressed by ACR In Exhibit 4.4.4.1 below, service gap items identified by the local ACR committees that were not as well addressed by the ACR components are listed. The items listed in Exhibit 4.4.4.1 are important to analyze and understand in relation to the ACR/SI. They should not be interpreted as shortcomings on the part of the ACR committees or the pilot projects. Consistent messages about them should be communicated to the ACR partners so that appropriate efforts can be employed to deal with them, as described below. Under the category "Economic factors" in Exhibit 4.4.4.1, high unemployment rates and low levels of income associated with certain jobs are economic factors which are not likely to be directly controllable within the parameters of the strategic initiative. However, the important issue with respect to these is the question of what message service providers in the ACR/SI send with respect to these issues. This is a complex issue because the rates of unemployment and relative levels of income from jobs vary from one geographic area to another. Analysis of the local micro economies will be important so that the ACR pilots can be designed specifically to deal with important regional differences, (e.g., rural, resource-based economy versus urban economy). The economic factors described above should be addressed through ACR components because service providers counselling IISs will be required to address these issues and the government partners will have to allocate their resources optimally. In the "Budgetary Factors" category listed in Exhibit 4.4.4.1, those service gaps which might be filled if additional budgetary resources were available are listed. These items could be addressed by first identifying which ones can reasonably be included within the ACR budgets. Secondly, service providers need to have and present consistent and appropriate messages to IISs with respect to funding of the items. As well, creative attempts at overcoming these items should continue to be made by the ACR partners, especially at the local level. These items in the "Information System Factors" category set out in Exhibit 4.4.4.1, raise the issue of the design and operation of information systems in support of the objectives of ACR. Partnership at much higher levels in the respective government organizations is required if the necessary information systems for referrals and client tracking and monitoring are to be developed. Exhibit 4.4.4.1: Service Gaps Not Addressed by ACR Components 1. Economic factors
2. Budgetary factors
3. Information system factors
4. Management factors
5. Capacity factors
The "Management Factors" listed in Exhibit 4.4.4.1 lists items identified in the local ACR committees' planning processes that would require, for the most part, more involvement on the part of the management level of one or more of the partners to address. The local ACR committees may not feel empowered to act on these items, because policy changes or broader resource or inter-agency decisions are required. Some of the items like "No bridges to the regular school system", "Lack of involvement with cooperative programs in the local education system", and "a policy requiring clients to go through an agreed-upon ACR process before being sponsored for training" may represent substantial opportunities for improvement and better outcomes that could reasonably be pursued through a broader partnership with local school systems and other organizations in the community. The "Capacity Factors" category in Exhibit 4.4.4.1 lists gaps not addressed by ACR components that may not have sufficient capacity to meet the needs of IISs in one or more of the pilot project communities. Decisions to add more resources to these programs and services do not appear to be within the control of the local ACR committees or perhaps the immediate partners to the ACR/SI. In summary, the ACR pilot projects have made very substantial progress in the integration of service design and the development and enhancement of client-centred programming for IISs. At the same time there are a number of very important and influential factors which do not necessarily fall within the scope of the ACR/SI to change, but nevertheless must be dealt with by front-line staff and service providers on a day-to-day basis in meeting the needs of recipients of income assistance. Some of the key problem areas are:
As part of the formative evaluation process, the ACR Evaluation Steering Committee wanted to have the Action Planning process reviewed. "Did the range of support/services/training opportunities provided assist participants to develop their action plans?" In summary, virtually all of the ACR program components in the pilot projects led to the development of an action plan, a skills plan, or a training plan. Two issues that should be considered in connection with the action planning processes of ACR are whether too many plans are created for an individual client or whether separate plans that focus on different issues of the same client are well-integrated and coordinated. IISs clients may participate in more than one component and, as a result may have more than one plan. They may also have plans set up through direct consultation with Training Consultants at MoEST or Employment Counsellors at HRDC. The question arises whether all these plans come together as a well integrated and coordinated "individualization skills plan" as envisioned in the ACR Cooperation Agreement. "What was the nature of clients' and others' involvement in developing action plans?" With approximately 38 individual components, it would be impossible to list all of the different ways in which clients and others were involved in the development of action plans. In general, action plans were developed at the end of a program component, after the processes involved in the component were complete. These processes variously included presentation of information, self-assessment processes, assessment of skills, personal and career diagnostics, counselling with respect to barriers to employment, and so on. In every instance, a client-centred approach was taken whereby the individuals were supported in developing their own plans. In many cases, subsequent ACR components were designed to further assist clients who had difficulty with the planning process in the initial ACR component. In the final analysis, involvement of the individual IIS in the action planning process was more intense and comprehensive than had been the case in previous programs at individual partner levels. "Was the nature of involvement of clients and others appropriate?", and "was the level of involvement of clients and others appropriate?" No significant findings were identified in connection with the nature or level of client and other involvement in the "action planning process". In general, it appeared that both the nature and level of involvement of clients in action planning processes were appropriate. A related issue was the concern expressed by some Committee members that the action planning process, as conducted in programs like Starting Points should begin at the point of first intake of a IIS and not seven months later under the business rules of the BC Benefits program. This issue could not be resolved in the course of the formative evaluation and would require more detailed study. "What suggestions/comments for involvement of selves and others would clients make?" This question could not be answered in the formative evaluation because a client satisfaction survey was not conducted.
"What tracking/monitoring mechanisms are in place to collect information on participants and activities of the ACR Pilot Projects?" There has been intense activity in the area of tracking and monitoring and it has been a somewhat problematic activity for some of the ACR pilot projects.
4.6.1 Feedback and Evaluation by Participants In almost all of the projects, IIS participants complete feedback and evaluation forms at the conclusion of their courses, to inform the service providers of their views about the strengths and weaknesses of the courses just completed. In some instances, these reports are summarized for management reporting purposes and the results are made available to the local ACR committee. 4.6.2 Monitoring of IIS Progress As well, most of the Starting Points programs and many of the other components featured the preparation of Action Plans which the service providers could use to monitor and follow up IIS progress. Follow-up efforts included facilitator reports on IIS participation, voluntary, one-to-one follow-up sessions and participating in events to commemorate various successes. Weekly progress reports were required from many service providers. Semi-annual contract summary reports were also required in some cases. Other monitoring methods included voluntary drop in sessions and a bi-annual participant survey. In general, all of the Starting Points components included monitoring activities to keep in contact with the participants and assist them in completing their Action Plans. Also, some of the pilots arranged to have Action Plans of IISs made available to the Training Consultants of MoEST and the Financial Assistance Workers of MSS. This was done after release of information authorizations were obtained from the IISs. At least one local ACR committee felt that the reporting and monitoring mechanisms in place were excellent. This view was shared by some of the service providers. Other service providers felt that the paperwork required to satisfy all the partners (ministries) was excessive. There are some concerns in connection with the monitoring processes. First, there is the question of how to achieve long term tracking which will make it possible to evaluate "long-term attachment to the work force" outcomes. Most of the tracking described in connection with the ACR components was short term in nature, focusing on time frames like six weeks, six months or one year after participants have completed their programs. These time frames are useful, but are not adequate for assessing lasting outcomes given the fact that many individuals on income support "cycle" in and out of the welfare system over a period of months and years. Second, there is the problem that in many cases it is the contracted service providers who are doing most of the monitoring through call backs described above. The issue here is what will be done after the contract terms have ended. It is very unclear whether long term monitoring of the kind required can actually be achieved by contractors whose contracts last only one or two years. Some sort of succession plan may be required for the ACR partners or subsequent contractors to carry on with client monitoring after the preceding contracts have expired. Third, there is the problem of information confidentiality discussed in more detail in the Starting Points section of this report. Efforts are currently under way to implement new and more useful information systems on the part of MoEST for purposes of monitoring the progress of IISs through training and into employment. Through these efforts and in partnership with the other ACR stakeholders, monitoring "long-term attachment to the work force" outcomes needs to be enabled in a manner which produces relevant, complete, consistent and continuous information over the longer term.
"What potential do the ACR/SI projects manifest for innovation?" In addressing this issue, it is worthwhile to note that an often repeated comment during the evaluation was "there is nothing new in the world of services provided to individuals on income support". This comment serves to illustrate that efforts to provide services to IISs that lead to enhanced employment outcomes have been ongoing for many years and any expectation that it is possible to develop a "silver bullet" solution is unrealistic. On the other hand, there is significant opportunity for innovation and new ideas in the development of assessment, counselling and referral strategies and program components. The best example of this is the Starting Points program which has been described earlier in this report. This program began as an innovation that was found to produce fruitful results and was subsequently developed further by HRDC and professional experts who refined the program to what it is today. In the first round of ACR/SI components there were many innovations at the level of the local ACR committee operations and the level of the service providers who ultimately delivered specific program components. One of the key innovations was the support centre concept that was developed and implemented in the Burnaby and Fort St. John ACR pilot projects. With the development of support centres, permanent locations were established where individuals on income support could go to obtain support, use facilities like computers and fax machines, and meet to discuss their progress amongst themselves and with other support workers. Other innovations took the form of existing programs which were modified to meet the specific needs of individuals on income support. An example of this was the Moving Forward component in Campbell River where a previous program for academic upgrading was modified to meet the specific needs of IISs. In summary, the ACR/SI gave rise to significant innovation in program delivery for recipients of income assistance, particularly because of the delegation of responsibility for design and development to the local community level, on a partnership basis. Although many of the pilot components were extensions or repetitions of previous programs, the flexibility inherent in local needs assessment and component design established the possibility for innovation and gave rise to a significant number of new initiatives focused on meeting the needs of IISs.
4.8.1 Background By undertaking the ACR/SI in partnership, the federal/provincial Cooperation Agreement brought together two autonomous governments, the Canadian federal and British Columbia provincial governments, to develop and operate the program components. In so doing, two separate governments with different jurisdictions, operations, organization structures, and decision-making processes had to work together toward a common goal. At the provincial level, the partnership included two separate ministries, MSS and MoEST. Differences existed between these two ministries. MSS focused on administering income assistance, including establishing and monitoring the eligibility of individuals on income support, while MoEST focused on programs of support and intervention to assist IISs to move from dependence on income assistance to re-joining the workforce. The partners' organization structures or operations created some challenges for the local ACR committees and the Steering Group. For example, in the case of HRDC, a delay in budget allocation at the regional level hampered the ability of at least one local ACR committee to establish contracts for specific ACR components. In the case of MoEST, this provincial ministry had a highly-centralized contract administration system before ACR. Contracting authority and the authority to make payments in respect of contracts was concentrated in headquarters. This contrasted sharply with the local ACR committees' need to be able to establish contracts at the local level on a timely basis to be able to secure ACR service providers. 4.8.2 Significant Change Dynamics at the Partner Level A number of changes at the partner level occurred that affected the achievement of the local ACR committees and the Steering Group. First, MSS underwent a series of reorganizations before ACR and the change management dynamics were continuing to be felt. A key outcome of these re-organizations was the separation of the Training Consultants from MSS. Prior to separation, the MSS Rehabilitation Officers who performed this function had worked in close proximity to the MSS FAWs. The Rehabilitation Officers moved to the new Ministry of Skills, Training and Labour and became Training Consultants. The ministry subsequently became the Ministry of Education, Skills and Training. The Gove Commission of Enquiry also created substantial challenges for MSS as that ministry re-organized and became the Ministry of Human Resources. At MoEST, the field structure which included the newly transferred Training Consultants described above was in the early stages of development. Even without ACR, the roles and responsibilities of Training Consultants and other MoEST staff at the local level were in the process of being developed. Changes were also occurring at HRDC, as that organization prepared for a conditional transfer of certain responsibilities to the provinces and the predecessor Unemployment Insurance scheme was being transformed into the new Employment Insurance program. The local ACR committees would have to deal with the changes described, while at the same time designing and implementing the new ACR components on a partnership basis. As a result, the overall set of challenges to the local ACR committees and the staff of the partner organizations was extreme, due to organizational and policy changes occurring within the partner organizations. 4.8.9 Partnership at the Field Level Partnerships created at the field level under the ACR Strategic Initiative were very effective at the planning stage of the pilot projects. In the planning phase most of the local ACR committees were assisted by independent consultants retained by the Provincial Steering Committee. In this way, the nature and extent of client needs in the local areas and the types of programs which were being offered by the various levels of government were identified. Also, the local ACR committees identified gaps in the services on offer to the IISs in the areas. Based on this information, the local committees prepared Service Delivery Plans which took into account the existing programs of all the government partners. This process was very effective and the partnership concept worked very well to produce integrated, locally specific plans for meeting the needs of IISs in the areas. The concept of partnership proved to be less effective once the local ACR committees began to put their Service Delivery Plans into operation. Due to a lack of infrastructure alignment within and between the separate government organizations comprising the partnerships, the operations and administrative processes required to implement the ACR components were not facilitated efficiently or effectively (e.g., contracting, information systems, processing payments, etc.). As well, the business direction of MoEST changed with the adoption of new policies under the BC Benefits Program and the Youth Works initiative. The result was that the local ACR committees had to reassess their priorities and Service Delivery Plans in relation to the new direction being taken. The impact was that the enthusiasm and commitment of at least some of the local ACR partners was dampened. They felt that their efforts had been wasted and their roles had been compromised because the assessments and decisions which they had made in good faith through the planning processes described above were being overridden without consultation, and indeed in ways which they did not necessarily agree with. Also, the new rules for BC Benefits and Youth Works were seen to create additional work for MSS staff responsible for referring IISs into the ACR components. This came at a time when workload capacity of MSS staff was already a serious concern. The result was that MSS staff in some areas refused to develop the additional work processes needed to refer IIS clients to ACR components. Referrals into the components ceased and the pilot projects in these areas bogged down without any changes in contractor costs. A curious feature in the implementation of the ACR pilot projects was that the local committees were not uniformly required to implement the new BC Benefits rules. Some did not, and chose instead to continue with their original Service Delivery Plans. In other cases, local management at the provincial level required the BC Benefits rules to be implemented irrespective of the wishes of the local committees. This produced uneven outcomes across the pilot projects, so that the pilot projects themselves would not be immediately comparable. In the final analysis, the results of the way in which the BC Benefits Program was rolled out were that the overall business direction of ACR/SI was made uncertain, the original roles and responsibilities of the ACR partners became confused and the partnerships at the local ACR committee level were weakened. However, in overall terms, although there were many challenges to be overcome, much progress was made by the partners to the ACR/SI in working together to innovate and enhance services available to IISs in the pilot project communities. 4.9 Best Practices and ACR/SI In the case of the ACR/SI, the flexibility given local ACR committees in designing pilot project components was intended to produce a number of variations in programming which could be observed for the purposes of determining best practices. In this formative evaluation, it was not possible to identify best practices from one or another pilot projects because performance measurements of the various alternatives were not taken and were not available. Also, a process of developing consensus around a potential "best practice" had not been implemented. Better information may be available as the result of a summative evaluation of the pilot projects. However, it is important to realize that a summative evaluation may not produce detailed assessments of individual practices because of the cost and level of analysis required for this purpose. Therefore, an ongoing best practice process is required.
| |||||||
![[Previous Page]](/web/20060127013508im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127013508im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127013508im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}