Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

10.1 Synopsis of Main Findings

The main findings of this study are clear:

• Training programs in Manitoba that have offered job training and work placement to job-ready clients reduce income assistance use and increase economic independence.
• Interventions that offer basic education to income assistance clients with low education, limited work experience, and family barriers to job readiness do not typically place clients jobs. Basic education is an essential pre-condition to job-related training.
• The average time to come off income assistance ranges from 16 to 23 months. The duration data show that many clients experience long spells on income assistance. These clients typically have substantial educational and work experience deficits requiring sustained and costly interventions to reverse.
• Day care and collateral supports are essential to encouraging single parents on income assistance to participate in training. These supports need to be available on-site where the training is delivered.
• Taking Charge! broadened the program and moved away from the objectives of the Strategic Initiatives program to meet client and employment goals. In this transition, it changed its relationship with community-based organizations. These organizations ceased being partners and became training contractors.
• Employability and individual job plans, a central feature of Taking Charge!, are now often done by training service providers. In supporting this transition, Taking Charge! has evolved to being a training broker.
• During the start-up, government appointed representatives of non-profit groups to sit on the Board. These organizations offer Taking Charge! critical insights into the needs of target clients. When these organizations also receive substantial contracts, an appearance of conflict-of-interest may be created. Taking Charge! has developed conflict-of-interest guidelines to deal with service providers who sit on the Board, and these guidelines have been reviewed by legal counsel. However, Taking Charge! should consider alternatives to direct representation of service providers on the Board. An advisory council, ex-officio representation, and rotating positions are possible approaches to obtaining the advice of service providers.
• Taking Charge! has not developed a structure to evaluate alternative approaches and interventions. The CAMS systems is not used for evaluation or strategic planning. Its main role is to support employment facilitators in matching clients to training opportunities, although its potential to support the evaluation of specific interventions will increase once financial and contract data are integrated.
• The training agenda within Taking Charge! remains determined by external contractors who propose courses and projects. Taking Charge! staff and management have started to challenge these proposals and approach certain providers with its own proposals. It is likely that Taking Charge! will become more proactive in developing training requirements and may even start to call for tenders on specific programming. Such a development marks an important evolution in the program. If Taking Charge! elects to serve higher needs clients, it will be required to define a training and development agenda and then request contractors to meet these needs. Taking Charge! may wish to hire training and development staff, as opposed to using contracts to meet the needs of such clients.
• As for individual outcomes, Taking Charge! and the Comparison Group programs raise post-intervention wages, increase the hours worked, and lower income assistance payments. Taking Charge! is an expensive program in relation to the Comparison Group, but several qualifications are important:
  • the follow-up period is short and limits our analysis of long-term impact
  • Taking Charge! has more clients with significant barriers to economic independence.
  • the collateral supports such as "cafeteria -style training" and day care, add to costs, but are essential to encouraging single parents to participate in training.
  • initial delays meant that Taking Charge! was not fully operational until late 1996 and this compromised full performance.
  • the favourable view of the Comparison Group arises from Employment Connections which offers a limited program to "job-ready" and "training-ready" clients who are easier to place than the clients in Taking Charge!.

10.2 Recommendations on Current and Future Delivery Options

Evidence from the Taking Charge! evaluation shows that short-term training interventions offered to "job-ready" clients can produce positive short-term outcomes in higher wages and departure from income assistance. Our qualification that these results may not be long-term is based on recent evaluations of training programs.⁷⁶

The evidence from three decades of research on labour market interventions is clear. Short-term training designed to get economically disadvantaged persons into jobs typically does not offer a sustained solution to economic independence. The results in Manitoba may be the coincidence of training delivered to job-ready clients and a rapidly growing economy.

We classify our recommendations into two broad areas:

• Changes for the delivery of training programming by the province; and
• Changes in how Taking Charge! operates.

10.2.1 Provincial training policy needs to offer a coordinated set of training programs

With the province assuming responsibility for labour market interventions and training, this is an opportune time to recast policy, based on the following recommendations:

• Develop the concept of an employability spectrum;
• Define the services based on client need;
• Training services should specialize in specific types of interventions;
• Coordinate services into a seamless delivery system;
• Follow-up and database development to assess outcomes.

Training of income assistance clients should be based on the idea of an employability spectrum.

The idea of employability is abstract. The current assessment process culminates in the assignment of levels of employability and work expectations which serve as the foundation of this process. With the advent of a one-tier approach to income assistance, these assessments will probably be completed with greater consistency. The current assessment questionnaire can probably continue to serve as the basis of this assessment, but several changes are needed:

• This assessment questionnaire has never been subjected to standard reliability and validity testing to refine the measures used to predict employability. No evidence exists that counselors can "predict" employment or develop training plans that lead to employment if they use this tool.
• Employment assessments need to be completed by E&IA counselors who then participate in the creation of a training plan with service providers such as Taking Charge!
• Periodic reassessments are needed to track the progress of income assistance clients toward independence.

In other words the employment assessment process should be a dynamic one that traces the progress of clients toward self-reliance. Income assistance clients can be placed on a spectrum based on their employability assessment. Programming can be designed to meet the needs of clients along the spectrum.⁷⁷

Training interventions need to be aligned with client needs.

Once income assistance clients are placed on the employability spectrum, counselors can assign them to specific interventions. The province should identify specific interventions and the six categories of training used by Taking Charge! and other programs are useful classifications (Table 32).

Implicit in the notion of assigning clients to interventions is the idea that expectations must vary by employability. It is expected that clients who have completed high school, have job experience, and have recently become income assistance clients should return to work quickly if unemployment remains low. These clients only need a brief intervention to help with a job search. Here the sanctions and benefit reductions associated with work expectations are part of the policy mix.

Clients with low education and no job experience will require sustained interventions. Others will be permanent income assistance clients. Some who are disabled or too old to become re-educated will likely not return to work, regardless of the amount of training or level of sanction.

Training providers need to specialize.

Employment Connections is effective because it specializes in job-ready clients. Taking Charge! attempts to meet the needs of a diverse client group through a series of contracts to external training providers. When compared with the performance of Employment Connections, it appears much less cost-effective. Such a comparison is unfair because it does not consider the extent to which educationally disadvantaged income assistance clients move along the employability spectrum when participating in programs such as Taking Charge!

In addition to assigning clients to specific interventions, specialization requires those training providers to serve specific client segments. Some training services should focus on raising the education of "Level 3" clients; others should work on placing clients that are job-ready.

A portfolio of training programs will offer a seamless continuum of services.

Once the province identifies discrete client needs, it can develop specific services. A service such as Employment Connections may serve as the final program, accepting job-ready income assistance clients that have recently come on the welfare rolls, as well as clients who have "graduated" from training and educational improvement programs. Alternatively, the province could contract with private placement companies to assist job ready clients in finding work.

Taking Charge! could be "reinvented" as a program specializing in clients with low education and having barriers to education/employment. The emphasis on a "welcoming" environment, personal development, volunteering, mentorship, and the collateral support services are particularly important to these clients.

An outcome follow-up process will allow government to manage the training programs.

Manitoba Measures requires all government departments and eventually third-party delivery agents to prepare business plans, develop performance measures, and report on plan fulfilment both internally and externally. The province will need an outcome follow-up process to track the cost-effectiveness of alternative interventions and the progress of clients toward economic independence.

Integrated Service Management and the one-tier initiative will result in better tracking of program inputs. In future, the province will have much better information on income assistance clients and the programming they receive. Nevertheless, a key omission is the measurement of outcomes. Without a systematic process of follow-up and outcome verification, the province will never be able to assess cost-effectiveness properly. This evaluation used the available information to the maximum, but we cannot make any inference about employment outcomes of those who disappear from the income assistance rolls.

Training providers such as Taking Charge! are in a poor position to complete such follow-up, because it conflicts with the non-bureaucratic style that is so integral to their service delivery model. Therefore, if follow-up is to be complete, as it must be, the province must undertake it as a core part of its accountability process under Manitoba Measures.

10.2.2 Taking Charge! is well positioned to offer services to the most needy of income assistance clients

If a program such as Taking Charge! did not exist, it would have to be invented. Taking Charge! offers services to single parents with limited education and job experience. It needs several specific adjustments:

• A focus on high needs clients.
• The Board should include some government representation.
• Training contracts need to be focussed.
• Partnerships with community organizations must be reviewed.

Taking Charge! should focus on clients with the greatest need.

Taking Charge! should focus on personal-development and educational services to help training-ready and multiple-barriered clients. Single parents with low levels of education and work experience can benefit most from the cafeteria-style training, personal-development programs at the head office, and day-care supports. The welcoming atmosphere and mentoring of clients who are making progress are very strong assets that Taking Charge! offers in encouraging income assistance clients to work.

The outcome of these programs is not employment but measurable improvement in employability (through the assessment process), and the increased ability and success clients have in pursuing additional/higher education plus trades training.

Taking Charge! should not offer work placement, job-search assistance, and other labour market skills training. By focusing on high needs clients and moving them to higher education and trades skills, Taking Charge! is alleviated of the responsibility to place clients into jobs. Taking Charge! has not had much success in connecting with private firms and creating work placements. Agencies that specialize in contacting employers and matching trained workers to vacancies do this best.

Board appointees should include some government representatives and exclude contractors.

Taking Charge! should remain an independent, non-profit organization, but with some Board appointments from key government departments and services (Family Services, Education and Training, Industry, Trade and Tourism).

This reconfiguration would increase communication between Taking Charge! and other education/training services, without compromising its ability to create partnerships and joint ventures with other organizations.

Taking Change! should focus its training contracts with a well-defined agenda.

One of the more important developments in the history of Taking Charge! was the expansion of contracting to community-based organizations and other training providers. This changed the relation between Taking Charge! and community groups, requiring management of large-scale contracts, and a level of monitoring that consumed significant resources. It also separated Taking Charge! from local community organizations.

Taking Charge! needs to define specific training goals to meet the needs of client segments. It then must decide whether to meet those needs through training its own staff or by contract. External service providers should be selected on the basis of competitive bids in response to a request for proposals. It is still possible to accept and review unsolicited proposals then Taking Charge! may choose to fund in whole or as a partnership.

Taking Charge! should use a tendering process that responds to the educational and training objectives of its clients. Contracts with service providers should be limited to courses that benefit the clientele. Typically these should be for educational upgrading and be offered by providers with a demonstrated capacity to offer these programs and certify that clients have reached levels to allow them to participate in further skills training.

Evaluating the outcomes of programs to enhance literacy and numeracy skills is more challenging than determining whether a client has found work. If Taking Charge! chooses to focus on high needs clients and offer basic education, it will need to develop measures to track the success of these interventions.

Taking charge ! needs to work is close, partnership with community organizations.

If Taking Charge! were to cease being a major source of contract funding for the training "industry" and non-profit organizations, it could participate in legitimate partnerships with local community organizations that address the needs of economically disadvantaged clients. The process of referral between Taking Charge! and these organizations would improve. Taking Charge! could offer services on location jointly with these agencies. Level 2 and 3 clients would then access the services of both Taking Charge! and the community organizations and probably move along the employability continuum faster.

10.2.3 Information systems are basic to evaluating training programs

An initial expectation was that this evaluation would be able to assess the cost-effectiveness of individual program components. For example, it is important to know whether literacy/numeracy training is more effective than to technical education, everything else being equal. Also desirable is the ability to compare the cost-effectiveness of the same intervention offered by different programs and different service providers.

To achieve this level of detail requires two important types of information not available to the evaluation:

• It is essential that all programs and service providers classify interventions similarly and maintain detailed information on client attributes. Programs and service providers seek to change their training programs to make them attractive to both clients and funders and classifying their offerings can be challenging.

  Further, outcomes need to be tracked in the same way for all participants in all programs. In this evaluation, employment outcomes were inferred from the income assistance database (i.e. SAMIN). Clients who disappeared from the database after the intervention clearly leave social assistance, but it is not possible to differentiate those who have become employed, moved, married, or are in jail.

• Another essential requirement for comparing the cost-effectiveness of various interventions is accurate information on costs. This is especially so if stand-alone programs are compared with training courses offered by government or a non-profit organization and embedded with other activities. Extracting accurate overhead costs to be attributed to the intervention can be very difficult when the training is part of a portfolio of activity. Activity based accounting systems go a long way to resolving this difficulty, but require that personnel log their time and that within the larger organization, the training activity is set up as a cost centre that purchases its inputs (rent, heat, etc.) from the host agency.

This evaluation has clearly identified the cost-effectiveness of Taking Charge! relative to the Comparison Group (within the constraints noted above). To compare training outcomes at the intervention or service provider level requires significant additional investment in information systems that record the nature of the intervention and track participants after their involvement. This is a key lesson from this evaluation that could usefully be applied evaluating training programs in general.

10.3 Final Conclusion

Taking Charge! has been a worthwhile experiment. Its experience underscores the importance of specialization and of offering services to economically disadvantaged people. However, by offering a broad spectrum of services, it limits its ability to partner with community organizations. Reconfiguring Taking Charge! will require the province to redefine an overall education and training strategy for income assistance clients.

Taking Charge! should be repositioned and sufficiently funded to deliver education and training programming to those facing more extensive barriers to employment. As with all government training programs, Taking Charge! should only support those clients who are interested in coming off assistance and who do not have severe family and personal problems that impede success. Taking Charge! must remain an education and job-preparation service and not become a one-stop centre to serve all client needs.

The expected outcomes of such a revised program would not be employment, but clients who are able to take further trades and technical training to survive in the modern labour market. The culture of the organization, the background of the staff, and its collateral programming are ideal for supporting these higher-needs clients to start the path toward employment. "Graduates" of Taking Charge! would then move to take higher levels of training and participate in job-readiness programming.

Footnotes

76	This review is forthcoming as part of the HRDC Lessons Learned Series.
77	Not all income assistance clients will benefit from training. Some may simply have too many barriers to self-sufficiency to make investment in training cost-effective.