|
|
     |
||
 |
     |
 |
|
Strategic Initiatives
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Table of Contents]](/web/20060127002936im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif) |
In any month, Winnipeg has about 7,500 income assistance clients who are single parents with dependent children. Encouraging and supporting these clients to seek and secure economic independence is challenging. Most are women and many have low levels of education and little job experience. Child care responsibilities significantly impede these clients from obtaining education and training.
Taking Charge! is a project1 funded under the Strategic Initiatives Program of Human Resources Development Canada (HRDC) and the Province of Manitoba (Department of Education and Training and the Department of Family Services). Taking Charge! was intended to be a pilot project to explore alternative approaches to social security reforms to enhance the employability of single parents on income assistance.
Specifically, government wished to ascertain which training approach was the most effective in returning income assistance clients to work and reducing the expenditures on welfare. Taking Charge! was to focus on fostering self-reliance, building partnerships with community-based non-profit organizations, between government departments, and with the private sector.
Recently in Manitoba, the growing economy has made it easier for those on assistance to find work. Manitoba has also introduced welfare reform with its "Making Welfare Work" program. These initiatives impose sanctions (reduced payments) on those income assistance clients (without dependent children under six years of age) who are judged employable and who do not search for a job or participate in training. These two factors have complemented and supported all training programs directed to income assistance clients.
Manitoba offers a range of training programs to income assistance clients. In addition to Taking Charge!, these include:
Taking Charge! is an experimental program targeted to single parents on income assistance to improve their self-reliance and their long-term economic and social well-being. It is an incorporated, non-profit agency, governed by a Board appointed by both the Government of Canada and the Province of Manitoba. Taking Charge! offers a broad range of services designed to help income assistance recipients complete training, locate employment, and increase their independence from welfare.
This program was originally allocated $26 million by both the governments of Canada and Manitoba for five years starting in 1994. Due to start-up delays, Taking Charge! will spend about $15 million by March 1999.
Taking Charge! accepts single parents on assistance who register for the service. Clients are assessed for their employability and classified into one of three levels ranging from job-ready (Level 1), training ready (Level 2) and those who have substantial educational/work experience deficits (Level 3). An independence plan is prepared to guide training and job search. Training is offered by Taking Charge! staff or more commonly training service providers under contract to Taking Charge!.
Taking Charge! offers a wide range of services including:
Taking Charge! programming serves a very diverse clientele. About a third of clients are "job-ready" in the sense that they should be able to make the transition to work with some skills upgrading and instruction on how to look for work. Other clients (about a third again) have higher barriers to economic independence such as low education and limited work experience. These clients require more extensive education and training to gain long-term employment. Finally, still other clients lack the education and experience but also have family and/or personal problems that impede their economic independence. Personal problems cannot be too extreme. For example, no provincial training program accepts clients with drug or alcohol dependencies.
Taking Charge! is intended to produce individual-based and system-based outcomes:
Individual-based goals are:
System-based goals include the following:
Government expected Taking Charge! to meet individual employment objectives by increasing clients' job readiness, employment, duration of employment, and earnings. It was also expected to meet employer's needs for job-ready employees. With respect to income assistance, if successful, Taking Charge! would reduce benefits paid, as well as duration and likelihood of future dependency of income assistance clients participating in the program.
In terms of service delivery, Taking Charge! success would show in:
An evaluation steering committee comprised of representatives from the federal and provincial governments, as well as Taking Charge!, created an evaluation framework. This framework presents detailed questions that guided all aspects of the evaluation.
The evaluation methodology defined Taking Charge! participants as the "Treatment Group." The other training programs offered by Manitoba, especially Employment Connections, Community Partnerships, Pathways to Success, and Opportunities for Employment became the "Comparison Group." Finally, income assistance clients who had received no training formed the "Control Group." Using income assistance files, program information on types and timing of training, and client interviews, the evaluation addressed most questions posed in the framework.
The Taking Charge! Summative evaluation included eight main tasks:
The case studies and survey of service providers replaced a previously intended task, the policy simulation tool (Task 14a). It was agreed that this latter task could not be supported with the information available, while the replacement tasks would provide valuable additional information.
These findings reflect program activity between January 1995 and September 1997, with outcomes tracked to June 1998.
In terms of systemic objectives, Taking Charge! shows qualified success.
Because Taking Charge! offers service to a diverse clientele with important collateral needs (such as child care), it will be a higher cost option than programs that offer short courses to job-ready adults with no dependent children. Therefore, over the relatively short follow-up period, Taking Charge! has higher costs than the Comparison Group programs, both in cost per client trained and cost per employed client. The cost-benefit analysis shows that Taking Charge! does not produce savings in payments to income assistance clients that offset the costs of the program within the follow-up period. In general, Comparison Group programs do return their costs in the form of reduced income assistance payments and less time on assistance.
The evaluation of outcomes after nine months is a brief period over which to demonstrate effectiveness. At the same time, Taking Charge!'s success in placing job-ready clients is largely influenced by the same factors that support Employment Connections and other Comparison Group Programs that work with these clients. Short-term, job-specific training offered to relatively well educated and experienced clients within a growing economy has made all job training programs serving these types of client appear to perform well.
Because Taking Charge! addresses a range of clients, many with low education, no job experience, and dependent children, its overall costs are higher and show less success in the job market. Taking Charge! could quickly improve its measured performance by concentrating only on job-ready clients. However, it makes little sense to focus more provincial training on job-ready clients, given the large and growing numbers of income assistance clients with poor education. As the evaluation showed, the typical income assistance client is less educated and has less job experience compared to those who graduated from job training programs. To become economically independent, future income assistance clients will need more costly and longer support than those who have moved through the system.
Taking Charge! decided to broaden the original design where its own staff assessed clients and created a tailored job plan. To meet the original (and ambitious) targets for the number of clients served required that training contractors undertake assessment and independence planning on behalf of Taking Charge!. Now many clients directly approach contractors to be assessed. Some of these contractors advertise directly to clients for their Taking Charge! funded training. Clients accepted in this way may have only cursory exposure to Taking Charge! and its programming and therefore, obtain few of the collateral benefits of participating in Taking Charge!.
In broadening the model, Taking Charge! has moved closer to other training programs offered by Manitoba, except for the extensive and expensive client supports. In making this transformation, Taking Charge! is not in a position to test alternative interventions. Even the pilot projects it funds are not systematically evaluated because it does not consistently complete client follow-up and does not maintain a database to support such assessments.
Taking Charge! is a valuable training program for income assistance clients that have low education, little job experience and who have dependent children. These clients benefit from the collateral supports such as childcare and personal development courses.
Taking Charge! is a flexible program that can respond to such client needs. Because the staff and management have lowered the barriers to accessing education and training for clients, it is well positioned to serve that growing number of income assistance clients who are eager to become independent from welfare but who lack education and experience. As such, the province needs interventions such as Taking Charge! if it hopes to address the needs of many of its income assistance clients.
At the same time Taking Charge! is not cost effective in supporting job-ready clients for their re-entry to the workplace. It also failed to meet many of the original expectations placed on it by the funders. It cannot support the evaluation of alternative training programs and social security reforms, largely because the management information system has not been sufficient to the task. Further, it is very difficult to welcome training clients and also follow-up to verify outcomes.
Taking Charge! has not created strong community and business partnerships. Its relationships are largely contractual. With its large budget allocation, Taking Charge! is viewed by many non-profit groups and private contractors more as a funding source and less as a partner to address social issues.
The evaluation was able to compare the cost effectiveness of Taking Charge! and the Comparison Group programs as a whole. However, it was not able to compare the effectiveness of specific interventions or individual service providers. Doing so would have required significant additional investment in information systems and a commitment to activity-based accounting.
Taking Charge! should be repositioned and sufficiently funded to deliver education and training programming to those facing extensive barriers to employment. The culture of the organization, the background of the staff, and its collateral programming are ideal for supporting these higher-needs clients on the path toward employment. It should not compete to train job-ready clients since this is already done by other programs. As with all government training programs, Taking Charge! should only support those who are interested in coming off assistance and who do not have severe family and personal problems that impede success. Taking Charge! must remain an education and job-preparation service and not become a one-stop centre to serve all client needs.
The expected outcomes of such a revised program would not be employment but clients who are able to take further trade and technical training to survive in the modern labour market. Graduates of Taking Charge! would then move to take higher levels of training and participate in job readiness programming.
Under such a revision to the terms of Taking Charge!, it will serve higher needs clients. The duration of participation will be longer and more expensive, simply because these clients have more ground to cover before they can reasonably expect to be self-reliant. It is likely that programs such as Taking Charge! can never show a net benefit in the short-term, and neither should they be expected to.
|
Manitoba Family Services and Housing and Education and Training, as the provincial co-sponsors of the Taking Charge! Initiative, jointly submit this response to the Evaluation conducted by Prairie Research Associates Inc.
An initial commentary about changes to Manitoba's income support program during the life of Taking Charge! is relevant. In May 1996, during the developmental phase of Taking Charge!, Manitoba introduced a major package of welfare reforms. A key element was the expectation that single parents in receipt of income assistance, the target group for Taking Charge!, would actively seek employment or take measures to enhance their employability. A process to assess a client's employability and to develop a job plan was instituted. As well, close linkages were established to training programs and opportunities offered through Manitoba Education and Training. These, and other reform measures, played a significant role in revamping the business practices of the Employment and Income Assistance program and in helping to shape the service delivery models implemented by Taking Charge!.
The following comments on specific points noted in the Evaluation are provided:
The implementation of Welfare Reform in Manitoba resulted in the development of several training and employment-focused partnerships that offer opportunities to income assistance clients. In addition to Taking Charge!, Manitoba Family Services and Housing refers clients to Manitoba Education and Training, created working arrangements with other provincial departments such as Manitoba Natural Resources and Manitoba Northern Affairs, and entered into agreements with outside agencies such as Opportunities for Employment.
This has provided single parents in receipt of income assistance with a range of options to assist them in decreasing or eliminating their dependence on welfare. Compared to these other training interventions, Taking Charge! has several unique features.
The Evaluation raises concerns about how Taking Charge! interacted with its service providers with respect to client assessment and follow-up, and effectiveness monitoring of these agencies. Manitoba agrees with these concerns and suggests that Taking Charge! be more proactive with service providers.
The Evaluation raises concerns about the assessment and referral process. Manitoba Family Services and Housing and Education and Training reviewed those processes with a view to improved quality and responsiveness. Both departments agree on the importance of proper assessment and follow-up, as well as the need for appropriate referrals to interventions in helping clients achieve self-sufficiency. As experience was gained, assessments and referrals to Taking Charge! and other interventions evolved and will continue to be refined.
It should be noted that Manitoba Education and Training developed the assessment tool and leveling concept referred to in the Evaluation. These were subsequently adopted by Taking Charge! and the Employment and Income Assistance program. The assigning of employability levels to clients was intended as a management tool to facilitate and organize resources to meet the programming needs of clients. The assessment tool was envisioned as a comprehensive tool to assist in determining the most appropriate intervention for an individual client.
The Evaluation indicates that these tools were not used consistently, or in some cases, appropriately to achieve their intended objectives. Manitoba agrees with these findings and suggests that a more comprehensive approach be used in the future to assess employability and the necessary interventions.
The need to analyze the impact of any intervention is critical to determining its success. Manitoba Family Services and Housing has implemented enhancements to its information system that provide dependency outcomes for clients by type of intervention undertaken. This information enables a continuous monitoring of employment, earnings, and time on assistance for all training interventions.
A key objective of the Taking Charge! Initiative for Manitoba was to learn how best to match clients with interventions by determining which interventions were most successful with which types of clients. This information would then be used to refine existing practices and processes, and to improve the results being achieved by the variety of employment training programs being delivered by provincial departments.
The Evaluation was not able to answer this question for several reasons, which are instructive in designing future evaluations of complex employment training interventions like Taking Charge!. First, there were too few clients in come programs to yield statistically significant information. Second, at the level of generic types of intervention like work experience or trade skills training, there was insufficient post intervention time to detect the full outcomes of the training. Evaluations employment training programs in the United states demonstrates the need for at least three years of post-intervention data to properly assess the net impact of, in particular, more expensive and lengthy interventions. This Evaluation had, at most, 18 months of follow-up information. Third, and activity-based accounting system was required to determine the overhead costs of each type of intervention provided by Taking Charge!. As that had not been an envisaged requirement from the outset, it was not implemented and, as a result, only the direct costs of those interventions contracted out to third parties were tracked. Consequently, comparative cost-effectiveness analyses of similar types of interventions carried out by Taking Charge! and other provincial employment training programs could not be carried out by the evaluator.
Nevertheless, the evaluator was able to confirm that, over the short follow-up time period covered by the Evaluation, training interventions featuring job placement/work experience and the provision of skills training were the most likely to result in increased levels of employment and reductions in income assistance. Job search interventions also result in reduced benefits due to earnings. By comparison, interventions providing technical/management skills, literacy/numeracy training, and academic upgrading showed no immediate impacts on employment and dependency on income assistance. These findings are helpful to Manitoba to focus Taking Charge! on the employment training gaps left by the current mix of provincial programming.
Additionally, the Province wanted to identify and test new approaches to service delivery via the Taking Charge! pilot. The Evaluation met that objective by identifying a number of the project's strengths such as being able to engage single parents not subject to a work expectation, offering a welcoming environment to clients with low self-esteem, providing innovative services such as clothing for job interviews and placements, and confirming the value of a holistic service model.
Overall, Manitoba agrees with the key recommendation of the Evaluation that Taking Charge! was successful and should have an ongoing role in assisting single parents income assistance clients to transition to work and self-sufficiency.
| ____________________ Dan Haughey, Executive Director Welfare Reform |
____________________ Mary Lou Kuxhouse, Executive Director Employment & Training Services Branch |
|
On behalf of Taking Charge! it is a pleasure to respond to the summative evaluation.
It should be noted that the evaluation was conducted during the 2 years after the programs' inception. This is important in that, as a relatively new organization Taking Charge! was continuing to evolve and was not more established as assumed in the evaluation framework. In spite of that shortcoming and some inaccurate conclusions resulting from that model, the evaluation is considered by management to have been a useful exercise. It has provided a milestone by which we are able to measure the success of our efforts over that two-year period and provided an opportunity to confirm many of our operating assumptions. Since the evaluation there has been significant accomplishments in our operations so that we are now more fully developed.
There is general agreement in the report that Taking Charge! was successfully delivering a very valuable and much needed service. The summative evaluation report concludes that Taking Charge! in only its first two years was able to demonstrate "the importance of specialization and offering services to the economically disadvantaged." I agree!
But Taking Charge! is a unique program and it continues to evolve as we seek innovative approaches to individual needs. As such, I must disagree with the reports' conclusion that: "by offering a broad spectrum of services we have limited our ability to partner with community organizations". It is my belief, that it is by accessing that broad spectrum of services that we have been successful in finding innovative solutions, which has allowed us to meet the needs of our clients.
The board of directors and staff were instrumental in creating an organization that represented every aspect of the community from social programs, daycare to business. This community based perspective and valuable first hand experience working with single-parents contributed to the success of Taking Charge!
I am confident that Taking Charge! will continue to meet the needs of single parents and I would like to thank Prairie Research for assisting us in that effort.
Rosa Walker
Executive
Director
|
This report presents the findings of the summative evaluation of Taking Charge!. This evaluation examines the impacts, outcomes, and effectiveness of Taking Charge!, while considering the social, political, and economic context within which it operates.
Taking Charge! formally came into existence in 1994 as a joint federal-provincial project under the Strategic Initiatives Program outlined in the federal budget of 1994. The federal government committed funds between fiscal years 1994/95 and 1998/99 to support a number of provincial and territorial government programs. These projects encompass a range of social security and labour market interventions including:
Strategic Initiatives projects usually have a target clientele such as single parents on income assistance, members of equity groups, disabled persons, youth facing job entry barriers, and mature workers needing retraining.
Allocated $26.2 million, to be shared equally by both levels of government, Taking Charge! was originally intended to assist 900 single-parent income assistance recipients annually with the expectation that 500 would be placed into employment each year.2 Over the five-year life of the program, 4500 clients were to have been assisted. The program goal is clear:
Test and demonstrate, over a five year period, from 1994 to 1999, an integrated, accountable model for delivering services to single parents who are income assistance recipients.
A most important point to stress is that program operations did not commence until well into 1995 as shown in Table 1. For reasons that are common to many Strategic Initiatives, initial activity concentrated on creating an organizational structure. Operations were delayed while the Board created the policies needed to support the range of services contemplated in the Memorandum of Understanding between the federal and provincial governments. In Section 5, we explore several other factors that delayed immediate implementation of Taking Charge!.
|
TABLE 1 Chronology of the Taking Charge! program |
|
| Date | Action |
| September 1994 | Memorandum of Understanding signed |
| March 1995 | Federal and Provincial governments appoint Board |
| April 1995 - January 1996 | Board engages in planning, facilities upgrading |
| November 1995 | First contract training (Health Care Aide and Call Centre Training) |
| January 1996 | First internal program offered |
| March 1996 | Facility opened |
| April 1996 | Full programming |
Taking Charge! is a joint federal-provincial program developed by the Manitoba Government (Department of Family Services and Department of Education and Training) in partnership with Human Resources Development Canada (HRDC). Taking Charge! is designed to enhance the employability of single parents on income assistance through training and/or work experience. An important objective of Taking Charge! is to forge partnerships with the community, broadly defined to include business, social services, educational institutions and the non-profit sector. As a pilot program, Taking Charge! is also intended to test an innovative model for delivering these services.
The Province of Manitoba's Employment and Income Assistance Act and its regulations form the legislative basis for Taking Charge!. Three agreements define its operations: A Memorandum of Understanding between Canada and Manitoba; the Canada/Manitoba Contribution Agreement Concerning Taking Charge!; and an Agreement Respecting the Taking Charge! Job-ready clients between the Government of Manitoba and Taking Charge! Inc.
Taking Charge! operates as an independent non-profit corporation with a Board of Directors appointed by the partners to the agreement. The original target set by the Board in its business plan was to assist 900 single-parent, income assistance recipients each year, with the expectation that 500 would be placed into employment. Over the five-year life of the program, this translates into 4,500 clients served by Taking Charge!. However, given the initial two-year process on which some programming was delivered, but much of the energy was devoted to planning and organizational development, Taking Charge! will have had only three years of operation by March 1999, which translates into 2,700 clients served. In fact, as of June 1998, Taking Charge! reported contact with 3,552 income assistance clients.3
Taking Charge! was not implemented as intended. Program impacts need to be judged in the context of actual implementation approach and timing. Several important factors, external to Taking Charge!, affected the timing of implementation:
The evaluation process started in June of 1997, just a year and a half into substantive programming and this summative evaluation is concluding three and a half years after effective program implementation.
Given delayed start-up, Taking Charge! had two choices:
As we shall show, Taking Charge! elected to do the latter and employed a "fast-tracking" model. This important decision, along with the delayed implementation, are the two salient issues that have qualified the outcomes for the program.
As a project funded in part through the Strategic Initiatives Program, Taking Charge! is expected to meet a series of objectives, as laid out in the three agreements that defined its operations.
Projects created and funded by Strategic Initiatives usually have a target clientele such as single parents on income assistance, members of equity groups, disabled persons, youth facing job entry barriers and mature workers needing retraining. In the case of Taking Charge! the target group consists of single parents on income assistance in Winnipeg. The vast majority of clients are women (92 percent), and about 40 percent are Aboriginal.
By way of summary, the objectives for Taking Charge! can be classified into three categories: service delivery, employment, and social security reform.
Service Delivery Objectives
Employment Objectives
Social Security Reform Objectives
The Phase 1 (formative evaluation) described the degree to which the stated objectives of the job-ready clients are reflected in the design and implementation of the project. It also assessed the adequacy of the information systems to address the issues of the second phase. The formative evaluation also collected information on the following aspects of the program:
Findings from the formative evaluation provided information about how successfully the project has been implemented. It revealed several important concerns about contracting training services and the information system. At the same time, the formative evaluation showed high client satisfaction.
The summative evaluation report focuses on the outcomes, impacts, and effectiveness of Taking Charge!. It has the following eight objectives:
An evaluation steering committee created the evaluation framework that specified the issues and questions to be addressed in the formative and summative evaluations. This committee was comprised of representatives from HRDC, the Department of Family Services, the Department of Education and Training and Taking Charge! This study systematically responds to each issue and question. For certain questions, data insufficiency impeded a response to particular issues and questions and we note these as they arise.
The Taking Charge! summative evaluation included eight main tasks:
Follow-up survey of participants (Client satisfaction)
The follow-up survey tracked outcomes for 1,001 individuals on income assistance. Participants were divided into three groups: Treatment (Taking Charge!), Comparison (Employment Connections, Opportunities for Employment, Pathways to Success), and Control (individuals receiving income assistance with no substantive intervention recorded prior to September 30, 1997). This survey provided information on program outcomes, including activity and income levels at least nine months following program completion. It also captured client satisfaction with income assistance programs and the latter effects on clients' attitudes.
Cost analysis
Financial data collected from Taking Charge! and other income assistance programs formed the basis for assessing the cost of delivering income assistance programming.
Follow up interviews with key informants
Prairie Research Associates Inc. staff interviewed 26 key informants, including Taking Charge!'s management and Board of Directors, government officials, community organizations, and training service providers. These interviews provided specific insights into Taking Charge!'s operations from individuals involved with the program. Many had been included in the interviews conducted during the formative evaluation.
|
TABLE 2 Evaluation questions |
| Relevance/rationale |
| 1. What is the legislative base for Taking Charge! What is the involvement of the federal and provincial government? |
2. Has Taking Charge! met the criteria established for Strategic Initiatives:
|
| 3. How many target group members are likely to be in need of Taking Charge!? |
| 4. To what extent does Taking Charge! reach the intended target group? Do participants represent the target group? If not, for what reasons do discrepancies occur? |
| 5. Are the services/interventions provided responsive to and consistent with participant needs? |
| 6. What similar services are being provided by other existing programs? |
| 7. What gaps in the ongoing/existing array of services/programs are being filled by Taking Charge!? |
| Design and delivery |
| 1. What are the design and delivery features of the program (i.e., components, activities, and relationship between components and activities)? |
| 2. What are the strengths and weaknesses of the program design? |
| 3. What are the strengths and weaknesses of the pilot project organizational structure? Are the roles and responsibilities of the various partners and service providers (e.g., delivery agent, management committee, Board members, staff, volunteers) clearly enunciated? |
| 4. How are individualized plans with participants developed and how do these plans meet their needs? |
| 5. Does Taking Charge! provide sufficient and appropriate resources (human, financial, physical) to participants, service providers, and employers? |
| 6. To what extent did participants discontinue before their anticipated completion dates? What were the main reasons for discontinuation? |
7. To what extent is the community involved in service delivery and development? What are the linkages, how have they been developed, and how successfully have they been developed?
|
| 8. a. What tracking/monitoring mechanisms have been put in place to collect information on participants and interventions? Are these adequate for measuring project impacts?
b. Have Control/Comparison Groups been identified? What criteria have been used? |
| 9. Have any operational/legislative/regulatory constraints been identified that impinge on the ability of the project (or single parents) to achieve objectives? Are the project design features (i.e., operational guidelines that define eligibility criteria, funding limits, etc.) consistent with the stated objectives of the project? |
| Project success - individual outcomes |
| 1. To what extent has the project prepared new participants for achieving self-sufficiency? a. Increased their motivation and self-esteem b. Helped them develop career action plans c. Upgraded their educational skills d. Provided them with occupational skills e. Provided them with pre-employment training f. Provided them with work experience g. Provided them with self-employment/business skills h. Provided them with mentors/role models |
| 2. What is the net impact on the participants' employability, by type of program/intervention and for completers/non-completers? a. Improvement in labour market attachment b. Improvement in employment earnings c. Improvement in quality of jobs secured |
| 3. What is the net impact on participants' self-sufficiency, by type of program/intervention and for completers/non-completers? a. Change in number and duration of spells on income assistance? b. Change in amount of income assistance and Employment Insurance (EI) received c. Type and amount of income assistance and Employment Insurance (EI) received d. Reason for continued dependency on assistance |
| 4. Has the project brought about any changes in participant's home/family life (e.g., family interrelationships, health status, involvement in other community activities, use of other community services, types of recreation, etc.)? |
| 5. To what extent has the project assisted participants to achieve self-sufficiency? a. What project activities or interventions were most effective and for which type of clients? b. For completers/non-completers? c. For what reasons do some participants remain unemployed and on income support after the program? d. Did Taking Charge! motivate participants to go on to further training? |
| 6. To what extent did the project improve the long-term prospects of participants and their families escaping the cycle of poverty? |
| Project success - delivery system outcomes |
| 1. To what extent has the project succeeded in developing successful partnerships among the various levels of government, employers, and community groups? To what extent has it succeeded in integrating federal/provincial dual services? |
| 2. How satisfied are participants with various aspects of the project (e.g., application and selection, services provided, etc.)? |
| 3. To what extent has the project succeeded in removing disincentives to employment and training (e.g., changing income assistance regulations, etc.)? Or alternatively, in increasing incentives (e.g., earned income supplements, day care, nurturing, health benefits, etc.)? |
| 4. To what extent did participants displace employees already on an employer's workforce? Were these jobs permanent? |
| Project cost effectiveness |
| 1. What are the costs of individual components comprising the program? |
| 2. What is the per diem cost per participant? Per component? |
| 3. What is the cost per participant, per program completion, per employed participant? |
| 4. What is the net cost or saving resulting from the difference in subsequent level of income assistance and Employment Insurance dependency attributable to program participation? |
| 5. What is the benefit cost ratio and payback period? |
| 6. Is the Taking Charge! model a cost-effective way of achieving project objectives? Are there more cost effective methods of achieving the same objectives? How do the results compare with those of other programs with similar objectives (e.g., brokered services, collocation)? |
| 7. What lessons can be learned from Taking Charge! on interventions to assist the target group? To what extent does it contribute to the development of a policy framework for social security reform? Does Taking Charge! lead to a more efficient delivery of services? To what extent can this experience be extended to/adopted by other jurisdictions? |
Survey of employers
The survey of employers collected the opinions of employers who hired individuals from a work placement program funded by Taking Charge!. Results from the survey were meant to provide a profile of employers involved with Taking Charge!, but change in the privacy legislation limited the response rate and sharply reduced the usefulness of this survey.4
Econometric analysis of program impacts
Data from the various training programs as well as the income assistance program managed by the Department of Family Services formed the basis of estimates of net program impacts (incremental changes in benefits paid, incremental duration on income assistance, etc.) across the participant population.
Survey of training service providers
A fax-back survey of 54 training service providers collected information on the types of services offered, the number of students graduated, the perception they have of clients, the challenges clients face and pose to service providers, and the relationship service providers have with Taking Charge!, other income assistance programs, and other service providers.
Case studies
Five case studies of Taking Charge! participants provided qualitative information on their experiences with the program. Case studies of program participants can reveal important features of a program. In particular, they show how a program transforms individual lives. To draw upon a range of experiences with Taking Charge!, respondents were randomly selected according to specific demographic characteristics (gender, ethnicity, family size).
File review
The file review collected information on all service providers contracting with Taking Charge! to provide training. Information was updated from the formative evaluation and verified by Taking Charge! for accuracy.
The case studies and survey of service providers replaced a previously intended task, the policy simulation (Task 14a). It was agreed that this latter task could not be supported with the information available, while the replacement tasks would provide valuable additional information.
This report is comprised of two volumes. Volume I is the Summative Evaluation and Volume II is the Appendices.
Volume I has 10 sections:
Volume II is provided under separate cover and includes the following:
Appendices
| Appendix 1 | Client Satisfaction Survey Results |
| Appendix 2 | Employer Survey Results |
| Appendix 3 | Training Service Provider Survey Results |
| Appendix 4 | Summary of Key Informant Interviews |
| Appendix 5 | Summary of Training Contracts |
| Appendix 6 | Econometric Estimates of Individual Outcomes |
| Appendix 7 | Case Studies |
|
Taking Charge! is a five year, federal/provincial Strategic Initiatives pilot project for single parents on income assistance. It was originally funded for five years with $26 million in funding from HRDC, the Manitoba Department of Family Services and the Department of Education and Training. Delays in program start-up have slowed program expenditures and Taking Charge! will spend about $15 million by March 31, 1999. Its employment oriented programming focuses on skill training, job preparation, academic upgrading and adult literacy. Direct client services include assessment, orientation, cafeteria-style training, child care support, and volunteer opportunities.
Taking Charge! was designed with several unique features:
Taking Charge! has the flexibility to adapt quickly to changing market conditions
Taking Charge! operates as a community-based, third party, non-share capital corporation with a Board of Directors appointed by the provincial and federal governments. This organizational structure is intended to enable Taking Charge! to react quickly and design innovative training programs without requiring ministerial approval. This is supposed to give it added degrees of organizational flexibility and the ability to respond to local need and opportunity.
Taking Charge! can provide additional support to clients while they are involved in the program
Taking Charge! provides an array of collateral client supports. To accommodate clients and improve their transition to employment, Taking Charge! stays open evenings and on Saturdays. It also provides one-stop convenience with an on-site employment income assistance office where clients can pick up their cheques. A well-equipped day care facility is available for short-term use with staff assisting clients to obtain permanent day care. Taking Charge! also has a computer laboratory, where clients can use a variety of education and business software (cafeteria style training), and an "executive closet" with clothing for clients to seek employment.
Personalized assessment and independence planning focuses training
Taking Charge! clients complete personalized training plans and assessments to focus their training on the most suitable option(s). Taking Charge! employment facilitators were originally intended to manage this process fully. Because of start-up delays, a client backlog formed and left Taking Charge! unable to assess the entire caseload. To deal with this issue, a "fast-tracking" assessment process enabled clients to enrol directly with service providers who completed the assessment and independence planning.
Enhanced service delivery through partnerships with private sectors, non-profit organizations and government
Taking Charge! has a flexible organizational structure enabling it to partner with private businesses, non-profit organizations, educational facilities, community-based groups and government. These partnerships come in two general forms. First, many organizations offer training under contract to Taking Charge!. Typical examples are post-secondary institutions, non-profit groups, and private companies. These relationships are basically commercial and contractual. Second, Taking Charge! networks and forms alliances with other community-based organizations and private firms. Through these partnerships, Taking Charge! attempts to offer a broad range of services to its clientele, increasing both access and options for training.
Three important developments have occurred simultaneously during the creation of Taking Charge!. These are:
Economic recovery has increased the number of job opportunities in Manitoba. The province's recovery has been faster than the national average and most economic forecasts predict continued growth. Increased employment growth means that job opportunities should expand throughout the economy. Income assistance recipients who are recent additions to the assistance rolls, and who have few barriers to re-employment, should have increasing likelihood of moving off assistance now compared to a few years ago.
However, prior to the recent recovery, income assistance caseloads did not decline. Clearly a complex relationship exists between income assistance caseloads and overall economic activity.
Manitoba's Making Welfare Work/Job-ready involves a range of training and labour market programs offered by the province, municipalities, and the private sector. These programs include employability assessments by the Department of Family Services to identify those who are employable. The Department of Education and Training offers training for those not job-ready and job placement to accelerate the re-entry of those with employable skills.
Under the "Making Welfare Work" Job-ready every income assistance applicant is expected to meet a number of conditions.
Deferrals for work expectations are granted for health or other reasons (e.g., family violence or children with special needs). Failure to comply with these provisions may result in sanctions in the form of reduced benefits (a budget reduction of $50) for those with a work expectation who do not meet the expectation.
Parallel to Taking Charge! are other training programs and initiatives. An example is Employment Connections, which assists income assistance recipients (not just single parents) to find employment. Other examples include Community Partnerships, Opportunities for Employment and Pathways to Success. Various non-profit organizations also offer high school equivalency, computer skills, life skills, job search and other courses related to increasing labour market attachment.
The federal government has devolved its employment services and training functions to the province. This has reduced the role of HRDC and increased the importance of provincial departments such as Education and Training and Family Services.
In summary, Taking Charge! has been launched in an economic environment that is improving. Policy changes have increased the incentive for income assistance recipients to seek employment. While Taking Charge! is not the only training intervention offered in Manitoba, the agreements that created this Program clearly envision it as a prominent and comprehensive portfolio of employability measures directed to single parents on income assistance.
The primary labour market interventions for provincial income assistance clients in Manitoba include Employment Connections, Community Partnerships, and Opportunities for Employment. In addition to Taking Charge!, these three programs handle most of the income assistance caseload in the province.
Employment Connections
Operated by the Manitoba Department of Education and Training, Employment Connections provides job search support services to help income assistance recipients market themselves to potential employers. This process includes a comprehensive, three week training program that focuses on job search skills and strategies, with ongoing follow-up support and placement services provided to registered clients. Employment Connections also has a Self-Marketing Centre offering incidental services such as resume production, printing and mailing services, and labour market information to individuals undertaking an independent job search.
In 1997/98 Employment Connections served 937 income assistance recipients, with 81 percent completing the program. It contracted with a number of service providers to offer training to its clients.
Community Partnerships
Community Partnerships, a government-run organization (Department of Education and Training), contracts with non-profit, community-based organizations to develop and deliver a variety of programs (employability skills training and/or work experience placements) to individuals on provincial or municipal income assistance. Activities include employability assessments, pre-employment preparation, academic/literacy training, specific skills training, work experience and employment placement. In 1997/98 Community Partnerships had 29 active projects and served 517 income assistance recipients.
Individuals are referred to Community Partnerships by the Manitoba Employment and Income Assistance and Municipal Assistance staff. Project partners (training contractors) and/or Education and Training staff are responsible for screening clients for selection into a program. In the past year, Community Partnerships has contracted with the Salvation Army (job readiness/job placement program), the Family Centre of Winnipeg (In Home Family Support Worker Program), the International Centre (customer service training), and the Aboriginal training and employment services (clerical and carpentry training) to deliver training programs. They have also contracted with 8 community-based adult literacy programs to deliver employability related literacy programming for 135 income assistance clients.
Opportunities for Employment
Opportunities for Employment is a private organization that helps job-ready individuals on provincial or municipal income assistance find employment. Opportunities for Employment provides short-term job orientation sessions, job placement, ongoing support once a client is placed and helps locate employment opportunities with private sector employers.
Opportunities for Employment has a unique funding arrangement with the provincial government. It receives $4,000 from the province for every client the program places into full-time employment (30 or more hours/week) for a six month period. Conversely, Opportunities for Employment receives no payment for participants who do not achieve this objective. In this sense, Opportunities for Employment is a true partner with government. If it is unable to place a client into full-time work, it must absorb the training costs.
The profile of the program is derived from three separate sources offering somewhat different perspectives.
The first two sources are aggregate in that they offer no information for individual clients. The third information source allows us to associate specific interventions with individual clients. It is a feature of the information management system at Taking Charge! that these three data sources offer different profiles of the program. It is outside the terms of this evaluation to reconcile these discrepancies.
At times we reached different totals and could not reconcile apparent discrepancies. These variations are not serious in most cases.
Taking Charge! offers a wide range of programming and client supports, such as individual employment assessments, cafeteria-style training, personal development programs, day-care supports, and transportation allowances. These services give Taking Charge! a unique profile among income assistance programs.
Taking Charge! started substantive program delivery in April 1996. Table 3 outlines program activity from the beginning of the 1996/97 fiscal year to the first quarter of 1998/99. The term "activity" refers to a program component within Taking Charge!. Clients may take more than one activity and most typically participate in two or three.
According to Taking Charge! year-end reports, 1,178 new clients registered with Taking Charge! in 1997/98. By this count, the number of new Taking Charge! clients decreased by 219 from the previous fiscal year (1,397 clients). These activity figures appear to under-represent Taking Charge! activity when compared with information from the Taking Charge! database (current July 21, 1998). In Table 12, registration for 1996/97 is shown as 1,596 clients, whereas 1997/98 registration is 1,284 clients. Differences between these two sets of figures may reflect changes in client status, updates to the database, or differences in how "clients" are defined and measured.5
In 1997/98, 983 individuals received a job-training/job-planning assessment (83 percent of clients registered) from either Taking Charge! or a service provider contracting with them. This marks an increase of 24 percent from the previous year, when 59 percent of clients had an assessment.
During the 1997/98 fiscal year, Taking Charge! increased its delivery of cafeteria-style training. During this time the number of clients receiving cafeteria-style training more than doubled, reaching 2,161 participants (Table 3), while spending on these services increased fivefold (see Table 4).
|
TABLE 3 Summary of program activity by fiscal year - all clients |
|||
| 1996/97 | 1997/98 |
1998/99 (first quarter) |
|
| Direct client services | |||
| Registration | 1,397 | 1,178 | 239 |
| Orientation sessions | 60 | 38 | NA* |
| Assessments | 830 | 983 | 388 |
| Presentations/events | 302 | 111 | NA* |
| Cafeteria-style training (clients) | 933 | 2,161 | NA* |
| Taking Care | |||
| Children in child care | 178 | 237 | 50 |
| Children in pre-school | 564 | 582 | 183 |
| Children in Saturday and | NA* | 342 | 40 |
| evening care | |||
| Children placed | NA* | 241 | 62 |
| Source: Information taken from Taking Charge! year end fiscal reports. Note: The "NA*" designates information not available at time of reporting. |
|||
Taking Charge! expenditures are divided between administration and programming, with program expenses for direct client services listed by component activity. The largest expenditure is attributed to training service provider contracts (representing 62 percent of total expenditures in 1997/98). Other notable expenditures include child care and client transportation, accounting for 12 percent of the total budget.
As outlined in Table 4, Taking Charge! spent about five times more on cafeteria-style training in 1997/98 than the previous year, raising its expenditures to $102,045. Increased spending reflects a concerted effort by Taking Charge! to further develop its in-house services. This has also been reflected in payments for professional services, which accounts for contracts with individuals involved with personal development workshops, computer lab training, Taking Jobs!, and the Taking Initiative! program.
|
TABLE 4 Taking Charge! expenditures by fiscal year |
||||
| Fiscal year (ending March 31) | ||||
| 1995/96 | 1996/97 | 1997/98 |
1997/98 % |
|
| Administration | $95,150 | $213,975 | $240,174 | 4.2 |
| Programs-Direct client services | ||||
| Cafeteria-style training | - | $21,993 | $102,045 | 1.8 |
| General expenses (cost of program delivery) | $141,276 | $650,059 | $723,198 | 12.5 |
| Program expenses (child care, program contracts) | ||||
| Child care | $30,401 | $220,534 | $312,114 | 5.4 |
| Client allowance | $11,203 | $20,388 | $10,031 | 0.2 |
| Client training supplies | $3,750 | $19,728 | $18,056 | 0.3 |
| Client transportation | $19,923 | $191,771 | $379,296 | 6.6 |
| Client tuition fees | $14,872 | $41,805 | 0.7 | |
| Programs (contracts to service providers) | $210,833 | $2,194,076 | $3,599,092 | 62.2 |
| Rent | - | $6,421 | - | - |
| Professional fees | - | - | $139,811 | 2.4 |
| Taking Care | $17,610 | $185,916 | $218,234 | 3.7 |
| Total | $530,146 | $3,739,733 | $5,783,856 | 100 |
| Source: Year-end fiscal/activity report. | ||||
Expenditures on contracts to external training service providers rose by $1,405,016 in fiscal 1997/98, for a total of approximately $3.6 million. When compared to the previous year's funding allocations, the proportion of funds spent on training contracts remained similar, increasing by only 3 percent.
Expenditures for client tuition fees also increased during the 1997/98 fiscal year, as more clients took training at educational institutions. This approach to training reflects the use of "purchase of training" where the agency or government acquires a seat in a class.
Taking Charge! uses clients of its programs as volunteers to complete follow-up calls and perform various clerical duties. This offers work experience as part of the employment training and is a formal part of the programming. Taking Charge! also uses former clients who volunteer time to work as mentors with current trainees (see Table 18).
Two client supports, child care and transportation allowances, also increased during the 1997/98 fiscal year by $91,580 and $187,525 respectively. These two supports are key elements in the Taking Charge! model, and are intended to help clients overcome the many barriers single parents encounter in training and job search. The ease with which Taking Charge! can directly assist clients is often cited by management as a benefit of operating as an independent non-profit organization.
Since commencing operation in 1995, Taking Charge! has formed partnerships with a variety of community organizations. These partnerships can be classified in two general categories:
Through these partnerships, Taking Charge! is intended to offer a broad range of services to its clientele, as well as increase access and options for training.
Of the two types of "partnerships," contractual relationships are by far the most prominent. Taking Charge! has developed contracts with many of the province's key service providers such as McKnight and Associates and Horizons Management Systems in the private sector, non-profit organizations such as New Directions and Urban Circle, and educational institutions such as the University of Manitoba, Red River Community College, and South Winnipeg Technical Centre. These contracts form the basis of the Taking Charge! delivery model as it is currently practiced.
Training service providers are the primary organizations delivering training for Taking Charge!. Under the current system, many training service providers assess clients, plan their programs, and deliver the training. As such, training service providers play a much more important role in providing training than originally envisioned.
Profile of programs by year and type
As indicated in Table 5, Taking Charge! has funded 178 training programs (103 training contracts, 61 individual seat purchases,6 10 personal development, 4 Computer lab7), with approximately $6 million spent on training programs as of March 30, 1998.
Profile of 10 highest contracts per value of training costs
Taking Charge! has relied extensively on established training service providers from the private, non-profit, and educational sectors to provide training. Organizations such as McKnight and Associates, New Directions and the University of Manitoba have all played important roles in the training process.
Values for individual contracts for training service providers ranged from a high of $665,000 for McKnight and Associates to a low of $777 for Lifelore Ltd (training costs less GST) (See Table 6).
McKnight and Associates and New Directions received the four highest priced contracts totaling $1,556,729 (training costs only). These four contracts represent 19 percent of the total expenditure on contracts by Taking Charge! and 14 percent of all clients trained in a substantive intervention. McKnight and Associates' programming focuses on job placement and job preparation, and deals primarily with individuals who are job-ready.
The top 10 contracts represent 32 percent of expenditures on training costs and 31 percent of total costs (relative to the total amount spent on all contracts). Nearly one-quarter of all client training is accounted for by these ten contracts.
New Directions offered programs to clients with education and work experience deficits (Level 3).
|
TABLE 6 Top ten individual contracts as per value of training costs |
|||||
| Taking Charge! file number | Name of service provider | Value of contracts | Number of clients | ||
| Training costs | Support costs | Total costs | |||
| 121 | McKnight and Associates | $665,000 | $83,384 | $748,384 | 176 |
| 77 | New Directions | $335,410 | $66,561 | $401,971 | 77 |
| 213 | McKnight and Associates | $285,000 | $102,839 | $387,839 | 75 |
| 145 | New Directions | $271,319 | $69,015 | $340,334 | 63 |
| 148 | Horizons Management Systems | $220,000 | $122,955 | $342,955 | 80 |
| 116 | SLA Computer Systems | $189,000 | $34,341 | $223,341 | 44 |
| 168 | Lifelore Ltd. | $184,320 | $50,371 | $234,691 | 50 |
| 181 | The University of Manitoba | $184,000 | $27,513 | $211,513 | 19 |
| 81 | Occupational Rehabilitation Group of Canada | $164,520 | $25,678 | $190,198 | 64 |
| 74 | Lifelore | $144,900 | $37,180 | $182,080 | 44 |
| Total | $2,643,469 | $619,837 | $3,263,306 | 692 | |
| Net total of all contracts | $8,316,343 | $2,206,105 | $10,522,443 | 2837 | |
| Top ten as percentage of all contracts | 32% | 28% | 31% | 24% | |
| Source: Training Contracts Database. Note 1: Costs are calculated less GST. Note 2: In some cases, support costs are paid to service providers, while in others they are paid directly to the client. As such, comparisons have been based solely on training costs, as these apply to almost all training contracts. Note 3: Information taken from file review of service provider contracts. See Volume II, Appendix 5 for information. |
|||||
Training costs per service provider - combined total of all contracts
When all contracts are combined, McKnight and Associates leads service providers with training contracts totaling $1,073,050 (combined total for all training contracts less GST). They were followed by New Directions ($735,312), the University of Manitoba ($622,650), and South Winnipeg Technical Centre ($601,359) (see Table 7 for details).
The top ten service providers (total of all contracts), account for 61 percent of the total budget spent on all contracts (training costs only) and 44 percent of all clients trained in a substantive intervention.
|
TABLE 7 Top ten service provider contracts (total of all contracts) |
|||||
| Name of service provider | Value of contracts | Number of clients | Number of contacts | ||
| Training costs |
Support costs | Total costs | |||
| McKnight and Associates | $1,073,050 | $198,224 | $1,271,274 | 275 | 3 |
| New Directions | $735,312 | $162,497 | $897,809 | 177 | 3 |
| The University of Manitoba | $622,650 | $95,726 | $718,376 | 89 | 6 |
| South Winnipeg Technical Centre | $601,359 | $154,449 | $755,808 | 128 | 6 |
| Horizons Management Systems | $454,452 | $172,689 | $627,141 | 197 | 2 |
| Red River Community College | $415,605 | $99,979 | $515,584 | 86 | 6 |
| Patal Vocational Preparation Schools | $343,148 | $106,652 | $449,800 | 73 | 4 |
| Lifelore Ltd. | $329,220 | $88,328 | $417,584 | 103 | 3 |
| Employment Projects for Women | $260,567 | $122,531 | $383,098 | 44 | 3 |
| Occupation Rehabilitation Group | $259,020 | $37,906 | $296,926 | 82 | 2 |
| Total | $5,094,383 | $1,238,981 | $6,333,400 | 1,254 | 38 |
| Net total of all contracts | $8,316,343 | $2,206,105 | $10,522,443 | 2,837 | 103 |
| Top ten as percentage of all contracts | 61% | 56% | 60% | 44% | 37% |
| Source: Training Contracts Database. Note 1: Costs are calculated less GST. In some cases, support payments are made to service providers, while in others they are paid directly to the client. Note 2: Information taken from file Review of service provider contracts. See Volume II, Appendix 5 for more information. |
|||||
Average contract cost per client
Measuring total training costs per service provider and training contract gives an indication of training expenditures relative to the total allocated budget, but it does not reflect the average contract cost per client. This latter measure gives a better indication of the relative cost of a training program. Before comparing average costs, two cautions must be made.
First, because training service providers may not offer the same kinds of programming, training cost per participant will vary widely. For example, one program may last one week while another may be six months long. Similar training courses may vary in terms of training intensity (number of instruction hours), making it difficult to directly compare these interventions. The average cost per client will therefore reflect these variations.
Second, in some cases high average training costs reflect the difficulty associated with training specific types of clients. For example, training service providers and community organizations such as Urban Circle and the Andrews Street Family Centre deal primarily with inner-city income assistance participants, many of whom are Level 3 clients with multiple barriers to training8 . In such cases, client withdrawals or terminations may be high, particularly if they are not yet "training-ready." In other cases, training service providers offer technological or trade skills training that may be more expensive per participant when compared to job placement/job preparation or job search interventions.
|
TABLE 8 Top ten individual contracts as per Average Value of Training Costs |
|||||
| Taking
Charge! file number |
Name of service provider | Average value of contracts per client | Number of participants | ||
| Training costs | Support costs | Total costs | |||
| 181 | University of Manitoba | $9,684 | $1,448 | $11,132 | 19 |
| 28 | Andrews Street Family Centre | $9,500 | $404 | $9,904 | 2 |
| 27 | Urban Circle | $8,480 | $1,268 | $9,748 | 15 |
| 180 | Red River Community College | $8,248 | $1,337 | $9,621 | 12 |
| 208 | University of Manitoba | $8,000 | $611 | $8,611 | 9 |
| 79 | Rosina's Fascio Magia | $7,256 | $1,053 | $8,309 | 9 |
| 157 | University of Manitoba | $7,117 | $1,096 | $8,212 | 12 |
| 118 | Red River Community College | $7,095 | $619 | $7,714 | 12 |
| 113 | South Winnipeg Technical Centre | $6,842 | $2,167 | $9,009 | 21 |
| 201 | University of Manitoba | $6,569 | $988 | $7,557 | 13 |
| Average value per client (all contracts n=100) |
$2,901 | $763 | $3,664 | 28.37 | |
| Source: Training Contracts Database. Note 1: Costs are calculated less GST. In some cases, support costs are paid to service providers, while in others they are paid directly to the client. Average is based on number of participants. Note 2: Information taken from file review of service provider contracts. See Volume II, Appendix 5 for information. Note 3: Average value per client was based on 100 contracts from our file review. One contract was excluded because it did not involve any clients, while 2 others were excluded because client information was not yet available. For this reason, the last row is slightly different than the amount that would be obtained by using the data from Table 8. |
|||||
Distribution of contracts by sector
The largest proportion of contracts went to service providers in the private sector (43). The remainder were distributed to non-profit firms (35) and educational facilities (24). One contract was given to a government institution (Table 9).
|
TABLE 9 Contracts by sector |
||
| Sector | Value of contracts* | Number of contracts |
| Private | $3,698,439 | 43 |
| Non-profit | $2,659,959 | 35 |
| Education | $1,960,181 | 24 |
| Government | $21,500 | 1 |
| Source: Training Contracts Database. *Value of training costs only. |
||
Profile of clients trained by training intervention
Through contractual partnerships with training service providers, Taking Charge! offers a variety of training interventions suited to different types of clients (Table 10 profiles the types of interventions taken by Taking Charge! clientele).9 In some cases, the number of graduates per intervention type overlap. This is particularly notable with interventions such as work/job entry and job placement/work experience and with academic upgrading and literacy/numeracy. As a result, the number of graduates per category will overestimate the actual number of graduates.
According to the Taking Charge! File Review, 1,312 clients have completed interventions with a training service provider.
The main goal of income assistance training is to help individuals find long-term employment and reduce their dependency on income assistance. Employers play an important role in the training process. It is their interests and needs that income assistance programs need to consider during program development.
The Taking Charge! Survey of Employers was meant to provide information on the relationship between employers and Taking Charge!, but did not do so. This failure can be attributed to two factors:
Because of the low survey response rate, information drawn from the survey of employers did not provide a representative profile of businesses that have worked with and/or hired income assistance clients following completion of a Taking Charge! sponsored intervention. Instead, a profile of employers has been drawn from the Taking Charge! database, as detailed in Table 11. This table includes data on employers who have hired Taking Charge! participants, with or without knowledge that these individuals had participated in a Taking Charge! sponsored intervention.
From Table 11, it is apparent that the large number of clients hired in retail services at $7.01 per hour depresses the overall wage rate.
One point is important about all training programs. The overall job market determines the demand for various skills and occupations. Training programs cannot create high wage employment where none exists. The average wage reflects the willingness-to-pay by business for the skills offered by the clients who graduate from the training programs.
This section provides a brief profile of Taking Charge!'s clients (Table 12).10 The tables in this section are based on an extract of the Taking Charge! database (July 21, 1998), representing program activity to date.
As the demographic profile from Table 13 indicates, almost every participant (99 percent) was a single parent. The small percentage not identified as single parents may have changed status either during or subsequent to their intervention.
Ninety-two percent of qualified participants were women. This proportion has steadily decreased over time, with a growing number of men registering with Taking Charge!.
More than one-third (39 percent) of Taking Charge! participants were of Aboriginal ancestry. The majority of these individuals are Status Indians (21 percent), while a further 12 percent were Métis.
Approximately two-thirds of Taking Charge! participants were between the ages of 17 and 34 at the time of registration, with an average age of 31. The proportion falling into this age group has remained consistent over time (Table 14).
As an aside, many Taking Charge! clients did not have a work expectation. Taking Charge! seems to have been successful in attracting and motivating younger income assistance clients to change their circumstances.
Levels assigned to Taking Charge! participants
As indicated in Table 15, approximately 75 percent of all Taking Charge! participants have been assigned an employability level. Although this proportion appears to be stable over time, this indicator may be misleading, as leveling information can be updated subsequent to an individual's registration.
One third of Taking Charge! participants are rated as "job-ready" (Level 1). The proportion of Level 1 clients has declined slightly over time, compared to 1995/96 levels.
The proportion of "multi-barriered" clients has increased between the 1995/96 fiscal year and 1997/98. Results from 1998/99 show a notable drop-off, but results are partial, based only on the first four months of the year (April 1 to July 21, 1998).
Approximately 25 percent of clients have no level recorded in the database, either because the assessment has not been done or it has not been recorded.
Just more than one-third of Taking Charge! clients have graduated from high school or received an equivalent diploma. The proportion of high school graduates registering in Taking Charge! has remained at approximately 36 percent for the past two fiscal years (Table 16).
Education and Employability Status
To test how well employability status (level assigned by income assistance counselors and/or Taking Charge! employment facilitators) was assigned, we compared it to education levels. Results are summarized in Table 17.
In general, education and employability levels appear to be correlated. Individuals with a higher education were more likely to be assigned a Level 1 status than those with less education.
|
TABLE 17 Qualified clients by education and employability levels |
|||||
| Highest grade completed |
Employability level | Total | |||
| No level | Job-ready | Training-ready | Multi-barriered | ||
| 6 to 9 | 151 | 77 | 175 | 187 | 590 |
| 26% | 13% | 30% | 32% | 100% | |
| 10/10 GED | 167 | 155 | 188 | 102 | 612 |
| 27% | 25% | 31% | 17% | 100% | |
| 11/11 GED | 88 | 145 | 165 | 48 | 446 |
| 20% | 33% | 37% | 11% | 100% | |
| 12/12 GED | 244 | 504 | 275 | 58 | 1,081 |
| 23% | 47% | 25% | 5% | 100% | |
| Unknown | 65 | 33 | 27 | 33 | 158 |
| 41% | 21% | 17% | 21% | 100% | |
| Total | 715 | 914 | 830 | 428 | 2,887 |
| 25% | 32% | 29% | 15% | 100% | |
| Source: Taking Charge! Client Database. Note: GED-General Education Diploma. |
|||||
Of the 2,887 qualified participants, 2,058 (71 percent) have completed a job training/job planning assessment with either a Taking Charge! Employment Facilitator or a service provider (See Table 18).
Comparative information is drawn from the Prairie Research Associates database used to support the econometric analysis (see Volume II, Appendix 6 for more details). This sample includes information on 820 clients completing a substantive intervention at Taking Charge!, 182 clients from Opportunities for Employment, 366 from Employment Connections, 64 from Community Partnerships, and 80 clients from the Pathways to Success program. We selected participants from the Comparison Programs to mirror the Taking Charge! population, choosing only those who were single parents (at time of sample construction). As such, demographic profiles do not reflect clients using the Comparison Programs, but they are comparable for the single parents that attended them.
Gender and age by programOf the five income assistance programs, Taking Charge! had the highest proportion of females (97 percent). Pathways was second highest at 95 percent followed by Community Partnerships (91 percent) and Employment Connections (90 percent) (Table 19).
On average, Taking Charge! and Pathways to Success had the youngest program participants, with an average age of 31. Approximately 30 percent of both Employment Connections and Opportunities for Employment clients were more than forty years of age.
|
TABLE 19 Gender and age by program (n=2,674) |
|||||||
| Program | Gender | Age | |||||
| Female | Male | 17-29 yrs | 30-40 yrs | 40+ yrs | Mean age | Median age | |
| Taking Charge! | 794 | 26 | 188 | 516 | 116 | 31 | 30 |
| (97%) | (3%) | (23%) | (63%) | (14%) | |||
| Employment Connections | 329 | 37 | 32 | 233 | 101 | 35 | 35 |
| (90%) | (10%) | (9%) | (64%) | (28%) | |||
| Opportunities for Employment | 138 | 44 | 8 | 121 | 53 | 36 | 6 |
| (76%) | (24%) | (4%) | (67%) | (29%) | |||
| Pathways to Success | 76 | 4 | 17 | 54 | 9 | 31 | 31 |
| (95%) | (5%) | (21%) | (68%) | (11%) | |||
| Community Partnerships | 58 | 6 | 6 | 48 | 10 | 33 | 34 |
| (91%) | (9%) | (9%) | (75%) | (16%) | |||
| Control Group | 1,164 | 88 | 393 | 664 | 195 | 30 | 30 |
| (93%) | (7%) | (31%) | (53%) | (16%) | |||
| Total | 2,559 | 205 | 644 | 1,636 | 484 | 32 | 31 |
| (93%) | (7%) | (23%) | (59%) | (18%) | |||
| Source: Taking Charge! Client Database. | |||||||
Ethnicity by program
Employment Connections had the highest proportion of individuals classified as visible minorities, followed by Taking Charge!. This data likely under-represents the number of individuals traditionally classified as "visible minorities." This under-representation is evident when one compares these results to the number of individuals classified as Status Indians12 (Table 20).
Pathways to Success has the highest proportion of individuals who are Status Indians (54 percent), which is not surprising given that this particular program was designed for individuals of Aboriginal descent. Community Partnerships had the highest proportion among the remaining income assistance programs at 30 percent.
|
TABLE 20 Ethnicity by program (n=2,764) |
||||
| Program | Visible minority? | Status Indian? | ||
| Yes | No | Status | Non-status | |
| Taking Charge! | 79 | 791 | 152 | 668 |
| (10%) | (90%) | (19%) | (82%) | |
| Employment Connections | 45 | 321 | 45 | 321 |
| (12%) | (88%) | (12%) | (88%) | |
| Opportunities for Employment | 15 | 167 | 44 | 138 |
| (8%) | (92%) | (24%) | (76%) | |
| Pathways to Success | 1 | 79 | 43 | 37 |
| (1%) | (99%) | (54%) | (46%) | |
| Community Partnerships | 5 | 59 | 19 | 45 |
| (8%) | (92%) | (30%) | (70%) | |
| Control Group | 109 | 1,143 | 303 | 949 |
| (9%) | (91%) | (24%) | (76%) | |
| Total | 254 | 2,510 | 606 | 2,158 |
| (9%) | (91%) | (22%) | (78%) | |
| Source: Taking Charge! Client Database. | ||||
Education by program
The five income assistance programs had a similar educational profile with the exception of Opportunities for Employment, which had fewer individuals with high school and more with an elementary education.
Family size by program
On average, Taking Charge! clients had younger families, with 58 percent having children under the age of six years, living at home for some period over the past three years.
Individuals from Employment Connections, Community Partnerships, and Opportunities for Employment all share similar family profiles.
|
TABLE 22 Family size by program (n=2,674) |
||||
| Program | Percentage of clients with children living at home by age group | |||
| <6 years | 6 to 11 years | 12 to 17 years | < 18 years | |
| Taking Charge! | 467 | 382 | 222 | 812 |
| (58%) | (47%) | (27%) | (100%) | |
| Employment Connections | 100 | 176 | 173 | 357 |
| (28%) | (49%) | (48%) | (98%) | |
| Opportunities for Employment | 36 | 112 | 91 | 182 |
| (20%) | (62%) | (50%) | (100%) | |
| Pathways to Success | 38 | 39 | 33 | 78 |
| (49%) | (50%) | (42%) | (100%) | |
| Community Partnerships | 21 | 39 | 31 | 64 |
| (33%) | (61%) | (48%) | (100%) | |
| Control Group | 848 | 509 | 356 | 1,245 |
| (68%) | (41%) | (28%) | (99%) | |
| Total | 1,510 | 1,257 | 906 | 2,738 |
| (55%) | (46%) | (33%) | (100%) | |
| Source: Taking Charge! Client Database. | ||||
Employability status by program
Taking Charge! had the highest proportion of individuals with an assigned employability level. This proportion reflects the concerted effort on the part of Taking Charge! to provide an employability level as part of its personalized training and assessment process (Table 23).
On average, 72 percent of all participants complete their intervention (Table 24). The decline in the rate of intervention completion is somewhat deceptive, as data omissions in 1996/97 and 1997/98 account for a large proportion of the decrease. If all individuals without an assigned status are assumed to have completed their interventions, the 1996/97 completion rate would rise to 80 percent, while the 1997/98 rate would rise to 68 percent, providing an overall completion rate of 77 percent. With these tentative assumptions in mind, the percentage of completions in 1997/98 is still 12 percent lower than that of the previous two years. These results may suggest that clients have become more difficult to train, as reflected in greater client attrition.
|
TABLE 24 Client participation attributes - (completed interventions only) |
|||
| Fiscal year clients accepted | Number of clients accepted | Number of completing | percent of client completing |
| 1995/96 | 235 | 196 | 83% |
| 1996/97 | 1,234 | 911 | 74% |
| 1997/98 | 476 | 296 | 62% |
| 1998/99 | 5 | 4 | 80% |
| Total | 1,950 | 1,407 | 72% |
| Source: Taking Charge! Client Database.
Note: Data in this table underestimate the actual number of completions by as much as 2 percent due to data entry omissions. |
|||
This section has established the following profile characteristics of Taking Charge! and other labour market interventions:
|
Section 3 of the report places Taking Charge! within the Strategic Initiatives, reviews its target client group and discusses program coverage.
The Government of Canada has entered into agreements with the provinces and territories to fund a variety of Strategic Initiatives (SI). These SIs are designed to test a range of innovative approaches to addressing employment barriers in ways that are consistent with Canada's social security reform. The ultimate objective of the SIs is to develop policies to reduce dependency on income assistance by increasing participation in the labour market.
In 1996/97, 7,963 single parents were on income assistance in Winnipeg every month. In 1997/98 this dropped to 7,499 each month, for an overall average of 7,731 per month. The costs to the public finances are large. Aside from the direct income assistance payments, these families make little direct contribution to the economy or tax revenues. Further, many single parents remain in poverty for an extended period and income assistance often becomes an inter-generational phenomenon, where children assume that welfare payments are a normal or even desirable source of income. Programs such as Taking Charge! seek to change this culture of poverty.
The rapid increase in the skills requirements of the modern service and manufacturing economy creates barriers for many with low education and training. Many single parents have left school well before high school completion and have had only cursory contact with the labour market.
For many years, social policy analysts have advocated increased labour market attachment to interrupt the cycle of income assistance dependency. Social services must emphasize self-reliance, both through training and by compelling eligible income assistance recipients to seek employment. Also, research from the United States emphasizes the role of partnerships with public and private training service providers, employers, and community resources as instrumental to the success of initiatives aimed at reducing reliance on welfare.13
Specifically, the objectives of Taking Charge! are:
The design of Taking Charge! is consistent with the rest of the Strategic Initiatives in terms of being client-centered and innovative, and in terms of its commitment to involve community. As well, evaluation is a requirement of Taking Charge! as it is for all the SIs.
One of the issues identified in the evaluation framework is program relevance. Taking Charge!'s potential for innovation and evaluation, as well as its potential to offer information that can be used in developing social policy, is central to its relevance. As we mentioned above, Taking Charge! elected to respond to the delayed start-up with a fast-tracking process. This process, examined in detail in Section 4, altered the original design and reduced the extent of innovation and experimentation embodied in the Taking Charge! "model." Once the Board and government adopted a target of 900 clients processed each year and 500 employed, the original design had to be amended. The one-stop shop model where clients could be assessed, offered a job plan, and directed to training, all the while receiving collateral supports (day care, cafeteria-style computer training, executive closet, etc.) was augmented with more traditional purchase of seats and contract training. This change reverted Taking Charge! to a more traditional portfolio of training interventions, and much of its uniqueness was eroded. Taking Charge! has therefore, moved some way from the original intent of experimentation inherent in the Strategic Initiatives.
As originally planned, Taking Charge! has a very direct objective: to accept 900 single parents on income assistance in each year from 1995 to 1999. Of these, 500 are expected to find employment. Over its formal lifetime, Taking Charge! was expected to train/place 4,500 single-parent income assistance clients. This represents a regular expected "inflow" of approximately 75 clients a month into Taking Charge!. The program is currently operating at this level. However, program operations date from April 1996. The initial delay in program operation meant that Taking Charge! had no activity for the first several quarters and is expected to serve about 3,700 clients by March 1999.
The "market" for Taking Charge! consists of single parents on assistance. As Figure 1 shows, at the beginning of 1998, Winnipeg had about 7,300 single-parent income assistance clients. Many of these have work deferrals as a result of disability or having children less than six years of age.
Of the total single-parent E&IA caseload, about 3,000 have had work expectations assigned.14 Taking Charge! is currently accepting about 920 clients annually, or 77 a month. By this measure, Taking Charge! is serving approximately 1 percent of its total client potential.
Whether Taking Charge! has gaps in service or duplicates what is available from other programs is a complex issue. In terms of fulfilling the original design of a one-stop service for training single parents on income assistance, Taking Charge! and its contract service providers offer a wide spectrum of services. While several service providers offer similar programs, it is of more concern whether Taking Charge!, through its wide range, duplicates other services. Specifically, an issue we raise in this section and address in more detail in Section 4 is whether Taking Charge! is expanding into areas well covered by other programs such as Employment Connections.
Taking Charge! offers a comprehensive portfolio of services both in-house and through training service providers. As is apparent from Section 2, Taking Charge! offers a comprehensive set of services for single parents on assistance. Most key informants believe that the target clientele is appropriate and few suggest that the program could expand to offer service to all E&IA clients.
In its design, Taking Charge! did not duplicate services offered in Manitoba. With the creation of Employment Connections, as well as other interventions, and decisions by the Board of Taking Charge! to stress programs to gain immediate employment, Taking Charge! evolved into a broker of a broad spectrum of training services. It has become one of several training organizations. Duplication and overlap among these interventions is likely.
After five years, the training context for income assistance clients has evolved. Taking Charge! is now one of several training programs and is unique only in the extent of collateral services such as day care and the computer lab it offers to its participants. It has moved away from the objectives of the Strategic Initiatives to meet what it perceives are the needs of single parents on income assistance. It is now less distinguished from other labour market interventions available to this target group.
|
Section 4 examines design and delivery issues related to Taking Charge!. First we examine the design features of Taking Charge!, including its organizational structure, client training services and supports, and its philosophy - all aspects intended to make it unique among training programs in Manitoba.
Second, we consider client participation and continuation in the Program, and factors that affect the likelihood of participation. This includes an assessment of personal barriers to employment and training and changes in job readiness.
Third, we examine Taking Charge!'s information systems - their structure, use and adequacy. This information builds upon findings in the Phase 1 formative evaluation.
Finally, we consider operational, legislative, and regulatory constraints to program design and delivery. We examine the structure of Taking Charge!'s Board, the leveling, and employability assessment process, Making Welfare Work and Taking Charge! within the income assistance system.
Taking Charge! is aimed at single-parents on income assistance. Unique design features include:
Management and members of the Board we interviewed emphasized that Taking Charge! offers the unique advantage over traditional government-based service in its ability to form partnerships with business and the non-government sector. With its independent Board as a non-profit corporation, Taking Charge! can enter into contractual relationships
with many diverse organizations. Specifically, it can initiate joint ventures with business, the non-governmental sector, and government.
Taking Charge! offers a range of collateral services and supports designed to help single parents on income assistance begin training. The ability to tailor service to meet individual clients' needs was cited by management and the Board as a unique feature of the service. Taking Charge! offers additional support to clients while they are involved in the program. Examples include: free day care while the client is taking courses at Taking Charge!, financial assistance for day care when taking courses at a service provider, transportation (bus passes), educational materials, course tuition, and emergency cash to meet contingencies. All Taking Charge! clients receive their normal income assistance support while participating in the program.
The importance of these supports was also affirmed by participants in the focus groups completed as part of Phase 1, and by respondents to the Follow-up Survey completed in the summative evaluation.
By way of summary:
Each client is assigned an employment facilitator who helps him/her develop career plans and obtaining pre-employment training, skills development, and job placement. Employment facilitators reported that they routinely help clients with a wide range of training and employment-related problems. Management and Board consider this a key feature of Taking Charge!'s design and delivery model.
"Welcoming" is a word commonly used by management and staff of Taking Charge! to describe how they are different from government. In interviews, most of the management and staff described their approach and the overall atmosphere at Taking Charge! as "client friendly" or "client driven." They contrasted this with government services that they represented as less sensitive to clients and, at times, more "confrontational."
Client participation to graduation is an indicator of program effectiveness. Employment facilitators monitor the training service providers and client participation. Low participation (high client drop out) is taken by the Board of Taking Charge! as an important indicator in the decision not to renew a service provider's contract.
We examined this issue in two ways:
We probed what factors contribute to client drop-outs. Most key informants identified the following factors:
Measures of job readiness (or, conversely, barriers to training and employment) include:
Table 25 summarizes the findings from the database we constructed for this study (see Section 6 for details). The number of children less than six years of age has dipped slightly, but both Taking Charge! clients and those within the Control Group are arriving for training with fewer prior jobs and less education. Most interesting is that the Control Group is a population that has spent an increasing percentage of its adult life on income assistance.
Figures 2 to 6 illustrate the changes in the attributes of the SAMIN extract over the study period. The age of clients has not changed, but fewer are arriving with children less than six years of age. Education has remained stable, but the number of jobs clients have had prior to entering income assistance has declined. Most interesting is that the amount of time clients have spent on income assistance since their 18TH birthday has increased.
This section provides an overview of the management information environment at Taking Charge! in general and the Client Abilities Management System (CAMS) database in particular.
Based on interviews with staff and on our own analysis, we conclude that CAMS has improved since the formative evaluation. Employment facilitators told us that they are using the system to identify training opportunities for clients and to match clients to proposed projects. At the same time, based on our most recent extract (July 21, 1998) we believe that the system could still be improved to better support program management and operations.
CAMS is a relational database covering many, but not all, aspects of Taking Charge!'s interaction with its clients, service providers, and employers. The database was designed in-house by Taking Charge! staff and is maintained in Microsoft Access 97, a leading end-user database product.
As a relational database, CAMS stores data into a series of tables, each containing information about a distinct object on which the program tracks information. Thus in the Taking Charge! database, client information is stored in one table, while information about the program's interaction with its clients is stored in another. Tables are related to each other through the use of common fields (called keys) that allow us to link, for example, a client to his/her interaction with the program. A full description of the database is found in the Phase 1 evaluation.
CAMS consists of two components: a set of data tables, plus an application used to populate those tables with data.
Separating the data from the application allows a developer to make and distribute changes to forms and programming code without having to migrate the data from one revision of an application to another. In an environment such as Taking Charge!, where revisions to the database were common, this distinction is consistent with the principles of sound application development.
We have three general concerns about the information systems in Taking Charge!16
Key fields in CAMS have not been defined.
As we cited in the formative evaluation, the most serious short-coming is that several important fields have not been controlled, but rather left as open text. Open text fields are extremely difficult to report. For example, the Training Program field in the Training Programs table captures the name of a client's previous training interventions. We found over 1,200 variants on training interventions for clients. We re-coded this information to support the evaluation.
Data omissions and poor quality required us to use alternate sources of information.
Inconsistent data quality within the Taking Charge! database has the following effects:
The main result of these data issues is weakened confidence in the database's ability to provide accurate statistics about Taking Charge! and its clients.
As we have pointed out, Taking Charge!'s management has taken steps in recent months to improve the quality of information in the database. Based on the extract in July, it is clear that follow-up data are being entered.
Nevertheless, there are many omissions on key fields such as client attributes, start and end dates of the intervention, and type of intervention. As we discuss elsewhere, Taking Charge! has not recorded outcome data with the precision and completeness needed for this summative evaluation (see Section 4.3.3 below). Therefore, for the measurement of individual impact, we set aside all information on client attributes, education, previous employment, and outcomes. Instead we relied on the SAMIN data and a supplementary data gathering activity for this information.17
This was also a problem for the Comparison Group. Information from Employment Connections, Community Partnerships, and Opportunities for Employment all record data on clients in different formats. Reconciling these various formats was well beyond the resources available for the evaluation, and we simply used SAMIN and EETIPJP data for all the client attributes, education/employment histories and outcomes.18
In an important sense Taking Charge! stumbled into the creation of its management system. It never had the benefit of professional database design and training that would have created an integrated management and evaluation resource tool.19 In hindsight, the expenditures to create such a system were modest in view of the pay-offs to management and evaluation.
Routine management information is difficult to locate and verify.
At this time, the Board of Directors receives a basic summary of program activity in the form of a "Quarterly Report" organized by fiscal quarter. This report summarizes the types of programs offered plus the numbers of clients attending, an overview of client attributes (visible minority, disabilities, gender, etc.), employment outcomes for graduates, and direct client service activities. Outside of the Phase 1 and the summative evaluation we found no additional in-depth analysis of client involvement by the service provider. It is clear that CAMS, has a limited role in preparing strategic information for management or the operations. We prepared the tables in Section 2 that relate client attributes and offer insights into program activity over time using very simple queries to CAMS, based on our extract. Clearly, the management at Taking Charge! is not sufficiently familiar with CAMS to prepare reports that would assist in planning future services and monitoring current activity.
Furthermore, financial information and data on training service provider activity have been maintained in separate databases or in hard copy. This means that preparing reports is complex. For example, our report on training service providers in Section 5 required extensive file review as well as a long verification process with management. Contracting with training service providers is a primary activity for Taking Charge! and reporting on it should be more convenient. We should note that Taking Charge! planned to integrate the financial management and client activity data by December 1998.
In the formative evaluation we wrote:
As a service planning aid, a database management system would allow employment facilitators to review client needs and project objectives and make a match. Independence planning is currently paper based, when it could be systems based as client attributes and needs are matched to service providers offerings. A database management system would also support the follow-up process. Simple reports could be generated on a recurring basis to flag those that should be re-contacted to establish their current activity.
A poor information system removes management from the strategic information needed to plan. A fully relational database, as Taking Charge! essentially has, would allow management to create unique and insightful reports on progress and activity. It would be a critical asset in monitoring the service providers. Rather than our manual review of files, information on average cost, outcomes, and the value for money offered by service providers would be continuously available. This would allow management to speak with greater authority on the program and its activities.
CAMS is being upgraded. This should have been an early priority, and not a process coming to fulfilment after three years. The inadequacy of the database system has not only frustrated this evaluation but also has limited management of the project.
The fact that we did not need this information from Taking Charge! (or any of the Comparison Group Programs) was fortunate. However, this exercise revealed a serious issue in the state of information systems in Manitoba available to track income assistance clients. SAMIN is designed as a financial payments management system. Recently (April and November 1997), SAMIN has been upgraded to track client activity. Additional screens have been introduced that capture education and employment history, permitting counselors to record training intervention, and referral dates of participation in an intervention. At this time, data entry is proceeding to populate the database.
The EETIPJP data are all paper-based, although they are now recorded for certain clients in PRA's database. Each training program maintains its own database and approaches to client tracking. Further, when coupled with the variation in employability assessments, these inconsistencies in data management place the province in a poor position to monitor and track initiatives in welfare reform, education and training programs, and services to income assistance clients. The Integrated Case Management initiative is also expected to enhance the province's ability to track program activity.
These improvements are important. Nevertheless, little sound information is collected on post-intervention outcomes. We can infer negatively. Clients who remain on income assistance clearly have not been successful in training. Those who are no longer on assistance may have found work, moved, gotten married, etc. It is not possible to determine the employment status of those who disappear from assistance and little can be said definitely about their economic independence.
The recent upgrades to SAMIN and Integrated Case Management are all worthwhile activities, but follow-up information on client outcomes will still be needed to assess the value of training.
Taking Charge! faces several operational, legislative, and regulatory constraints that are outside the effectiveness of the implementation of the program (discussed in Section 5).
Membership in the Board is by appointment by the Ministers of Family Services (Manitoba) and the Minister of Human Resources Development Canada. Part of the initial start-up delay arose because government delayed the Board appointments. Vacancies are also filled quite slowly with limited representation from the business community.20
The system of "leveling" is central to Taking Charge! and the employment and assistance counseling process. Upon the initial application for income assistance, counselors assign a "level" that summarizes job readiness or employability. Clients may also be assessed for employability by Taking Charge! (and its service providers) as well as by EIA counselors. Everyone uses the same general set of questions to establish the degree to which clients fall into one of three levels as shown in Table 26.
|
TABLE 26 Levels used by Employment and Income Assistance and Taking Charge! |
||
| Level | Attributes | Typical intervention |
| 1 (job-ready) | Recent work experience and at least one of:
|
job placement, on-the job training, etc. |
| 2 (training ready) | Readiness to enter training, with no health, family
or behavioral barriers and one of following:
|
pre-work orientation, occupational skills training, community internships |
| 3 (multi-barriered) | One or more of the following:
|
literacy, ESL training, community programs to impart life and coping skills, development of basic work habits. |
Leveling is common throughout the health and social services system. "Triage" in emergency health care assigns patients to three levels based on the immediacy of care needed. In the home care system, leveling is used to first determine eligibility for home care and then to assign clients to resources. In long-term care, clients are placed into four (soon to be nine) levels indicative of their ability to care for themselves, the level of nursing care required and their potential for harming themselves and others. The leveling process also becomes a mechanism for assigning resources.
Each of these leveling systems is based on questionnaires, similar to those used by EIA counselors and Taking Charge! employment facilitators.21 The triage used in emergency health, home care, and long-term care is based on question sets that have been subjected to careful reliability and validity testing. A core set of questions used in both the home care and long-term care questionnaires refers to activities of daily living (ADL). Several ADL measures exist in the literature, and others are being refined through a systematic process of comparing outcomes with the ratings obtained in the questionnaires. The process of comparing outcomes to the levels assigned to clients is fundamental to the integrity of the leveling process. Without this testing, leveling functions neither assign clients to appropriate care, nor allocate resources based on need.
The leveling process used by the EIA counselors and Taking Charge! has not been applied with consistency. The fact that levels are incomplete in the Taking Charge! database suggests that they are not seen as crucial planning information. In fact, based on interviews with staff in Family Services and Taking Charge!, although different counselors and facilitators may complete the same form, no assurance exists that this produces consistent results. Although the assignment of levels is partly the result of past education and employment, intangibles such as personal/family problems and "motivation" are factored into a client's level. Clearly, different counselors may rate the same client differently.
Another important feature of the employment assessment process is that it should be dynamic, updated as clients proceed through various employment enhancement measures. Any updating that is done presupposes that assessments are completed for everyone.
Assessment of the reliability and validity of the employability assessment and leveling process has not been done. Although EIA and Taking Charge! personnel complete similar forms, they are not identical. Coupled with the fact that many people complete the assessment on clients, a high likelihood exists that assessments and leveling vary among counselors. Most respondents we interviewed in government confirmed that this was probable. Further, no assessment appears to have been done of the basic set of questions used to predict whether a client will be successful in a course of training. This is not a criticism of Taking Charge!. Rather, it is a statement about the fragility of the general process for evaluating income assistance clients for job readiness.
Employment facilitators emphasized that the employability assessments are not used to create levels. For most, the "level" of a client is quickly determined after a few background questions on work history and education. Facilitators reported that the detailed assessments may be used to identify potential barriers to training and employment and to develop programs. The application of the assessment to this process probably varies among counselors and facilitators.
As we show in Section 5, training providers also complete the Taking Charge! assessments. Training providers funded by Taking Charge! are expected to forward client assessments to the Taking Charge! office before accepting clients into the program. Many may complete these assessments in relation to their programs. For example, McKnight and Associates accepts only those clients who it believes will benefit from an intensive work preparation and who will be successful in securing employment. In general, a training contractor may well have different goals than an employment facilitator in assigning a level to a client. The range of people and organizations completing these assessments has the potential to substantially corrupt the employability and leveling process.
Each Taking Charge! employment facilitator must identify barriers to training and employment based on individual interpretations. EIA counselors are required to complete employment assessments as a basis for job plans for all clients assigned work expectations. Initially Taking Charge! and EIA processed assessments independently, but they now share employability assessments.
A need exists for a reduced-form employability assessment, tested for reliability (controlling inter rater variation) and validity (having the ability to predict outcomes). The province is addressing these issues. All counselors have been trained in completing employability assessments. SAMIN will build a history of each client who is assessed. With the advent of Tier 1, all provincial clients will be assessed by the end of 1998. These are important changes that will improve the province's capacity to evaluate training.
Taking Charge! began coincident with a major welfare reform initiative called Making Welfare Work. Although it is not possible to completely disentangle the effects of these two initiatives on the employment of income assistance clients, an operational issue for Taking Charge! is how Making Welfare Work has affected its clients. Also important is the role of Taking Charge! in welfare reform.
Key informants made the following points:
Many staff believe that a key element of their success is that Taking Charge! is both apart and seen to be apart from government. This is evident in the Program's focus on being an independent entity and presenting a friendly and welcoming face to income assistance clients. This is distinct from the bureaucratic approach of government, especially the approach of those departments whose function is to verify that clients are eligible to continue receiving income assistance.
The outcome of this view may be seen in two ways:
Although Taking Charge! would never counsel a client to conceal employment income, staff and volunteers would likely not think it their responsibility to seek out fraud. To create an agency that is empathetic with the client simply means that it is hard to draw it into the service stream of income assistance. Its success with clients depends on its being seen as an alternative program where clients need not be concerned about being investigated.
Another dimension is that aligning services offered by Taking Charge! with other public sector training programs requires negotiation rather than decree. Taking Charge! management and staff are accountable to the Board and not the Minister. Along with the benefits of being an independent agency comes the reduced connection to other services funded by government.
It is important to stress that Taking Charge! Board and management take pains to coordinate services and discuss options with other departments. As we see in Section 5, this degree of cooperation has only emerged late in the pilot period.
The concept of follow-up is important. From the perspective of government and this evaluation, follow-up is easily interpreted as a process of verifying the outcome of training. From the perspective of Taking Charge!, it can mean that, but it usually means a process of re-contacting clients to determine whether further interventions are needed or desirable. This is part of the welcoming approach that Taking Charge! wishes to advance to clients.
Taking Charge! faces a constraint in adopting an investigative approach to client follow-up. If clients perceive that follow-up is part of the investigative processes used by Family Services, Taking Charge! may lose important advantages in attracting clients. Accordingly, follow-up within Taking Charge! must present a benign face to its clients.
Strengths and weakness of Taking Charge!'s design and delivery model are as follows:
The presence of training service providers on the Board is unusual. At its inception, good reason may have existed to include training service providers on the Board to ensure this expertise was available to management. We were informed that legal counsel reviewed and accepted the conflict of interest procedures developed by the Board to deal with this specific issue. Therefore, Taking Charge! has fulfilled the legal requirements to avoid conflict of interest.
Nonetheless, the presence of training service providers on a Board that awards substantial contracts to the industry (including organizations that employ the Board members) is bound to create the appearance of a conflict of interest. The organization is small and Board members may have regular contact with staff who evaluate training service proposals. Further, a Board member who is a training service provider will have access to the proposals of other providers who are their competitors.
Taking Charge! could consider alternatives, such as:
|
Taking Charge! was intended to serve as a facilitator or catalyst for a community response to chronic unemployment. The target group, single parents (mostly women) on income assistance, faces critical barriers to employment, by virtue of child care responsibilities and low levels of education and work experience.
Partnerships in Taking Charge! and other Strategic Initiatives in Canada are modeled after the Job Training Partnership Act in the United States. The key idea is that a partnership approach is most effective in creating the training and employment opportunities needed by this target clientele. The idea of partnership also involves that of "leverage." The assets of a single organization can "lever" additional resources from partner organizations, leading to enhanced impacts for clients.
One key informant observed that the idea of "shared risk" is inherent in the notion of partnership. Partnerships create ventures that have a better chance of success because risk is spread across several partners. This idea is useful in assessing the nature and success of the partnerships created by Taking Charge!.
Several partnerships are crucial to the success of Taking Charge!, including:
The fact that we had no private employers to interview in the formative evaluation shows that Taking Charge! had not yet created this partnership. A central recommendation of that report was that Taking Charge! needs to create a business advisory council. Work on this is reportedly continuing. Management has identified a preliminary list of potential members, and a presentation to the Chamber of Commerce has been planned.23
Several larger employers (e.g., Standard Aero) are reportedly also involved in designing training and placement programs. In the past Taking Charge! has created similar ventures with the garment sector and home care providers. These projects have all worked, to a degree, but it is accurate to state that they have yet to become a consistent way to create jobs for the clients.
The most important function for employers is to hire the graduates placed by the program. Few are involved with Taking Charge! before being approached to accept training graduates. Also, many trainees are placed by the training service provider. We are unable to estimate the exact number placed by Taking Charge!, training service providers, or who found jobs on their own. It is likely that many employers would be aware of some of their recent hires who were income assistance clients who recently completed a course, but they would have had contact with the training service provider and not with Taking Charge!.
Aside from a few pilot programs such as with We Care (a home care service provider) or TanJay (garment manufacturer), few employers define a training program with Taking Charge! to meet their future needs. Most respond to enquiries from the Taking Jobs! component of Taking Charge! or the marketing by a training service provider to fill positions in their companies.
Table 27 shows the number of trainees in work placement by the largest employers who have engaged a client of Taking Charge! or its service providers. Based on the Taking Charge! database, it seems that no employer has hired more than nine trainees in the work placement program. Most have engaged only one trainee. Note that this table presents the work placement program. We have no way of tracing where Taking Charge! graduates end up working. Taking Charge! placed eight of its clients in their own volunteer program. The ISM placements for Taking Charge! are the result of a University of Manitoba Network Administrator course.
|
TABLE 27 Work placement activities |
|||
| Employer | Industry | Program | Number of trainees placed |
| Telespectrum | Telemarketing | Service Provider | 9 |
| Faneuil ISG Inc. | Telemarketing | Service Provider | 8 |
| Taking Charge! | Volunteer experience | Taking Jobs! (Taking Charge!) |
8 |
| Canada Safeway | Retail | Service Provider | 6 |
| The Bay | Retail | Service Provider | 6 |
| Olsten Health Services | Home care | Service Provider | 5 |
| Gateway Industries | Publishing | Service Provider | 5 |
| Complete Care | Home care | Service Provider | 4 |
| AT&T | Telemarketing | Service Provider | 4 |
| Allied International | Credit/financial | Service Provider | 4 |
| ISM | Software | Taking Jobs! (Taking Charge!) |
3 |
| Source: CAMS database. | |||
Table 28 shows that training service providers and Taking Charge! placed 563 clients with 417 employers. Training service providers placed 86 percent of the clients (486/563).
|
TABLE 28 Work placements by employer (largest number of placements) |
|||
| Number of trainees placed (A) | Service providers (B) | Taking Charge! (C) | All clients (D*) |
| 1 | 280 | 52 | 332 |
| 2 | 50 | 7 | 57 |
| 3 | 16 | 1 | 17 |
| 4 | 3 | - | 3 |
| 5 | 2 | - | 2 |
| 6 | 2 | - | 2 |
| 7 | 1 | - | 1 |
| 8 | 1 | 1 | 2 |
| 9 | 1 | - | 1 |
| Total placements | 486 | 77 | 563 |
| Total employers | 356 | 61 | 417 |
| *This is the sum of Column B and Column C. | |||
In summary, employers have a passive relationship with Taking charge! They respond when a training service provider or Taking Charge! approaches them with an offer for a work placement. Little analysis appears to have been completed by management on the outcomes of these work placements, and we do not know whether they have translated into permanent jobs.24
The delay in creating close contacts with the business community is a critical gap in the programming options for Taking Charge!. In our interviews with management and staff, some key informants had unrealistic views about what business could and should do in hiring Taking Charge! trainees. Three or four remarked that businesses need to recognize their responsibility and "hire 50 graduates of our programs." Overall, Taking Charge! management is keenly aware of the need to create viable contacts with the business community.
The most effective way to increase contact with industry is to make strategic business appointments to the Board. Besides opening direct lines to industry, increased business representation on the Board would offer Taking Charge! important perspectives and impart additional private-sector values to the organization. These perspectives would complement the social service background that many staff members bring to the organization. In part, the failure of this form of partnership may result from the fact that many on the staff have little experience with the business community, have few contacts, and may be intimidated to take a proactive approach.
The delay in creating close contacts with the business community cannot be assigned to the Board or management of Taking Charge!. Since appointment to the Board is the prerogative of the federal and provincial governments, specifically the responsible Ministers, it is at this level that this lack may be remedied. The creation of a business advisory panel is within the purview of the Board and management, but assistance by senior members of the two responsible provincial departments would be helpful.
Training service providers have become the "agents" of Taking Charge! and recruit, assess, train, and report on outcomes for many clients. Employment facilitators and Taking Charge! management told us that without service providers doing this, Taking Charge! could never meet the demand for training and services. Table 29 shows how employability assessments and training plans are created for income assistance clients in Taking Charge!.
Recall that one of the original objectives was for Taking Charge! to assess and prepare an individual job plan for each client. From Table 18, we find that 2,058 clients have been assessed and 1,462 have participated in training programs offered by service providers. This and Table 29 suggest that 146 have received an assessment and plan from assessed by neither Taking Charge! nor the service provider (presumably an E&IA counselor); and for 541; the service provider completed the assessment and plan.
|
TABLE 29 Employability and training plans of Taking Charge! clients — fiscal year 1997-98 |
||||||
| Employment assessment and job training plans | ||||||
| Taking Charge! does both (A) | Taking Charge! does Assessment (B) | Neither Taking Charge! nor Service Provider (C) | Service providers do both (D) | Other (E) | Total | |
| Number of clients | 102 | 192 | 312 | 370 | 24 | 1,000 |
| Percent | 10% | 19% | 31% | 37% | 2% | 100% |
| Source: Survey of
Training Service Providers and the Taking Charge! file review of training
service provider contracts.
See Appendix 3. Note: We cumulated the clients for each respondent in 97/98. We then allocated a respondent (service provider) client to the five possible outcomes in Table 29 and totaled across all programs. For example, service provider A may have trained 100 clients in 97/98 and stated that 20 percent of these clients were assessed by Taking Charge! and 80 percent by themselves. Therefore, this organization contributes 20 clients to column A and 80 clients to column D. The percentages in Table 29 do not equal the percentages in Table 4, Appendix 3. |
||||||
The original intent that Taking Charge! have initial client contact, perform an assessment and then create an individualized job plan for all clients is not happening. The goal to generate high employment numbers required Taking Charge! to "out-source" the assessment and planning process to external contractors. Between 25 percent and 30 percent of all Taking Charge! clients have neither an assessment nor a plan.
The training "industry" in Manitoba comprises both for-profit human resource firms and non-profit organizations that may or may not have a record of training income assistance recipients. Many of these training organizations have had long histories of funding from federal and provincial governments - others recently emerged in response to the funding available from Taking Charge!. None of this is especially problematic or unusual. Taking Charge! inherited a well developed and well-connected group of training providers. The fact that Taking Charge! did not have a developed training function meant that these external contractors became and remain the primary vehicles for successful labour market outcomes for this program. In essence, Taking Charge! has become a funder for training services directed to single parents on income assistance, as opposed to offering training services in-house.
No question exists about the importance of third-party training service providers. Without these contracts, Taking Charge! could never have delivered training and education services to its clients. Essentially, to meet its target for serving clients and placing trainees into jobs, Taking Charge! had to contract this service. Further, it has contracted recruiting, employability assessment, job planning, and follow-up to the point where many Taking Charge! clients receiving assessments have these services performed outside the program.
It is very hard to offer a firm estimate of the number of Taking Charge! clients who are assessed by training service providers. Taking Charge! believes the number is only 24 percent, while information from service providers indicate that it is 37 percent. The fact that some contracts now explicitly include the requirement that service providers assess and create plans demonstrates that Taking Charge! recognizes that it cannot maintain the number of clients and complete assessments.
Training service providers are also largely responsible for defining the overall direction of training within the program. Until very recently, training service providers would propose programs they believed to be valuable. Initially, Taking Charge! accepted most of the proposals provided they involved training the client groups. The Board still would request changes, but by and large most proposals that met an apparent need received funding.
Lately, Taking Charge! management and staff have subjected proposals to closer scrutiny. Proponents are expected to outline what client needs will be met and how success will be achieved. The proposal review process has tightened. Employment facilitators now have previous performance to assess a contractor's likely performance. Also, proposals are sometimes (but not always) assessed against occupational and labour market projections. Finally, certain types of courses, such as basic word processing or basic computer skills, are not being funded, partly because the in-house computer lab can cover this and partly because clients often get these skills as part of a basic educational upgrading.
What Taking Charge! has yet to do is to take a more proactive role in defining the training agenda. Several key informants, including Taking Charge! staff and management, stated that Taking Charge! should determine client needs and call for tenders for the programming needed.
Some respondents expressed ambivalence about the relationship between Taking Charge! and the service providers. On the one hand, the excellence and professionalism of some trainers was widely acknowledged. These trainers, who are well known within Taking Charge!, Education and Training, and Family Services, are seen as offering the taxpayer excellent value for money. Several key informants suggested that contracts could be let directly to these organizations.
Also, some key informants questioned whether decisions made on funding or renewal were always at arms length. By having community organizations serving directly on the Board and acting as service providers, Taking Charge! has the benefit of obtaining high levels of access to its target community. However, the appearance of favouritism may exist when Taking Charge! awards service contracts to organizations that have representation on the Board. It is important to note that the vast majority of funds allocated by Taking Charge! are to organizations with no direct representation on its Board. Also, the evidence suggests that Taking Charge! has received value for money from the contracts awarded to organizations with Board representation.25
Community and non-profit organizations offer valuable insights and experience into the client group and may have programs that are immediately applicable to clients. It makes sense to develop close relationships with community organizations to accelerate the training opportunities open to Taking Charge! clients. This relationship with the community may be expressed through Board membership, informal contacts with community groups, joint ventures, and joint funding of training and other support activities.
At the outset, Taking Charge! encountered a cool reception from the non-profit and community-based organizations. Its creation represented a perturbation in the established funding relationships. Many existing "players," offering third-party training and service to the target group, saw the new agency as a rival for funding. Some groups quickly adapted and realized that Taking Charge! would become an important source of revenue. Several non-profit groups with no history of training also created programs to serve this client group and received funding from Taking Charge!26
In our interviews with community organizations we discovered the following:
As mentioned in several places earlier in this report, Taking Charge! has had uneven relationships with provincial departments and other programs. In part, Taking Charge! was positioned as a unique portfolio of interventions and initial efforts sought to ensure that it operated as an independent agency. The joint federal-provincial Task Group in 1994 developed a detailed program model as part of the planning for Taking Charge!. The Taking Charge! Board elected to adopt a separate approach and used the initial year to refine the structure and operation of the Program. This reflects a decision by the Board to make Taking Charge! clearly "not government" and thereby encourage income assistance clients to participate.
Taking Charge! funds a range of programs offered by non-profit groups that also offer services to the provincial government. The process of defining roles and responsibilities for creating and managing employment enhancement measures has been complex and contributed to an initially difficult relationship between Taking Charge! and various provincial programs.
Fortunately this relationship is improving. With the co-location of an Employment and Income Assistance office within the Taking Charge! facility, clients now have immediate access to Department of Family Services counselors. Taking Charge! management and personnel from the Departments of Family Services and Education and Training are starting to meet to coordinate services.
A key lesson is that new initiatives such as Taking Charge! take time to define, especially if configured as a non-profit agency with an independent Board. While this approach fits with the ethos of using non governmental delivery mechanisms, it also requires additional attention to synchronize the governmental and non-governmental systems. Another key lesson is that government needs to take a more active hand in the planning and delivery of services. Taking Charge! has occurred simultaneously with Making Welfare Work, the creation of a single tier of income assistance,27 and the devolution of training and labour market services from the federal to the provincial governments. Taking Charge! could have used increased connection with government to help coordinate training services.
At the outset of the program, Taking Charge! had a close relationship with the federal government at the senior departmental and political level. The first Director of Operations for Taking Charge! was seconded from Human Resources and Development Canada. Further, as signatory to the agreement, the federal government made appointments to the Board.
The federal government continues to attend meetings and participate in the planning and evaluation process, but interactions with the provincial government are much more important to the operations of Taking Charge!.
Job training programs have the potential to displace regular workers. This occurs because the recently trained income assistance clients may be offered to private employers with a wage subsidy. Further, these trainees may continue to receive various benefits and can afford to work for lower wages.28 The evidence on job displacement remains largely anecdotal, since few employers would respond affirmatively that they had laid off regular employees. In Manitoba, the recent growth in the economy makes job displacement from provincial training programs unlikely.
According to the Follow-up Survey, Taking Charge! places about 37 percent of its clients into full-time employment and 17 percent into part-time employment (8 percent full-time equivalency) for an effective employment rate of 45 percent. Employment Connections places 44 percent of clients in full-time and 20 percent in part-time work for an effective employment rate of 54 percent. These programs have about the same number of clients each year (950); therefore, we expect that in the last fiscal year, Taking Charge! added 428 new employees and Employment Connections added 513 new workers to the Manitoba economy. These are high-side estimates, since some of these workers may be working full-time but in transitory occupations or on short-term contracts.
Over 1995-1998 (since June) the Manitoba economy added 22,000 new jobs to the economy and had the second lowest unemployment rate in Canada (5.7 percent behind Alberta's 5.5 percent).29 In the last year (since June 1997), Manitoba has added 6,000 new jobs.
In this economic context, it is unlikely that the activities of job training programs have displaced regular workers. More than likely these new workers have been welcome. Displacement in the United States occurs in certain urban and regional economies such as New York City. It is most unlikely to occur in upper midwestern states where unemployment is currently running at well under 3 percent and is measured at 0 percent in some centres. The issue in these areas is not displacement, but whether government needs to spend any money on job training since employers are more than willing to train to get workers.
We had anticipated that the employer survey would offer some insight into deployment of graduates of the programs. We were unable to give employers graduate identities as part of the questionnaire.30 A generic question that simply asked whether new hires had been used to replace regular workers seemed unviable and pointless without being able to identify previous income assistance clients in the employer's workforce.
Accordingly, we conclude that job displacement in Winnipeg as a result of the Taking Charge! program is unlikely.
Considerable variation exists in the quality of partnerships that Taking Charge! has with the business community, community organizations, and government. Most important is that Taking Charge! has not forged links with the business community. Considering the evidence from the evaluations of the Job-Training Partnership Act in the United States that show the importance of partnerships with the private sector, this lack is the most important partnership deficiency of Taking Charge!. Recent efforts to increase the involvement of business are promising and need to have the highest priority for the organization.
By contrast, Taking Charge! has very close relationships with community organizations, especially those that are training providers. Largely this relationship exists because Taking Charge! is a source of funding for these organizations. However, Taking Charge! does not have close relationships with community organizations that are not under contract. In this sense the networking circle for the organization is restricted. This limits the program in joint venturing with other organizations and developing a fuller range of partnerships that may benefit clients.
Relationships with provincial government are improving after an initial period of poor cooperation. Cooperation and joint planning are increasing and this is drawing the program closer to the job training and assistance system in Manitoba.
|
This section presents econometric estimates of program impact on individual outcomes. It starts with a description of the variables available from the analysis and how we obtained them, and then discusses how to measure self-sufficiency and what indicators are available. Increased economic independence is associated with increased hours of employment, higher earnings, an increase in benefit-reductions (reduction in income assistance paid because of earnings)31 , and a reduction in assistance payments.
Next, we examine the Comparison Group, and the Control Group32 for differences in earnings, hours worked, and quality of work. Quite precise estimates can be obtained of the impact of interventions on hours worked and income.
The next issue addressed is the change in use of the income assistance program. Labour market interventions directed to income assistance clients are intended not only to increase their level of economic independence, but also to reduce the costs to government. Less involvement with the income assistance program is a direct measure of that cost reduction.
We then examine the impacts of the programs on clients' home and personal lives using the follow-up survey data as well as five case studies.
We use three main data sources to measure program impact at the individual level:
The Control Group (SAMIN)
The Social Allowances Management Information Network (SAMIN) is maintained by the Department of Family Services. The main purpose of SAMIN is to track payments to provincial income assistance clients. This database also tracks payments made to support education (special needs), employment earnings for those clients on partial assistance and several basic demographic variables. Because it is primarily a payment tracking system, it provides reliable measures of an individual's participation in the income assistance program. Most useful is that SAMIN allows us to create a history of a client's involvement with the income assistance system.
We obtained extracts from the SAMIN database at several points, and most recently in July 1998. This allowed us to track income assistance clients for a minimum of nine months after the last completed intervention (September 1997). Besides serving as the Control Group, the SAMIN database also provided key variables (duration of income assistance, benefits paid, income assistance history, etc.) for participants in the Comparison and Treatment Groups.35
Treatment Group (Taking Charge!)
Taking Charge! created a client management information system (CAMS) to capture client information, describe the intervention, and track post-intervention client progress (specifically the nature of employment and/or training). We reviewed the CAMS system in detail in the Phase 1 evaluation and summarized important concerns in Section 4.
Comparison Group
As described in Section 2, Manitoba offers several training programs directed to income assistance recipients. Of the four we studied, Employment Connections and Opportunities for Employment are focused interventions in the sense that they design and deliver the training. Community Partnerships and Pathways to Success constitute a range of interventions offered by different agencies. These latter programs represent funding to community-based organizations that have identified specific needs and client groups. Pathways to Success is directed to Aboriginal clients.
Because of the variability and inconsistency with which client demographic data are maintained for all programs and because follow-up by programs is often incomplete, we used only the "intervention" information from Taking Charge!, Employment Connections, Community Partnerships, and Pathways to Success (e.g., program start and end dates, type of intervention, etc.).
We obtained type and duration of intervention from each program.36 By matching the social insurance and income assistance numbers of clients with SAMIN, we created complete records of income assistance involvement and a partial picture of post-intervention employment.
We treated these Comparison Group Programs as a whole. Sample sizes for all programs except Employment Connections are small and preclude a separate analysis of these interventions. We are able to isolate Employment Connections in some analysis, allowing us to directly compare Taking Charge! with this intervention.
Key client attributes (Education, Status Indian, visible minority) are maintained differently by the various income assistance training programs. We elected to collect information from the job plans completed by Employment and Income Assistance counselors. Taking Charge! graduates coded this information for all 2,764 clients in the database that we created for this research. One important qualification is that these data are derived from information that E&IA counselors completed at the time of application. For many clients, some years may have elapsed before acceptances into an intervention, leaving the information on education dated.
Follow-up survey of clients
One concern for tracking individual outcomes is the quality of post-intervention data. Our initial review of the Treatment and Comparison Group information revealed that these programs did not track outcomes consistently. The SAMIN database reliably measures the income assistance payments made and whether a client is on or off welfare. It is less reliable in measuring employment outcomes, especially for those who find work and leave assistance entirely.37
The follow-up survey of clients generated some outcome data such as hours worked and incomes earned from employment at six and nine months after the intervention.38 It also offered insights into personal and home lives of participants (reported in Section 6.5) plus their satisfaction with the program (reported in Section 7.1). However, we elected to use the SAMIN information throughout to maintain a large sample (2,764 versus 1,001).
Five case studies
To offer qualitative insight into how Taking Charge! affected participants, we selected five respondents to serve as case studies. We identified five important client groups, then randomly selected one respondent in each of these groups from the Taking Charge! database. The case studies are reported both in Appendix 7, Volume II and Section 6.7.
A central goal of all income assistance training programs is to raise client self-sufficiency and thereby reduce income assistance usage. Self-sufficiency simply means the desire and ability of an adult to earn a living. For income assistance programs it means the ability to be economically independent of government support. We measured changes in self-sufficiency in two ways:
Second, we asked respondents in the follow-up survey whether they had worked more after the program. We also asked them whether they thought the program had improved their chances of gaining employment and becoming economically independent. Taking Charge! clients reported that:
Based solely on client perceptions, Taking Charge! has had a strong positive impact on client self-confidence and self-sufficiency.39
A main objective of Taking Charge! is to move income assistance clients toward employment. Two questions need to be answered:
The quality and types of jobs held by Taking Charge! and Comparison Group clients are not well recorded in the databases available for this research. The follow-up survey and CAMS support inferences about what kind of work clients are getting (see Appendix 1, Vol. II). Post-intervention wages typically run at about $1,000 per month. Taking Charge! has some follow-up information in its database, but it is highly variable. Information for other programs and the Department of Family Services either did not exist or was unavailable.
A difference-in-difference methodology is the foundation for the analysis with the following dependent variables shown in Table 30.40 The source for these variables is the SAMIN database.
|
TABLE 30 Dependent variables in linear regression models |
||
| Variable name | Description | Commentary |
| DIFFWAGE | Difference of monthly average wages between pre- and post-intervention while on income assistance. | Program success leading to economic independence is associated with higher wages. |
| DIFFHOUR | Difference of average hours worked per month between pre- and post-intervention while on income assistance. | Program success leading to economic independence is associated with higher hours worked. |
Regression modeling allows us to study the independent effects of a range of influences on pre- and post-intervention hours and wages. For example, we can isolate the effects of the Treatment (Taking Charge!) and the Comparison Groups. We can also measure the effect of client attributes (e.g., Aboriginal status or the number of children less than six years of age) at the start of intervention plus the effects of different types of intervention. The independent variables we used are shown in Table 31.
|
TABLE 31 Independent variables in the general linear regression models |
||
| Variable name | Description | Commentary/data source |
| GROUP1 | ||
| Taking Charge! vs.Others (Taking Charge!=1, Others=0) | By definition when Group1 =Group2 = 0 we have defined the Control Group. | |
| GROUP2 | Comparison vs Others (Comparison=1, Others=0) | |
| EDUC2 | Education in years | From EJTTP |
| DSTATIND | Registered Indian status (Yes=1, No=0) | From SAMIN |
| GENDER | Female=0, Male=1 | From SAMIN |
| DKIDS6PR | Number of children less than six years of age pre-intervention | From SAMIN |
| DKID6-18PR | Number of children between the ages of 6 and 18 years of age pre-intervention | From SAMIN |
| DIFFKID6 | Difference of children less than six years of age between pre- and post-intervention | From SAMIN |
| DAGE | Age at intervention start | From Program data |
| DWRKXPPR | Work expectation pre last intervention (Yes=1, No=0) | From SAMIN. Indicates whether a work expectation been assigned. |
| DJOBPRSA | Number jobs prior to entering income assistance | From SAMIN |
| DURINTVN | Number days on intervention | From Program data |
| DIFFUINC | Difference of unearned income between pre- and post-intervention | From SAMIN. Unearned income is other government transfers and maintenance payments. (Lower unearned income is generally associated with higher self-sufficiency.) |
| Total | Total exposure to income assistance = (Total # months on SAMIN (AGE*12) - 216) | Total time spent on SA between June 1998 and the person's 18th birth date. Those with less history on assistance often have an easier time in gaining work. |
Most of the independent variables are self-explanatory. We created some difference variables such as DIFFUINC to evaluate the impact of government transfers (child supports, pensions, etc.) on hours worked and wages.
One important group of variables is the type of interventions. We classified the interventions from each program into one of seven categories,41 as shown in Table 32. We coded only one intervention for each participant.42 We also coded interaction variables in an attempt to compare different interventions across programs, but because some programs only had a single intervention, this resulted in perfect collinearity. Therefore, we report only results for the type of intervention.43
|
TABLE 32 Independent variables related to interventions |
||
| Variable name | Description | Source |
| DINT1 | Work, job entry/job search | Program |
| DINT2 | Trade skills | Program |
| DINT3 | Technology/management skills | Program |
| DINT4 | Literacy/numeracy | Program |
| DINT5 | Academic upgrading | Program |
| DINT6 | Job placement/work experience | Program |
| DNUMINT | Total number of interventions per person | From Program |
| Note: DINT1 - 6 are dummy variables coded 1 if intervention taken and 0 otherwise. A seventh intervention ("Other") becomes part of the Constant term in the regression. We assigned only one intervention per client and used DNUMINT to capture clients with more than one intervention. | ||
Table 33 shows the results for DIFFWAGE and DIFFHOURS. We have rounded the regression coefficients. Bolded entries denote coefficients that are statistically significant at either the 0.1 (single asterix *) or 0.05 level (double asterix **). The sample size of 2,526 is lower than the total sample (2,764), as 238 individuals did not have both pre- and post-intervention income assistance records.
|
TABLE 33 Difference-in-difference regression: average monthly wages (DIFFWAGE) and average hours (DIFFHOURS), pre and post-intervention |
||
| DIFFWAGE | DIFFHOURS | |
| Sample size | 2,526 | 2,526 |
| Constant | 19 | 6 |
| Group 1 (Taking Charge!) | 77 ** | 10 ** |
| Group 2 (Comparison group) | 89 ** | 13 ** |
| Educ2 | 2 | 0 |
| Gender | -18 | -4 |
| Age | -1 | 0 |
| # Children 6-18 years of age | 27 ** | 3 ** |
| # Children < 6 years of age | 0 | -1 |
| Status Indian | -44 ** | -5 ** |
| Number of interventions | -8 | -2 |
| Intervention 1 (work/job search) | 34 | 7 ** |
| Intervention 2 (trade skills) | 39 * | 2 |
| Intervention 3 (tech/mgmt skills) | -48 * | -6 |
| Intervention 4 (lit./num) | -17 | -1 |
| Intervention 5 (academic upgrading) | -91 | -13 |
| Intervention 6 (job placement) | 55 ** | 8 ** |
| Work expectations | 67 ** | 9 ** |
| Number of jobs before intervention | 1 * | 0 |
| Total time on SA | 17 | 2 |
| Difference in unearned income | 0 ** | 0 |
| Difference in children <6 years of age (pre-post) | -19 | -3 |
| Duration of intervention | 0 | 0 |
| Adjusted R2 | 0.0915 | 0.087 |
|
Entries that are bold are statistically significant at the 0.05 level or lower (**) or the 0.10 Level (*). The other entries should be treated as equal to 0. |
||
Table 33 shows that both the Treatment (Taking Charge!) and the Comparison Groups had statistically significant impacts on average monthly wages. What we are measuring is the increase in wages earned while still on assistance. Based on the regression model, Taking Charge! clients earned an additional $114 per month while clients in the Comparison Group enjoyed an increase of $126.44 Those in the Control Group had a wage increase of $37 after intervention.45 These increases indicate that both the Treatment and Comparison programs have an impact on wages, with the Comparison Group having a slightly more positive effect.
Taking Charge! has the autonomous effect of increasing wages by $77 per month upon intervention completion, while participation in one of the Comparison Groups increased wages by $89. This occurs when all other statistically significant variables are considered equal to zero.
Education and gender have no impact on wages. Because the vast majority of single-parent clients on income assistance are women (95 percent), we are unable to measure whether being a male or female participant in these programs makes a difference. Neither is education statistically significant. Throughout the analysis, education does not register as having a significant impact on economic independence (when tested at 0.05 significance). This flies in the face of both theory and other studies. We suspect that education has not been well-tracked by programs or the income assistance system in general.46
Having children between 6 and 18 years of age is a positive and significant help to increased incomes ($27) but having children less than six is not a significant hindrance. Being a Status Indian reduces post-intervention incomes by $44 (see Table 34 for details).
Of the interventions, only job placement had a significant impact (at the 5 percent level), increasing wages by $55. The results probably reflect the quick success of job-ready clients who have the education and training to get work with job oriented training.
The assignment of a work expectation was also shown to have a positive effect on increasing wages ($67). More post-intervention time is needed to fully observe the effect of work expectation assignments, as they have only recently been introduced to the system.47
Hypothetical Case Studies
Table 34 shows three hypothetical cases and how this regression predicts differences in pre- and post-intervention monthly income. The regression allows us to illustrate how certain client characteristics and interventions interact to produce a change in monthly income after an intervention.
|
TABLE 34 Illustrative (hypothetical) cases from the difference-in-difference in pre- and post-intervention monthly wages |
||||||
| Client attribute | Case 1 | Case 2 | Case 3 | |||
| Participant in Taking Charge! | YES | $77 | NO | $0 | YES | $77 |
| Participant in Comparison group | NO | $0 | YES | $89 | NO | $0 |
| Number of children 6 - 18 | 1 | $27 | 0 | $0 | 2 | $54 |
| Status Indian | NO | $0 | YES | ($44) | YES | ($44) |
| Intervention 6 (job placement) | YES | $55 | NO | $0 | YES | $55 |
| Work expectation | YES | $67 | YES | $67 | NO | $0 |
| Total effect (Change in monthly income) | $226 | $112 | $142 | |||
In Case 1, the client has participated in Taking Charge! ($77), has one child between the ages of 6 and 18 ($27), is not a Status Indian, has participated in Intervention 6 (Job Placement) ($55), and has had a work expectation assigned ($67). This individual could expect an increase in monthly income of $226 after participating in the program.
Case 2 is a client in the Comparison Group ($89), has no children between the ages of 6 and 18 (no benefit), is a Status Indian (-$44), has not participated in Intervention 6 (no benefit), and has also had a work expectation assigned ($67). Post-intervention wages are $112.
Case 3 shows a Status Indian (-$44) with two children between the ages of 6 and 18 ($54), who has completed Intervention 6 ($55) with Taking Charge! ($77). This individual could be expected to gain $142 in monthly income from this combination of personal situations and participation in Taking Charge!.
These cases show the barriers facing Status Indians, validating the need for programming directed to Aboriginal populations.
Taking Charge! has a statistically significant impact on the hours worked post-intervention (See Table 33). On average, Taking Charge! participants work 16 hours more each month following completion of their intervention, while Comparison Group participants work 19 more hours.48 Control Group participants worked 6 hours more a month.
Participating in Taking Charge! results in an additional 10 hours per month while the Comparison Group's change in hours is 13 (holding all other variables constant).
As for the other independent variables, the results are broadly similar to those obtained for differences in wages. Education, age, and gender have no impact on post-intervention hours worked. The number of children between the ages of 6 and 18 has a small positive impact (three hours for each child), while being a Status Indian has a negative effect (-5 hours).
One important finding is that only two interventions are significant. Participation in job search/job entry (Intervention 1) or a job placement program (Intervention 6) increases post-intervention hours by 7 and 8 hours, respectively. The imposition of work expectations also increases post-intervention hours by 9 hours on average.
The results from Table 33 show that both Taking Charge! and the Comparison Group interventions (predominately Employment Connections) have had a positive impact on client earnings and the number of hours worked after the intervention. Comparison Group clients experience a slight increase in income and hours worked over Taking Charge! clients.
Only interventions that stress job placement have a significant effect on both wages and hours worked. Since we are only tracking outcomes after a short time (less than 24 months at the maximum and nine months at the minimum), the analysis is biased toward these interventions. The results also show the significant employment barriers faced by Status Indians.
Interpretation of the results requires care. The analysis shows that job placement and job-search programs are most effective. As we have shown elsewhere, evidence exists to confirm the claim that the job readiness of income assistance clients is declining. Further, the recent recovery of the provincial economy is also responsible for the success of job-ready clients participating in these interventions. If economic growth slows, the success of these work-readiness interventions may moderate.
This analysis does not support shifting the emphasis from basic academic upgrading to job-oriented programs. The employment-oriented programs may have worked largely because of the availability of job-ready candidates and an improved labour market. We do not have a sufficient post-intervention period to test the long-term outcomes of literacy, numeracy, and academic upgrading. Evidence from other studies confirms the importance of academics in long-term labour market success.
Interventions leading to lower income assistance payments and higher benefit reductions lead to lower overall income assistance budgets. We used the difference-in-difference model to examine how training interventions and differences in personal characteristics influenced income assistance payments and benefit reductions resulting from earnings (dependent variables shown in Table 35).49
| TABLE 35 Dependent variables for income assistance regression models |
||
| Variable name | Description | Commentary |
| DIFFBENF | Difference in average benefit paid pre- and post-intervention | Program success leading to economic independence is associated with lower benefits. |
| DIFFBNRD | Difference in the average benefit reduction between pre- and post-intervention | Benefit reduction is the reduced income assistance payable because clients earned income through employment. Successful interventions lead to higher benefit reductions and lower assistance payments. |
The regression results shown in Tables 33 and 36 support the cost analysis presented in Section 8.
| TABLE 36 Difference of average benefits paid and benefit reduction, pre- and post-intervention |
||||
| DIFFWAGE | DIFFHOURS | |||
| Sample size | 2,526 | 2,526 | ||
| Constant | -166 | ** | 8 | ** |
| Group 1 (Taking Charge!) | -168 | ** | 42 | ** |
| Group 2 (Comparison group) | -118 | ** | 45 | ** |
| Educ2 | 7 | * | 1 | |
| Gender | 91 | ** | -8 | |
| Age | 0 | - | 1 | |
| # Children 6-18 years of age | -1 | 17 | ** | |
| # Children < 6 years of age | 79 | ** | 0 | |
| Status Indian | 35 | * | -23 | ** |
| Number of interventions | 38 | ** | - | |
| Intervention 1 (work/job search) | -67 | * | 23 | * |
| Intervention 2 (trade skills) | -120 | ** | 29 | ** |
| Intervention 3 (tech/mgmt skills) | 98 | ** | -21 | |
| Intervention 4 (lit./num) | 34 | -4 | ||
| Intervention 5 (academic upgrading) | -49 | -39 | ||
| Intervention 6 (job placement) | -17 | 37 | ** | |
| Work expectations | 48 | ** | 39 | ** |
| Number of jobs before intervention | -19 | ** | 1 | |
| Total time on SA | 63 | ** | 9 | |
| Difference in unearned income | 0 | 0 | ||
| Difference in children <6 (pre-post) | 31 | -9 | ||
| Duration of intervention | -0.2 | 0 | ||
| Adjusted R2 | 0.107 | 0.089 | ||
| Note: Entries that are bold are statistically significant at the 0.05 level or lower (**) or the 0.10 Level (*). The other entries should be treated as equal to 0. |
||||
Results for income assistance paid are similar to what we obtained for monthly income and hours worked. As Table 36 illustrates, both the Treatment (Taking Charge!) and the Comparison Groups had statistically significant impacts on assistance paid. The average Taking Charge! client can be expected to reduce his/her average assistance benefits by $258 per month compared to $208 for Comparison Group participants.50 Control Group participants, who receive no intervention, lower costs to the program by approximately $90 (Appendix 6, Volume II). This is due entirely to the increase in earnings that reduces their total needs.
Participating in Taking Charge! alone will decrease benefit payments by $168, per month, while participating in one of the Comparison Group programs (Employment Connections, Opportunities for Employment, Pathways to Success, or Community Partnerships) will decrease benefits paid by $118 (holding all other variables constant).
For each child less than 6 years of age, monthly benefits increase by $79. Unlike the effect on wages and hours worked, being a Status Indian has no statistically significant effect (at 0.05 significance level) on benefits paid subsequent to an intervention.
Intervention 2 reduces income assistance by $120 per month, while Intervention 3 increased payments by $98. The total number of interventions also increases income assistance paid by an average of $38 per month.
Imposing a work expectation is associated with an increase of $48 after the intervention. This is counter-intuitive, and could reflect the diversion of clients into literacy and numeracy programs that delay job entry.
For each job before intervention, assistance payments fall by $19. Increases in the time spent on assistance also affect post-intervention benefits (+$63).51
Hypothetical Case Studies
|
TABLE 37 Illustrative (hypothetical) cases from the difference-in-difference in pre- and post-income assistance benefits paid |
||||||
| Client attribute | Case 1 | Case 2 | Case 3 | |||
| Constant | YES | ($166) | YES | ($166) | YES | ($166) |
| Participant in Taking Charge! | YES | ($168) | NO | $0 | YES | ($168) |
| Participant in Comparison group | NO | $0 | YES | ($118) | NO | $0 |
| Gender | F | $91 | F | $91 | M | $0 |
| Number of children under 6 | 0 | $0 | 2 | $158 | 1 | $79 |
| Number of interventions | 1 | $38 | 2 | $76 | 1 | $38 |
| Intervention 2 (trade skills) | YES | ($120) | YES | ($120) | NO | $0 |
| Intervention 3 (tech/mgmt skills) | NO | $0 | NO | $0 | YES | $98 |
| Work expectations | YES | $48 | NO | $0 | YES | $48 |
| Number of jobs before intervention. | 1 | ($19) | 0 | $0 | 2 | ($38) |
| Total time on SA | 0.25 | $16 | 0.1 | $6 | 0.5 | $30 |
| Total effect (change in monthly income | ($280) | ($73) | ($79) | |||
The three hypothetical cases show how client attributes and intervention interact to produce different expected changes in income assistance after the intervention. Table 37 shows a net decrease in assistance payments for Case 1 ($280), a decrease of $73 for Case 2 and a reduction of $79 per month for Case 3.
The results for benefit reduction, are similar to the changes in benefits paid, as we would expect. Admittedly a convoluted concept, the benefit reduction resulting from employment responds to many of the same influences as income assistance payments. Recall that benefit reduction measures the reduced income assistance benefits paid because clients earned some income from employment earnings. An increase in benefit reduction means a lower benefit payment. A decrease in benefit reduction means that clients worked less and the province pays more income assistance.
The average benefit reduction for Taking Charge! participants is $68 per month, compared to $71 for the Comparison Groups.52 The Control Group has an average benefit reduction of $26 per month.
Based on Table 36 the Control Group showed an increase in benefit reduction of $8, holding all other variables constant. This means that for a group of income assistance clients who have no interventions, the normal process of labour market re-entry results in their earning income and a reduced income assistance liability of $8 on average for the province.
Participating in Taking Charge! alone increases the benefit reduction by $42 per month, while participating in one of the Comparison Group programs increases benefit reduction by $45. To reiterate, a benefit reduction is a lowering of the eligibility for income assistance because the recipient is able to augment income through employment earnings. Therefore an increase in benefit reduction reflects greater economic independence on the part of the income assistance recipient.
For every child between the ages of 6 and 18 years, the benefit reduction rose by $17. If the income assistance recipient is a Status Indian, the benefit reduction falls by $23. As expected, Trade Skills and Job Placement (Interventions 2 and 6) produced positive benefit reductions of $29 and $37 respectively. Finally, having a work expectation assigned leads to a higher benefit reduction of $39 for each participant.
Taking Charge! and Comparison Group programs reduce monthly income assistance payments to clients by $258 and $208 respectively. These programs also increase the benefit reduction (i.e., reduce the liability for income assistance) by $68 and $71, above the $26 that those without intervention produce.
The job-oriented interventions (Interventions 2 and 6) offer the most impact (Section 6.3), with trade skills programming decreasing income assistance payments and increasing benefit reductions. Work experience programming also increases benefit reduction while technology/management skills leads to an increase in income assistance payments.
A problem faced in this analysis is that we do not have a long follow-up period to observe client outcomes for many clients. The period of post-intervention observation ranges from about 22 months (for those who left their intervention in July 1996) to nine months for those who left Taking Charge! in September 30, 1997. We have almost 30 months of observations on the Control Group. Therefore, long-term outcomes must be inferred from a restricted set of observations.
One method for predicting the long-term outcome of training is survival analysis. This statistical procedure is commonly used in epidemiology and medicine to predict the probability of contracting a disease. The procedure can also be used to predict positive events such as getting work or leaving income assistance. Using the monthly observations that record whether a client has left income assistance, we can estimate the average time to leaving assistance or, in other words, the number of months a client remains on assistance after the intervention.
Figure 7 shows the results of a survival analysis for Taking Charge!, the Comparison Group and the Control Group. The dependent variable is the number of months on income assistance after the start of intervention. Many participants remained on assistance until the end of the observation period (June 1998), and these observations are "right censored."53
Figure 7 shows that at 12 months, about 58 percent per cent of Comparison Group clients remained on income assistance. This compares with about 75 percent for Taking Charge! and 77 percent for the Control Group. At 15 months, 40 percent of the Comparison Group remains on income assistance compared with about 60 percent for Taking Charge! and 70 percent for the Control Group. We only have 22 months of follow-up for Taking Charge! and the Comparison Group, but have been able to construct longer histories on the Control Group from SAMIN. Once again this confirms the long-term nature of training programs for these clients: single-parent income assistance clients face major barriers to economic independence that will require extended interventions. This finding is consistent with the above analysis and with what is generally known about education and training of economically disadvantaged persons.
For the Control Group, we estimate the average time to come off assistance after "intervention" at 23 months.54 For Taking Charge! this drops to 20 months, and for the Comparison Group, the average time on assistance after intervention is 16 months. These differences are easily explained by the dominance of Employment Connections clients in the Comparison Group. We estimated that Employment Connections clients have an average of 14 months on assistance after their intervention. Income assistance recipients who participate in Employment Connections are the least disadvantaged in the sample. Typically they are judged to be job-ready apart from some minor work-skills upgrading and job-search training. Therefore, this group leaves income assistance faster than the Treatment and the Control Group.
The small difference between the mean time off assistance between Taking Charge! and the Control Group is less understandable. The increase, in hours worked and earnings for Taking Charge! over the Control Group, shows that some participants in this program are finding work and earning incomes. This may be partly explained by Taking Charge! clients maintaining partial assistance and therefore remaining on the "rolls."
The percentage of time off assistance pre- and post-intervention can be assessed using a regression strategy reported as Table 6 in Appendix 6 of Volume II). First, we extracted a sub-sample of respondents from the database for whom nine months of observations existed, before and after their intervention. The difference in percentage of time each respondent had been off assistance for the 9 months before and the 9 months after her/his intervention was regressed against the standard set of independent variables used above (age, education, work experience, Aboriginal Status, etc). What this regression indicates, is that, participants in the Control Group experienced a post-intervention reduction in the percentage of time on assistance of 21 percent. For Taking Charge! this was 35 percent, and 33 percent for the Comparison Group. These results vary somewhat from those obtained by the survival analysis, but the general orders of magnitude are the same. The Treatment and Comparison Groups offer some improvement over the Control Group, but two thirds of the pre/post-intervention reduction of income assistance usage is the result of changes that would have probably occurred anyway.55
The econometric analysis tells only part of the story about program outcomes at the individual level. By asking clients directly, through sample surveys and case studies, we are able to offer an additional perspective on how the programs have affected clients.
Outcomes related to child-care, domestic conflict, and other indicators of home life are important for clients. Key informants repeatedly told us in the interviews that chaotic home and personal lives present clients with major obstacles to completing training and then securing/holding employment. Tracking changes in home and personal life was an important goal of the follow-up survey.56
We asked several questions on the Follow-up Survey to explore the quality of home and personal life for those in Taking Charge!, and the other programs.
As shown in Table 38, two important results emerge from this aspect of the follow-up survey:
Key informants confirmed the importance of increased self-confidence as an outcome of Taking Charge!. Those closely involved with the clients remarked how often they were struck by the dramatic change in attitude and demeanor displayed by clients after their involvement with the program.
|
TABLE 38 Impact on home life "What kinds of changes have you noticed in your home as a result of joining the program?" |
||||
| Type of change | Taking Charge! | Employment Connections | Opportunities for Employment | Pathways to Success |
| Moved into a better house | 5 | 1 | 0 | 0 |
| (1.8%) | (2.1%) | (%) | (%) | |
| Children are in a more stable day-care | 36 | 1 | 3 | 0 |
| (12.7%) | (2.1%) | (11.1%) | (%) | |
| More stability/less conflict in life | 43 | 10 | 5 | 0 |
| (15.2%) | (21.3%) | (18.5%) | (%) | |
| Got married/new relationship | 3 | 1 | 0 | 0 |
| (1.1%) | (2.1%) | (%) | (%) | |
| Health improved | 1 | 1 | 1 | 0 |
| (0.4%) | (2.1%) | (3.7%) | (%) | |
| Better financial and/or time management | 55 | 9 | 5 | 1 |
| (19.4%) | (19.1%) | (18.5%) | (50%) | |
| More self-confidence/self-esteem | 126 | 17 | 10 | 1 |
| (44.5%) | (36.2%) | (37%) | (50%) | |
| Better role model for the kids | 45 | 12 | 5 | 0 |
| (15.9%) | (23.5%) | (18.5%) | (%) | |
| More discipline in my life | 27 | 5 | 2 | 0 |
| (9.5%) | (10.6%) | (7.4%) | (%) | |
| Higher income | 40 | 16 | 0 | 0 |
| (14.1%) | (34.0%) | (%) | (%) | |
| (NEGATIVE)
Too much time away from kids |
20 | 3 | 0 | 0 |
| (7.1%) | (6.4%) | (%) | (%) | |
| Other negative changes | 13 | 1 | 0 | 0 |
| (4.6%) | (2.1%) | (%) | (%) | |
| Source: Follow up survey of clients. | ||||
Case studies offer a highly personal insight into the lives of clients and how interventions such as Taking Charge! operate. While this information cannot be used to infer results for the entire population of Taking Charge! clientele, case studies offer important understanding of the program and how it serves different clients.
Several themes flow through these case studies:
Clients found the training by service providers to be good and reported that this aspect of the program had been good.
Overall, Taking Charge! is revealed as a program that has made a positive impact on the lives of these five individuals. Experience with the employment facilitators varies, with some respondents reporting a high level of support and empathy, and others seriously criticizing Taking Charge! staff for not understanding their situation or remaining in close contact. It is likely that client variations in attitude and approach are as responsible for these different experiences as is Taking Charge! and its staff.
Facilitators and the individual job planning are clearly central features that contribute to client satisfaction. These are the distinguishing features of the program.
This section has established the following basic features of Taking Charge! and other labour market interventions:
|
A central issue for the evaluation is client satisfaction among income assistance clients, employers, and training service providers. We collected information from three sources:
This section provides an overview of client satisfaction among income assistance clients, employers, and training service providers.
In February 1998, we interviewed 1,001 income assistance clients. Of these clients 400 were from Taking Charge!, 178 from Employment Connections, 85 from Opportunities for Employment, 38 from Pathways to Success and 300 from the Control Group (individuals who had not completed a substantial intervention before September 30, 1997). We also re-interviewed in June 1998, to create a nine month follow-up for those who had left training in September 1997.
In the follow-up survey we asked income assistance clients (see Appendix 1, Volume II for further details) to evaluate the following:
Taking Charge! clients expressed satisfaction with the training interventions provided either in-house or through contact with training service providers (results are summarized in Table 39).
| TABLE 39 Satisfaction with training intervention |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| 358 | |||||
| a.The program was at the right level for me | 42 | 38 | 113 | 165 | |
| 12% | 11% | 32% | 46% | 101% | |
| b. The program did not help my job finding skills | 155 | 60 | 55 | 69 | 339 |
| 46% | 18% | 16% | 20% | 100% | |
| c. I am more self-sufficient because I took the program | 63 | 39 | 86 | 167 | 355 |
| 18% | 11% | 24% | 47% | 100% | |
| d. My job skills are the same as before I took the program | 42 | 67 | 50 | 94 | 353 |
| 40% | 19% | 14% | 27% | 100% | |
| e. The program helped me become more motivated | 51 | 21 | 60 | 224 | 356 |
| 14% | 6% | 17% | 63% | 100% | |
Note: Row totals may not sum to 100 percent due to rounding. |
|||||
In general, clients expressed a high level of satisfaction with Taking Charge!. Most clients reported that Taking Charge! treated them with respect, showed personal interest, and gave them good service (results are summarized in Table 40).
| TABLE 40 Satisfaction with Taking Charge! |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| a. Taking Charge! showed an acceptable level of interest in me | 40 | 36 | 119 | 239 | 434 |
| 9% | 8% | 27% | 55% | 99% | |
| b. Taking Charge! did not meet my needs | 199 | 83 | 82 | 71 | 435 |
| 46% | 19% | 19% | 16% | 100% | |
| c. They respected me at Taking Charge! | 23 | 21 | 95 | 289 | 428 |
| 5% | 5% | 22% | 68% | 100% | |
| d. My future is the same as before I went to Taking Charge! | 209 | 88 | 50 | 79 | 426 |
| 49% | 21% | 12% | 19% | 101% | |
| e. Taking Charge! gave me good service | 38 | 31 | 108 | 256 | 433 |
| 9% | 7% | 25% | 59% | 100% | |
| f. I am more self-confident because I went to Taking Charge! | 67 | 47 | 102 | 213 | 429 |
| 16% | 11% | 24% | 50% | 101% | |
| g. Taking Charge! helped me find a role model | 155 | 101 | 78 | 86 | 420 |
| 37% | 24% | 19% | 21% | 101% | |
| Note: Row totals may not sum to 100 percent due to rounding. | |||||
Job satisfaction is a composite measure of several factors, including:
The overall measure of job satisfaction depends not only on how much the individuals earn, but on how this earning is related to their expectations and level of training. It is a measure not only of present earnings and working conditions, but future aspirations and opportunities to advance within the workplace.
Although job satisfaction does not provide a direct assessment of income assistance programs (such as Taking Charge!), it offers a snapshot of client satisfaction with jobs obtained subsequent to Taking Charge! training.
These results are excerpted from the follow-up survey, where we collected information on clients at the 6, 9 and "present day" 58 period following training completion. Only clients finding work at the specified period were asked whether they were satisfied with their job. Nine month data are presented in Table 41, as this is the most consistently measured and most distant period following training completion. For further details, see Appendix 1, Volume II.
| TABLE 41 Job satisfaction 9 months after completing of Taking Charge! training |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| The pay was about right for my level of training | 42 | 22 | 56 | 87 | 207 |
| 20% | 11% | 27% | 42% | 100% | |
| The job provided the right opportunities to get ahead | 51 | 31 | 39 | 84 | 205 |
| 25% | 15% | 19% | 41% | 100% | |
| The program gave me the number of hours I wanted to work per week | 43 | 15 | 32 | 116 | 206 |
| 21% | 7% | 16% | 56% | 100% | |
| The benefits fit the job | 61 | 21 | 36 | 80 | 198 |
| 31% | 11% | 18% | 40% | 100% | |
| I was satisfied with the job | 31 | 8 | 55 | 112 | 206 |
| 15% | 4% | 27% | 54% | 100% | |
| Note: Row totals may not sum to 100 percent due to rounding. | |||||
The follow-up survey of income assistance clients allowed us to compare the perceptions and activities of Taking Charge! participants (the Treatment Group) with those from Employment Connections, Opportunities for Employments, and Pathways to Success (the Comparison Group).59 A sample of individuals who had not completed a substantial training intervention prior to September 30, 1997, also participated in the survey (the Control Group).
We asked only qualified individuals from the Treatment and Comparison Groups questions pertaining to training interventions and income assistance programs in general. All individuals responded to job satisfaction questions if they were employed at the specified time period following completion of a training intervention.
To measure client satisfaction with training interventions we asked respondents whether their training:
Only participants who said they had completed an intervention with their organization responded to this set of questions. Results for the Pathways to Success program have been included in the following tables but the sample sizes are too small to support any conclusions in relation to Taking Charge!.
For most individuals, training programs were at the right levelOpportunities for Employment participants gave the highest rating, with 89 percent agreeing (somewhat or strongly) that the program was at the right level for them. Participants from Employment Connections and Taking Charge! had similar perceptions, with approximately 75 percent agreeing with the statement (Table 42).
|
TABLE 42 The program was at the right level for me |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 42 | 38 | 113 | 165 | 358 |
| 12% | 11% | 32% | 46% | 101% | |
| Employment Connections | 9 | 12 | 32 | 28 | 81 |
| 11% | 15% | 40% | 35% | 101% | |
| Opportunities for Employment | 41 | 3 | 11 | 21 | 36 |
| 3% | 8% | 31% | 58% | 100% | |
| Pathways | - | - | - | 1 | 1 |
| - | - | - | 100% | 100% | |
| Total | 52 | 53 | 156 | 215 | 476 |
| 11% | 11% | 33% | 45% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways. |
|||||
Most programs have only a moderate impact on job finding skills
The most satisfied participants were those from the Opportunities for Employment programs, of which 74 percent disagreed or strongly disagreed with the statement "The program did not help my job finding skills," suggesting their program improved their abilities in this area. About two-thirds of Taking Charge! and Employment Connections participants felt that their job finding skills had improved (disagreed strongly or somewhat) (Table 43).
|
TABLE 43 The program DID NOT help my job finding skills |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 155 | 60 | 55 | 69 | 339 |
| 46% | 18% | 16% | 20% | 100% | |
| Employment Connections | 32 | 20 | 17 | 15 | 84 |
| 38% | 24% | 20% | 18% | 100% | |
| Opportunities for Employment | 16 | 9 | 5 | 4 | 34 |
| 47% | 27% | 15% | 12% | 101% | |
| Pathways | 1 | - | - | - | 1 |
| 100% | - | - | - | 100% | |
| Total | 204 | 89 | 77 | 88 | 458 |
| 45% | 19% | 17% | 19% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways. |
|||||
Treatment and Comparison programs have had a similar impact on increasing self-sufficiency
Taking Charge! and Employment Connections participants had similar perceptions about the effect their training intervention had on improving their sense of self-sufficiency (approximately 71 percent agreed with the statement). OFE respondents gave their program a slightly higher rating (77 percent), though sample sizes are small (Table 44).
|
TABLE 44 I am more self-sufficient because I took the program |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 63 | 39 | 86 | 167 | 355 |
| 18% | 11% | 24% | 47% | 100% | |
| Employment Connections | 11 | 14 | 28 | 30 | 83 |
| 13% | 17% | 34% | 36% | 100% | |
| Opportunities for Employment | 6 | 2 | 8 | 19 | 35 |
| 17% | 6% | 23% | 54% | 100% | |
| Pathways | 1 | - | - | - | 1 |
| 100% | - | - | - | 100% | |
| Total | 81 | 55 | 122 | 216 | 474 |
| 17% | 12% | 26% | 46% | 100% | |
| Note: We have small sample sizes for Opportunities for Employment and Pathways. | |||||
Training programs have slight impact improving job skill for Taking Charge! and Employment Connections participants
Approximately 59 percent of Taking Charge! participants and 66 percent of those from Opportunities for Employment believed that their job skills had improved as a result of their program. Employment Connections participants were the most dissatisfied, as 54 percent agreed somewhat (24 percent) or strongly (30 percent) that their job skills were the same as before taking the program. Conversely, a sizable group of Taking Charge! participants saw an improvement in their job skills (40 percent disagree strongly). Recall that Employment Connections is a limited intervention designed to place people in work. Increasing job skill is not a major goal for EC. For Taking Charge!, many of the interventions are intended to raise job skill. The different ratings may reflect perceived differences in program effect (Table 45).
| TABLE 45 My job skills are the same as before I took the program |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 142 | 67 | 50 | 94 | 353 |
| 40% | 19% | 14% | 27% | 100% | |
| Employment Connections | 16 | 22 | 19 | 24 | 81 |
| 20% | 27% | 24% | 30% | 101% | |
| Opportunities for Employment | 16 | 7 | 3 | 9 | 35 |
| 46% | 20% | 9% | 26% | 101% | |
| Pathways | 1 | - | - | - | 1 |
| 100% | - | - | - | 100% | |
| Total | 175 | 96 | 72 | 127 | 470 |
| 37% | 20% | 15% | 27% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways. |
|||||
Employment training programs are good at increasing motivation
A similar proportion of participants from the Treatment and Comparison Groups report increased motivation as a result of their training intervention. On average, 80 percent of participants agreed strongly or somewhat with the statement. Little difference exists among the programs (Table 46).
| TABLE 46 The program helped me become more motivated |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 51 | 21 | 60 | 224 | 356 |
| 14% | 6% | 17% | 63% | 100% | |
| Employment Connections | 9 | 9 | 14 | 50 | 82 |
| 11% | 11% | 17% | 61% | 100% | |
| Opportunities for Employment | 3 | 3 | 6 | 24 | 36 |
| 8% | 8% | 17% | 67% | 100% | |
| Pathways | - | - | - | 1 | 1 |
| - | - | - | 100% | 100% | |
| Total | 63 | 33 | 80 | 299 | 475 |
| 13% | 7% | 17% | 63% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways. |
|||||
To measure client satisfaction with income assistance employment training programs, we asked respondents whether their organization:
Responses in this section reflect the perceptions that individuals had of both their own program and any of the other Comparison Group program they may have taken an intervention with.
For most participants, the Treatment and Comparison Programs showed an acceptable level of interest
Overall, responses were largely positive (see Table 47). Among participants from all programs, respondents from Taking Charge! and Employment Connections were slightly more likely to report that their program did not show enough interest in them (17 percent either disagreed strongly or somewhat) (Table 47).
| TABLE 47 The program showed an acceptable level of interest in me |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 40 | 36 | 119 | 239 | 434 |
| 9% | 8% | 27% | 55% | 99% | |
| Employment Connections | 17 | 16 | 57 | 106 | 196 |
| 9% | 8% | 29% | 54% | 100% | |
| Opportunities for Employment | 7 | 6 | 30 | 82 | 125 |
| 6% | 5% | 24% | 66% | 101% | |
| Pathways | 1 | - | 4 | 4 | 9 |
| 11% | - | 44% | 44% | 99% | |
| Total | 65 | 58 | 210 | 431 | 764 |
| 9% | 8% | 27% | 56% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Income assistance training programs meet the needs of most participants
As shown in Table 48, approximately two-thirds (64 percent) of all participants said that the program had met their needs (disagree strongly or somewhat). Of the four organizations, participants from Taking Charge! were most satisfied, with 46 percent disagreeing strongly with the statement (Table 48).
| TABLE 48 The program DID NOT meet my needs |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| 82 | |||||
| Taking Charge! | 199 | 83 | 71 | 435 | |
| 46% | 19% | 19% | 16% | 100% | |
| Employment Connections | 74 | 49 | 33 | 35 | 191 |
| 39% | 26% | 17% | 18% | 100% | |
| Opportunities for Employment | 49 | 25 | 23 | 25 | 122 |
| 40% | 20% | 19% | 20% | 99% | |
| Pathways | 4 | 2 | 1 | 1 | 8 |
| 50% | 25% | 13% | 13% | 101% | |
| Total | 326 | 159 | 139 | 132 | 756 |
| 43% | 21% | 18% | 17% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Nine out of ten respondents felt respected in their program
Participants from Taking Charge!, Employment Connections and Opportunities for Employment all had similar reactions to their programs, with 90 percent or more either agreeing strongly or somewhat with the statement "They respected me at the program" (Table 49).
| TABLE 49 They respected me at the program |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 23 | 21 | 95 | 289 | 428 |
| 5% | 5% | 22% | 68% | 100% | |
| Employment Connections | 11 | 6 | 42 | 136 | 195 |
| 6% | 3% | 22% | 70% | 101% | |
| Opportunities for Employment | 6 | 5 | 27 | 87 | 125 |
| 5% | 4% | 22% | 71% | 102% | |
| Pathways | 1 | 1 | 2 | 4 | 8 |
| 13% | 13% | 25% | 50% | 101% | |
| Total | 41 | 33 | 166 | 516 | 756 |
| 5% | 4% | 22% | 68% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Taking Charge! participants are the most optimistic about the future after having participated in their program
Seventy percent of Taking Charge! participants believe that their future is better because they have been involved in the program (disagree strongly or somewhat). This rating is higher than either Employment Connections (57 percent) or Opportunities for Employment (53 percent) (Table 50).
| TABLE 50 My future is the same as before I went to the program |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 209 | 88 | 50 | 79 | 426 |
| 49% | 21% | 12% | 19% | 101% | |
| Employment Connections | 73 | 37 | 34 | 46 | 190 |
| 38% | 19% | 18% | 24% | 99% | |
| Opportunities for Employment | 45 | 19 | 17 | 41 | 122 |
| 37% | 16% | 14% | 34% | 101% | |
| Pathways | 3 | 4 | - | 2 | 9 |
| 33% | 44% | - | 22% | 99% | |
| Total | 330 | 148 | 101 | 168 | 747 |
| 44% | 20% | 14% | 22% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Income Assistance training programs give clients good service
Approximately 90 percent of participants from Employment Connections and 85 percent of these from Opportunities for Employment and Taking Charge! reported receiving good service from their program (agree somewhat or strongly) (Table 51).
| TABLE 51 The program gave me good service |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 38 | 31 | 108 | 256 | 433 |
| 9% | 7% | 25% | 59% | 100% | |
| Employment Connections | 11 | 6 | 61 | 116 | 194 |
| 6% | 3% | 31% | 60% | 100% | |
| Opportunities for Employment | 14 | 5 | 26 | 80 | 125 |
| 11% | 4% | 21% | 64% | 100% | |
| Pathways | - | 2 | 3 | 4 | 9 |
| - | 22% | 33% | 44% | 99% | |
| Total | 63 | 44 | 198 | 456 | 761 |
| 8% | 6% | 26% | 60% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Taking Charge! participants are more likely to report increased confidence
Seventy-four percent of Taking Charge! participants, and 69 percent of those who went to OFE, agreed that they were more confident after having become involved with the program. Respondents from Employment Connections were slightly less likely to report an increase in confidence (63 percent said they agreed strongly or somewhat) (Table 52).
| TABLE 52 I am more confident because I went to the program |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 67 | 47 | 102 | 213 | 429 |
| 16% | 11% | 24% | 50% | 101% | |
| Employment Connections | 33 | 37 | 49 | 70 | 189 |
| 17% | 20% | 26% | 37% | 100% | |
| Opportunities for Employment | 26 | 13 | 28 | 56 | 123 |
| 21% | 11% | 23% | 46% | 101% | |
| Pathways | 1 | 1 | 1 | 5 | 8 |
| 13% | 13% | 13% | 63% | 102% | |
| Total | 127 | 98 | 180 | 344 | 749 |
| 17% | 13% | 24% | 46% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Less than 40 percent of participants report finding a role model
Approximately four out of ten Taking Charge! and Opportunities for Employment participants said that they found a role model at their program. Individuals from Employment Connections were less likely to do so (26 percent agreed strongly or somewhat) (Table 53).
| TABLE 53 The program helped me find a role model |
|||||
| Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total | |
| Taking Charge! | 155 | 101 | 78 | 86 | 420 |
| 37% | 24% | 19% | 21% | 101% | |
| Employment Connections | 87 | 53 | 27 | 23 | 190 |
| 46% | 28% | 14% | 12% | 100% | |
| Opportunities for Employment | 46 | 27 | 16 | 33 | 122 |
| 38% | 22% | 13% | 27% | 100% | |
| Pathways | 1 | 4 | 1 | 3 | 9 |
| 11% | 44% | 11% | 33% | 99% | |
| Total | 289 | 185 | 122 | 145 | 741 |
| 39% | 25% | 16% | 20% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Pathways to Success. |
|||||
Overall, about 43 percent of clients reported having full- or part-time employment nine months after completion of their intervention. Clients who gained employment were asked a series of questions to measure their level of satisfaction. In general, employed clients were satisfied with jobs acquired after training, although 25 percent to 35 percent appeared to be quite unhappy about various aspects of their employment. Results are presented in Tables 54 to 58. For further details, see Appendix 1, Volume II.
Approximately two-thirds of respondents are satisfied with their level of pay
Taking Charge! participants were most satisfied with their level of pay (69 percent agreed somewhat or strongly). Employment Connections and Opportunities for Employment participants had very similar perceptions (both 66 percent) (see Table 54). Respondents from the Control Group report similar satisfaction levels.
| TABLE 54 The pay was about right for my level of training |
|||||
| Program | Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total |
| Taking Charge! | 42 | 22 | 56 | 87 | 207 |
| 20% | 11% | 27% | 42% | 100% | |
| Employment Connections | 15 | 11 | 19 | 33 | 78 |
| 19% | 14% | 24% | 42% | 99% | |
| Opportunities for Employment | 7 | 4 | 5 | 1 | 17 |
| 22% | 13% | 16% | 50% | 101% | |
| Pathways | 3 | 0 | 1 | 1 | 5 |
| 60% | - | 20% | 20% | 100% | |
| CONTROL | 31 | 8 | 25 | 42 | 106 |
| 29% | 8% | 24% | 40% | 101% | |
| Totals | 98 | 45 | 106 | 179 | 428 |
| 23% | 11% | 25% | 42% | 101% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways to Success. |
|||||
Approximately six of ten participants believe that their job provided the right opportunities to get ahead
Participants from Taking Charge!, OFE and EC had similar perceptions regarding the future opportunities provided by their job. For each of these programs, approximately 60 percent of participants agreed strongly or somewhat with the statement below. Control Group respondents are slightly less optimistic (Table 55).
| TABLE 55 The job provided the right opportunities to get ahead |
|||||
| Program | Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total |
| Taking Charge! | 51 | 31 | 39 | 84 | 205 |
| 25% | 15% | 19% | 41% | 100% | |
| Employment Connections | 22 | 10 | 15 | 31 | 78 |
| 28% | 13% | 19% | 40% | 100% | |
| Opportunities for Employment | 8 | 3 | 10 | 11 | 32 |
| 25% | 9% | 31% | 34% | 99% | |
| Pathways | 1 | 0 | 3 | 1 | 5 |
| 20% | - | 60% | 20% | 100% | |
| CONTROL | 37 | 15 | 25 | 31 | 108 |
| 34% | 14% | 23% | 29% | 100% | |
| Totals | 119 | 59 | 92 | 158 | 428 |
| 28% | 14% | 22% | 37% | 101% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways to Success. |
|||||
More than two thirds of participants work the number of hours that they want to
More than 70 percent of OFE and Taking Charge! participants were working the number of hours that they wanted to (agree strongly or somewhat), while EC and Control Group participants gave a slightly lower satisfaction rating (66 percent for each) (Table 56).
| TABLE 56 The job gave me the number of hours I wanted to work |
|||||
| Program | Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total |
| Taking Charge! | 43 | 15 | 32 | 116 | 206 |
| 21% | 7% | 16% | 56% | 100% | |
| Employment Connections | 20 | 6 | 10 | 41 | 77 |
| 26% | 8% | 13% | 53% | 100% | |
| Opportunities for Employment | 7 | 1 | 6 | 18 | 32 |
| 22% | 3% | 19% | 56% | 100% | |
| Pathways | 1 | 1 | 1 | 2 | 5 |
| 20% | 20% | 20% | 40% | 100% | |
| CONTROL | 28 | 10 | 16 | 56 | 110 |
| 26% | 9% | 15% | 51% | 101% | |
| Totals | 99 | 33 | 65 | 233 | 430 |
| 23% | 8% | 15% | 54% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways to Success. |
|||||
Only a small majority of participants said that the benefits fit their job
Taking Charge! participants were most satisfied with the benefits associated with their job (58 percent agreed strongly or somewhat). Satisfaction ratings were lowest among OFE participants, though sample sizes are small and the differences are slight (Table 57).
| TABLE 57 The benefits fit the job |
|||||
| Program | Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total |
| Taking Charge! | 61 | 21 | 36 | 80 | 198 |
| 31% | 11% | 18% | 40% | 100% | |
| Employment Connections | 32 | 4 | 12 | 25 | 73 |
| 44% | 6% | 16% | 34% | 100% | |
| Opportunities for Employment | 12 | 5 | 4 | 9 | 30 |
| 40% | 17% | 13% | 30% | 100% | |
| Pathways | 2 | 0 | 1 | 2 | 5 |
| 40% | - | 20% | 40% | 100% | |
| CONTROL | 43 | 7 | 21 | 33 | 104 |
| 41% | 7% | 20% | 32% | 100% | |
| Totals | 150 | 37 | 74 | 149 | 410 |
| 37% | 9% | 18% | 36% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways to Success. |
|||||
On average, 80 percent of respondents were satisfied with their jobs
More than 80 percent of Taking Charge! and OFE respondents were satisfied with their jobs (agree strongly or somewhat). A slightly smaller proportion of EC participants were similarly satisfied (75 percent) (Table 58).
| TABLE 58 I was satisfied with the job |
|||||
| Program | Disagree strongly | Disagree somewhat | Agree somewhat | Agree strongly | Total |
| Taking Charge! | 31 | 8 | 55 | 112 | 206 |
| 15% | 4% | 27% | 54% | 100% | |
| Employment Connections | 15 | 5 | 23 | 35 | 78 |
| 19% | 6% | 30% | 45% | 100% | |
| Opportunities for Employment | 3 | 2 | 11 | 16 | 32 |
| 9% | 6% | 34% | 50% | 99% | |
| Pathways | 0 | 0 | 2 | 3 | 5 |
| - | - | 40% | 60% | 100% | |
| CONTROL | 16 | 5 | 31 | 56 | 108 |
| 15% | 5% | 29% | 52% | 101% | |
| Totals | 65 | 20 | 122 | 222 | 429 |
| 15% | 5% | 28% | 52% | 100% | |
| Note 1: Row totals may not sum to 100 percent due to rounding. Note 2: We have small sample sizes for Opportunities for Employment and Pathways to Success. |
|||||
Overall program satisfaction
Participants in the follow-up survey all expressed high satisfaction with their programs. Taking Charge! participants showed slightly higher scores for some items (such as "I feel more confident because I went to the program"). In other aspects, Taking Charge! participants gave slightly lower scores. Overall, training participants reported being more motivated, self-confident, and optimistic about the future after having participated in their training programs.
Overall success of programs
In general, most clients are satisfied with their jobs. A majority of respondents (about 70 percent for Taking Charge! and Employment Connections) accept that their pay matches their skills. This compares closely with Control Group members, of which 64 percent agreed with the statement "The pay was about right for my level of training."
There are, however, 25-37 percent who appear to be unhappy with various aspects of their employment. These negative reactions may reflect more on the workplace than on the training program(s) which preceded it.
The Taking Charge! survey of employers was meant to examine employers' perceptions of individuals hired as a result of a work placement program. As designed, this survey would have allowed us to compare overall employer satisfaction with placed individuals, the nature of the relationship between the business community and income assistance programs, and any differences in the types of employment that individuals received.
In the past year, new provincial legislation was introduced to further guarantee personal privacy by prohibiting government and non-government organizations from releasing specific types of information (such as health or income assistance information) to the public without the prior consent of that individual. We could not identify employees who had been on income assistance to employers. This meant that many employers would not be able to assess the outcomes of training interventions unless they knew that the employee had participated (i.e. through a work placement).
We redesigned the employer survey to give it a more generic format in which individual employees were not identified to employers. The final survey thus differed from the original concept and design, substantially limiting its effectiveness (see Appendix 2 for further details).
Survey response rate
PRA Inc. administered the survey by fax to 209 employers selected randomly from information provided by Taking Charge!. If employers were unable to identify work placement individuals, we asked participants to fax back the first page of the questionnaire indicating why they were unable to answer the survey. As we suspected, only 18 percent (37 respondents) of those surveyed believed they had enough information to complete the survey. Meanwhile, 33 percent of participants called or sent the survey back and indicated that they were unable to respond because they had no knowledge of the employees we were referring to. The remainder apparently discarded the survey.
Because of this low response rate, the following provides only a general view of employers' perceptions. The results are therefore broadly "indicative," not statistically significant.
Approximately 80 percent of respondents agreed (60 percent) or strongly agreed (19 percent) that individuals hired as part of a work placement program "worked hard" (see Table 59 for results).
Two-thirds of respondents reported that their work-placement candidates showed initiative (agreed or strongly agreed), while one-quarter remained neutral.
Nearly 60 percent of employers were either neutral (43 percent) or disagreed (16 percent) with the statement that their work placement employees worked well independently.
Approximately half the respondents (48 percent) reported that their employees worked well with others (strongly disagree or disagree). One quarter did not think that their work placement employees worked well with others (27 percent agree) while the remainder were neutral (22 percent).
Over 75 percent of respondents said that their work placement candidates worked well with supervisors (strongly agree or agree).
Most employees met or exceeded employers' expectations
On average, three-quarters of work placement employees met or exceeded employer requirements (see Table 60 for results).
Approximately half of the respondents either agreed (49 percent) or strongly agreed (3 percent) with the statement "The work placement program understood my business needs."
Most respondents were neutral (49 percent) when asked whether the programs and/or service providers had provided "well trained candidates."
More than three-quarters of respondents agreed (60 percent) or strongly agreed (16 percent) that business "has a responsibility to work placement programs." A similar proportion of respondents said they would "look to work placement programs to fill future job vacancies" (65 percent agree and 11 percent strongly agree).
On average, respondents rated input into training 4.1 on a scale of one to five (Table 62). This suggests that input into training is an important factor when employers are deciding whether to hire a work placement candidate.
Wage subsidization was rated 3.5 on this same scale, suggesting that it too is an important factor when employers are making a hiring decision, but not as important as input into training.
Other key factors independently cited were social responsibility and good work skills. Both factors rated a 5 among all respondents who cited them, but note the low response rate.
| TABLE 62 When making the decision to hire a work placement candidate, how important are the following factors: (Rate on a scale of 1 to 5, five being most important) |
|
| Factors influencing employment | Average rating |
| Input into training (n=28) | 4.1 |
| Wage subsidy (n=28) | 3.5 |
| Social responsibility (n=4) | 5 |
| Good work skills (n=4) | 5 |
| Source: Survey of Employers. | |
In general, comments reflected favorably on work placement programs. Several comments pointed to the strong and mutually beneficial relationship between employers and work placement employees:
For the most part, staff enjoy having students. It is a two way street. We can all learn methods of interacting to make life better.
We always welcome people from this source with the hope of eventual placement.
In addition to these positive comments, respondents raised three concerns/issues:
Adjusting to the workplace: Expectations are sometimes too high at the outset
Although trainees are aware and capable of using computers, they lack office procedure, understanding, independent motivation, and initiative. They take a six- week training course and think they should be making $10 to $12 an hour. They do not understand that small organizations and businesses with limited incomes can't normally pay that amount. They also do not seem to understand that an organization that has a starting wage of $7 to $8 an hour will not give increases of $2 or $3 an hour and that it takes time, dedication, loyalty, perseverance, good work ethic, punctuality, and a whole lot more, to attain desired wage increases.
Beyond the workplace: dealing with personal problems in addition to training
People need much more bridging to make the transition from assistance to employment. The sudden disruption in subsidies to childcare (housing, drug costs, and in our case we required a car) causes great stress and financial instability. ... They are also in no position to negotiate personal loans from traditional sources because they have no credit rating.
For the most part they want to work. They need more help when their children are sick. Day care will not take sick children.
Input into training: working with employers to identify needed skills
Many of the work placement candidates are coming through with low skill levels which places a large demand on us to train these individuals.
Being a high-tech firm, most job entry programs do not provide the level of candidates we can easily integrate into the organization.
There is a need to have training in areas of transferable skills with input from the labour market.
These three issues all indicate areas for future work placement program development, particularly as case loads become more challenging.
A central feature of the Taking Charge! program is its reliance on third party organizations to assess clients and deliver training. These groups are in a unique position to influence the way training is designed and delivered.
The survey of training and other service providers (SP) measured:
We sent questionnaires to 48 qualified training service providers who had completed a contract with Taking Charge!. Of these, 34 responded for a completion rate of 75 percent.
To assess client satisfaction with training targeted to income assistance clients, we asked service providers whether:
Service providers gave mixed opinions about changing caseloads, though some feel that motivation has increased
As summarized in Table 63, training service providers gave a mixed report on changing caseloads, though some indicated that clients now show more desire to succeed:
Service providers are mixed on the issue of service duplication
Thirty-eight percent of service providers either disagreed (19 percent) or strongly disagreed (19 percent) with the statement "Other service providers have developed similar training services as the ones we offer income assistance clients." At the same time, one-third reported that service duplication was occurring (agreed with the statement) (See Table 64 for details).
Service providers see little impact of work expectations on client attitudes
Almost 40 percent of those surveyed did not think that clients with a work expectation had a "better attitude than clients who volunteer for the program" (11 percent strongly disagreed, 28 percent disagreed). Only 17 percent agreed with the statement (See Table 64 for details).
No trend in completion rates has been detected by service providers
Nearly one-fifth of respondents reported that "a higher percentage of clients complete their courses now compared to two years ago" (19 percent agreed). The majority offered no opinion or remained neutral (See Table 64 for details).
| TABLE 64 General perceptions of clients (n=36) |
|||||||
| Strongly disagree | Disagree | Neutral | Agree | Strongly agree | No opinion | No response | |
| Clients with a work expectation have a better attitude than clients who volunteer for the program | 4 | 10 | 4 | 6 | - | 8 | 4 |
| (11%) | (28%) | (11%) | (17%) | - | (22%) | (11%) | |
| Other service providers have developed similar training services as the ones we offer income assistance clients | 7 | 7 | 2 | 12 | - | 4 | 4 |
| (19%) | (19%) | (6%) | (33%) | - | (11%) | (11%) | |
| A higher percentage of clients complete their courses now compared to two years ago | 1 | 4 | 10 | 7 | - | 10 | 4 |
| (3%) | (11%) | (28%) | (19%) | - | (28%) | (11%) | |
|
Source: Survey of Service Providers. Note: Row totals may not sum to 100 percent due to rounding. |
|||||||
To examine how Taking Charge! interacted with service providers, we asked service providers to evaluate:
Most service providers commend Taking Charge! for its client support, with many considering it a needed income assistance program
To compare Taking Charge! with other service funders, we asked respondents whether they had contracted with Employment Connections and/or another funder to provide training or other services in the last two years. Other service provider funders include: Employment Connections, the Winnipeg Development Agreement, Community Partnerships, the Manitoba Métis Federation, the Winnipeg Aboriginal Management Board, the Regional Aboriginal Management Board, the federal government, and a number of educational facilities.
Other funders are good at communicating client information in a clear manner and assisting with problems as they arise
As indicated in Table 66, 58 percent of service providers reported that the Other Funder was "clear in communicating the needs of their clients to us" (agreed or strongly agreed). A similar proportion of respondents (59 percent) reported that the Other Funder "assisted with any problems that arose with their clients."
A slightly smaller proportion of respondents said that the Other Funder "supported their clients throughout their involvement" in the training program (29 percent agreed and 18 percent strongly agreed).
The majority of respondents were either neutral (41 percent) or responded with not applicable (29 percent) when asked about the usefulness of an assessment/training plan provided by the Other Funder. The development of a personal training program was meant to be a unique feature of the Taking Charge! program, and Other Funders may not use it as part of their training programs.
To strengthen income assistance programs, it is necessary to understand the challenges clients present to service and other training providers. A general assumption is that clients are becoming harder to train. Many clients report being confronted by personal challenges that hinder their attempts to successfully complete (and in some cases enter) a training program.
To further this discussion, we asked service providers to give general comments on the challenges they face with funders (such as Taking Charge!, Employment Connections, etc.) and with other service providers. These questions help reveal whether the main challenges lie with clients, and/or with the relationship which service providers have with other organizations within the income assistance system.
Collateral supports such as day care and bus passes are seen as essential for reducing dependency on income assistance
Only 36 percent of respondents reported that "having a work expectation assigned by the province" was essential or important. A slightly larger proportion offered no opinion (14 percent) or remained neutral (25 percent responded 3 on a 5 point scale) (See Table 67).
Personal motivation and a positive attitude lead the list of "other factors" essential to helping individuals reduce their dependency on income assistance
Twelve service providers listed other factors as essential for reducing income assistance dependency through employment (i.e. each of the listed attributes was rated a 5 out of 5 by the service provider giving the listing).
The three most common attributes were receiving basic supports (cited 4 times), having a positive attitude (3) and personal motivation/goal setting (3). Service providers also listed the need for pre-training experience (2), adequate training allowances (1), and having a role model (1) as key factors for reducing dependency.
We asked service providers to indicate what "special types of challenges" they face in training income assistance clients. We categorized responses in three ways:
Service providers see low self-esteem and personal problems as the two main challenges faced by clients
Low self-esteem and personal problems led the list of challenges presented by clients. Each was cited by 39 percent of those providing responses.
Low motivation (26 percent), family issues (16 percent) and lack of basic education (16 percent) were also frequently cited as key challenges. Others were financial in nature, including the need for child-care (26 percent) and having adequate financial supports while involved in a training program (26 percent) (Table 68).
| TABLE 68 Challenges with clients (n=31) |
||
| Challenge | Number of responses | Percentage of cases |
| Low self-esteem | 12 | 39% |
| Personal problems | 12 | 39% |
| Child care supports required | 8 | 26% |
| Finances | 8 | 26% |
| Low motivation | 8 | 26% |
| Family issues | 5 | 16% |
| Lack of basic education | 5 | 16% |
| Unrealistic expectations | 4 | 13% |
| Lack of control over clients | 3 | 10% |
| Attendance | 3 | 10% |
| Other | 5 | 16% |
| Source: Survey of Service Providers. Note: Respondents could choose more than one answer. Column totals sum to more than 100 percent. |
||
A high proportion of service providers report communication problems with some funders
Lack of effective communication was cited as the most frequent challenge that service providers face in their relationships with funders. This response was proportionately higher with Taking Charge!, but sample sizes are too small to draw conclusions.
Financial restraints (26 percent), too much paperwork (22 percent), lack of proper assessments (19 percent), and more time needed to develop programs (15 percent) were other commonly cited problems (Table 69).
| TABLE 69 Challenges with other funders such as Taking Charge!, Employment Connections, or Other Funders |
|||
| Challenge | Total (n=27) | Taking Charge! (n=6) | All (n=21) |
| Lack of effective communication | 10 | 3 | 7 |
| (37%) | (50%) | (33%) | |
| Financial restraints | 7 | 2 | 5 |
| (26%) | (33%) | (24%) | |
| Lack of proper assessments | 5 | 2 | 3 |
| (19%) | (33%) | (14%) | |
| More time needed to develop programs | 4 | 2 | 2 |
| (15%) | (33%) | (10%) | |
| Too much paperwork | 6 | 2 | 4 |
| (22%) | (33%) | (19%) | |
| Poorly organized | 2 | 2 | - |
| (7%) | (33%) | ||
| More administrative support required | 2 | 1 | 1 |
| (7%) | (17%) | (5%) | |
| Lack of client supports | 2 | - | 2 |
| (7%) | (10%) | ||
| +Unrealistic expectations of trainers | 2 | - | 2 |
| (7%) | (10%) | ||
| Lack of support for young clients | 2 | - | 2 |
| (7%) | (10%) | ||
| Low/poor referrals | 2 | - | 2 |
| (7%) | m | (10%) | |
| Other | 5 | 2 | 3 |
| (19%) | (33%) | (15%) | |
| Source: Survey of Service Providers. Note 1: Respondents could choose more than one answer. Column totals sum to more than 100 percent as percentages represent percentage of respondents listing a specific challenge. Note 2: Several service providers indicated that comments were in relation to Taking Charge! only. |
|||
Challenges with other service providers
Respondents cited limited training time as the main challenge shared with other service providers (44 percent). Frustration over keeping statistical data and the cost of accessing programs were also mentioned more than once.
Other issues included lack of incentives, transfer of credits between programs, willingness to invest time, recommendations not needed, pre-training program needed, the need for more training options, difficulty placing students and inexperienced trainers (all cited once).
| TABLE 70 Challenges with other training service providers (n=9) |
||
| Challenge | Number of responses | Percentage of cases |
| Limited time for training | 4 | 44% |
| Frustration over keeping statistics | 2 | 22% |
| Cost of accessing programs | 2 | 22% |
| Source: Survey of Service Providers. | ||
At the end of the survey, we asked respondents two open-ended questions:
Overall, responses tended to be critical of Taking Charge!. This stands in contrast to the generally positive assessment in the sections where responses were constrained to numeric representations. This is common in survey research, as respondents critical of a program often provide lengthy commentary, while those who are satisfied reserve their positive comments to a few short sentences.
At the same time, the comments offer insight into the perceived weaknesses of Taking Charge!, highlighting areas that may need to be strengthened. Negative comments may relate to the profile that Taking Charge! has had with community centres and the interest of some training service providers to work directly with the government. Comments may also reflect the fact that Taking Charge! has become more defined in what it needs (i.e., employment programming), resulting in rejections of proposals that do not fit.
Key findings
| TABLE 71 Do you have any comments about how the training of income assistance recipients is organized by Taking Charge! (n=26) |
||||
| Issue | Negative | Positive | Neutral | Total |
| Internal organization and communication | 11 | 2 | - | 13 |
| 39% | 7% | 0% | 46% | |
| Referral process | 5 | - | 1 | 6 |
| 18% | 0% | 4% | 21% | |
| Pre-training and assessment | 9 | 1 | 3 | 13 |
| 32% | 4% | 11% | 46% | |
| Program design | 6 | 4 | - | 10 |
| 21% | 14% | 0% | 36% | |
| Community ties and recruitment | 2 | - | - | 2 |
| 7% | 0% | 0% | 7% | |
| Funding and basic supports | 4 | 4 | 2 | 10 |
| 14% | 14% | 7% | 36% | |
| Total | 37 | 11 | 6 | 54 |
| Source: Survey of Service Providers. | ||||
This section provided the following key findings regarding client satisfaction:
Service providers gave mixed response regarding their relationships with Taking Charge!. In questions constrained to numeric responses, service providers commended Taking Charge! for its clients supports, with many considering it a needed income assistance program. However, in written commentary, several service providers criticized Taking Charge!'s internal organization and communication, its referral process, its pre-training and assessment, and the overall program design.
|
Section 2 summarized the financial activities for Taking Charge! by component. Between Taking Charge!'s inception in 1994 and 1999, the federal and provincial governments allocated a total of $26 million. As a result of the start-up delays, substantial programming did not begin until the spring of 1996. Based on activity to date and projections for fiscal 1998/99, Taking Charge! will actually spend about $15 million.60
In this section we consider the cost-effectiveness for the program relative to other interventions. The main analyses compare Taking Charge! with all programs in the Comparison Group. However, often it compares Taking Charge! with just Employment Connections, the two main training interventions offered by the province. Data limitations prevented us from including all the Comparison Group programs. Most important is that Community Partnership and Pathways to Success could not offer estimates of management or infrastructure (rent, utilities) overhead costs.
These estimates offer a general view of cost effectiveness, but the different interventions deal with different clients and offer a varied mix of programming. Since we know the average reduction in income assistance payments for clients in Taking Charge! and the Control Group, the expected time clients remain on assistance in the various programs, and the cost and duration of the training, we can project the cost savings relative to program costs.
One caveat is important for the analysis in this section. We have not observed long-term outcomes. The maximum post-intervention period is 24 months. The research literature has concluded that the minimum time needed to assess long term outcomes is 36 months. In our sample, many participants have only nine months follow-up after the conclusion of their intervention. Over the next three years, some will find employment and others will cycle on and off assistance. The cost-effectiveness analysis must be read with this limitation.
By examining the activity of each program in the last year, we can calculate the average costs of training per participant for Taking Charge! and Employment Connections.61 Table 72 summarizes the overall activities and costs for the programs. We obtained program costs from audited financial statements for Taking Charge! and statements provided by Employment Connections. These statements included all direct and indirect costs including overheads. The costs of Opportunities for Employment (OFE) are simple. OFE is paid on a per client basis, but only when a client has been employed full-time for six months. Payments to OFE are used to cover all direct and indirect costs, and its average cost per client is easy to calculate.
We were unable to obtain total program costs for Community Partnerships and Pathways to Success. Although we can obtain costs for the projects funded, we cannot extract the indirect costs such as program management, rent, utilities, etc. These costs are completely transparent for Taking Charge!, Employment Connections and Opportunities for Employment.62 We used the costs of Employment Connections as a general estimate for direct training costs of the Comparison Group as a whole.
It is important to emphasize that major elements of the cost-effectiveness analysis include all programs in the Comparison Group. Our estimates of average time on assistance, average training time, and change in benefit reductions are based on regressions that include all of the programs in the Comparison Group. Therefore, the benefit cost-exercise does largely incorporate the experience of the total Comparison Group.
Calculation of average client costs for Taking Charge poses the following three challenges:
Because Taking Charge! offers a broad range of services, it often has contact with clients over an extended time. The recorded number of registrations, assessments and training placements may not reflect the true client contact "load" during any given year. Some clients recorded as registering may not incur substantial costs until the following fiscal year. Others, registered in previous years may impose the greatest demand in the following year.
Taking Charge! claims that the true numbers served are higher than is recorded by Table 12 in any given year. It claims that it serves clients from 1996/97 in fiscal year 1997-98, our selected test year. At the same time, some clients recorded in Fiscal year 1997-98 will not be served until 1998/99.
Taking Charge! management claims that it often awards supplementary support to clients. These are one-time discretionary payments to help clients overcome barriers to project participation. These may range from purchase of additional training supplies, to assistance with costs of dealing with a family problem.
These discretionary expenditures are embedded in the costs of Taking Charge! and cannot be extracted. These payments are not typical of other programs, although clients in those programs may also receive supplementary educational and training support from Family Services. To compensate partially for these expenditures, we estimated the average costs of special education allowances received by participants in the Comparison Group programs. These additional costs, average $55 per client during the period when they took their intervention.63 It is possible that Comparison Group clients received other non-recorded payments, comparable to the discretionary support offered by Taking Charge! to its clients. We have no way of tracking these. We have elected to include the special training costs awarded in the Comparison Group programs, as opposed to attempting to extract these costs from Taking Charge!
Again, the entire approach of Taking Charge! reflects a commitment to client service that is more supportive than traditional training programming. These extra costs are part of what makes Taking Charge! unique and are included as part of its pilot design to determine whether client success is enhanced.
For Taking Charge! we decided to calculate average costs per client and per employed client (Table 72), by dividing the clients served into the total program costs to March 31, 1998. For Employment Connections we used the costs for 1997/98 divided by the clients served. Because Employment Connections is a shorter intervention and does not serve clients over a prolonged period, we can estimate average costs for a single year.
| TABLE 72 Overall costs and activity for the main interventions (fiscal year 1997/98) |
|||
| Taking Charge! 1994/95-1997/98 |
Employment Connections 1997/98 |
OFE 1996/97 |
|
| Clients served | 3237 | 937 | 569 |
| Total program cost | $10,053,735 | $1,256,200 | $156,000 |
| Average cost per client | $3,106 | $1,341 | $274 |
| Special needs payments | $6 | $12 | $33 |
| Average cost per client | $3,112 | $1,353 | $307 |
| Percent of Clients employed** | 0.45 | 0.55 | 0.07 |
| Clients employed | 1,456 | 515 | 39 |
| Average cost per client employed | $6,905 | $2,439 | $4,000 |
| * Note: For Taking Charge! this is clients served from program inception to March 31, 1998. For Employment Connections this is clients served in 1997/98 and for OFE clients served in 1996/98. ** After nine months of intervention termination, based on the Follow-up survey - See Section 6. Sources: Taking Charge! annual reports, Employment Connections Annual Reports, OFE Annual Reports. |
|||
Taking Charge! appears to be a high cost program in comparison with Employment Connections, but this conclusion should be read with the following qualifications:
Determining the cost effectiveness of Taking Charge! requires a more detailed analysis of gross and net impacts that includes the benefits of reduced income assistance as training program participants find work. This analysis is presented in the next sections.
This section examines the gross impact and cost-effectiveness for the main training interventions in Manitoba. The gross impact and cost-effectiveness measures are calculated per program. They extend the analysis in the previous section and assesses the unit cost of a program. Unit cost is defined as the cost per participant trained, cost per graduate, or cost per employed graduate.
We measure gross impacts and cost-effectiveness through the following processes:
To reiterate, for this evaluation we have three groups available for the cost comparison:
We are required to use Employment Connections as the representative of the Comparison Group. Table 73 summarizes the gross impacts and cost-effectiveness, using the measures discussed.
Several points need to be mentioned about Table 73:
Nine months after their intervention, our analysis of the SAMIN data reveals that 35 percent of Taking Charge!, 45 percent of Employment Connections, 42 percent of the Comparison Group,65 and 26 percent of the Control Group were off assistance. These results are close to those obtained in the Follow-up Survey (Appendix 1, Vol. II).
With respect to employment, 34 percent of Taking Charge! clients reported employment after 3 months dropping to 25 percent after 9 months. For Employment Connections, 39 percent report employment after 3 months before falling back down to 27 percent in month 9. For the Control Group, 14 percent found work immediately after the randomly assigned intervention. This remains stable at the three, six, and nine month periods. In part, this may be due to the changing nature of the caseload as clients come off assistance. If everyone remained on assistance, the employment rate would have increased or remained stable.
In Table 73 the average benefits paid fall by month for each group. For Taking Charge! the average benefit falls from $884 to $523 per month. For Employment Connections the reduction is $785 to $460 per month. The Control Group has the lowest decline in income assistance benefits paid falling from $897 to $722.
Reduction of benefit due to earnings (Benefit Reduction from Table 73) reflects the decrease in income assistance benefits paid as a result of increased employment earnings.66 Benefit reductions peak at approximately the six month period before falling. At the six month period, Taking Charge! participants have an average benefit reduction of $113, compared with $119 for the Comparison Group and $43 for the Control Group. The decline after six months may reflect a process where some clients move into full-time work, leaving behind clients who work less and therefore have less benefit reduction.
Another perspective on cost-effectiveness is to imagine that the programs did nothing. Everyone on assistance at the start of the intervention remains on assistance, without any employment earnings through the three, six, and nine month periods. This represents the maximum "liability" government has for a client and we can calculate that just by taking the average payment made one month prior to the end of interventions and multiplying that by the number of months (1, 4, 7, and 10).67
Program interventions lead to employment incomes and complete exits from assistance. This reduces the maximum liability for assistance payments. The difference between the average maximum liability and the average benefits paid represents the savings for government arising from the program.
Table 74 shows the average maximum liability68 per client for 0, 3, 6, and 9 months. It also shows average benefits paid, with the difference being the average savings (per client). The last column shows the advantage the program offers relative to the Control.
Based on this calculation, Employment Connections offers the largest cost savings per client at the nine month period relative to the Control Group ($2,378). The relative cost savings of Taking Charge! and the Comparison Group are comparable. Figure 8 shows the percent of liability avoided by the program. As represented in this Figure, Employment Connections appears to be the most effective labour market intervention. Taking Charge! is slightly more cost-effective than the Comparison Group, on average.
Gross impact analysis examines the relative performance of programs in terms of reducing income assistance payments. From this perspective Taking Charge! does slightly better than the Comparison Group as a whole, but not as well as Employment Connections by itself.
As is clear from the preceding analysis, a portion of the Control Group obtains employment and some even leave income assistance entirely. Using the information in Table 73 as a guide, at the ninth month after intervention, 26 percent of the Control Group are no longer on the SAMIN database and have apparently left income assistance. This represents a baseline cost savings to the provincial government in the form of avoided payments. The costs for achieving these savings is minimal and no greater than the costs of serving those remaining on assistance.
The net impact assessment considers the interventions relative to each other and the Control Group. Based on these impacts, we can prepare a benefit cost analysis. This framework integrates the overall costs of offering training, with the benefits received by government in the form of lowered income assistance payments and higher taxes from clients who earn income.
Table 75 presents the post-intervention experience in terms of the numbers remaining on assistance. We estimated the average time to come off assistance (last column of Table 75) using survival analysis presented in Section 6.5.69
| TABLE 75 Summary of post-intervention experience |
||||
| Group | Percent remaining on assistance | Average time to come off assistance** (months) | ||
| 3 months | 6 months | 9 months | ||
| Treatment (Taking Charge!) | 79% | 71% | 65% | 20 |
| Employment Connections | 76% | 63% | 55% | 14 |
| Comparison* | 79% | 67% | 58% | 16 |
| Control | 87% | 82% | 74% | 23 |
| * Note: Includes EC, OFE, Community Partners and Pathways. The average time to come off assistance just for the three groups net of Employment Connections is 17.5 months. ** See the discussion in Section 6. |
||||
On average, Control Group members take 23 months to come off assistance (after a randomly assigned "intervention" date")70 . Comparison Group clients are off assistance 16 months after intervention, and Taking Charge! clients leave assistance 20 months after their intervention.
This calculation is from the perspective of government, since it funds the training programs and the income assistance programs. Certainly benefits are obtained by clients in the form of higher incomes and economic independence, but this perspective is not included in this analysis.
Calculating Costs
Costs include:
Calculating Benefits
We identified three impact indicators that summarize the benefits received by government:
The benefit-cost framework is straightforward. The question is simple - does the investment in training generate an offsetting return in the form of lower income assistance payments and increased tax recoveries? Table 76 presents the general benefit-cost framework.
| TABLE 76 The benefit cost framework |
|||
| Costs | Benefits | ||
| A. Income assistance payments during training | E. Increased benefit reductions because of earned income | ||
| + | B. Training allowances (books, special needs) | + | F. Reduced income assistance due to lower time on welfare |
| + | C. Cost of training | + | G. Increased taxes from employment earnings |
| = | D. Total costs | = | H. Total benefits |
Item A (Table 76) may be estimated in several ways. One conception (that we prefer) is to argue that a training intervention delays employment by the length of the program. For Taking Charge! clients, this adds 4.4 months of income assistance. For Comparison Group clients, this averages 2.7 months.
Another perspective is to adjust the intervention durations for the programs. To measure income assistance payments during training, we calculated the average monthly income assistance received during training by the adjusted average duration of training. The average duration of training is adjusted to reflect the "natural" rate of exit from income assistance based on rates of exit for the Control Group during the first six months following intervention completion.
As an example, for Taking Charge! we multiplied the original average duration by the exit rate for the Control Group at 4.4 months (15.4 percent),71 resulting in an adjusted intervention duration of 3.7 months (4.4*(1 minus 0.154)). Similarly, we calculated the exit rate for the Comparison Group at 2.7 months (12.1 percent), resulting in an adjusted intervention duration of 2.4 months (2.7*(1 minus 0.121)). These times represent an "expected increase in time on assistance" as a result of participation in training.
Finally, one can argue that the incremental cost of training is the difference between the adjusted time on assistance and the actual time on assistance. For Taking Charge! this would then be 0.7 months (4.4 minus 3.7) and for the Comparison Group it is 0.3 months (2.7 minus 2.4).
We have included ranges in the benefit-cost calculations to reflect these differing perspectives. Since the same ideas are applied to Taking Charge! and the Comparison Group (represented by Employment Connections) varying the calculation approaches affects the benefit-cost ratios of both programs in the same general magnitude and direction. Once again, we believe that income assistance costs while on training represent an extra cost associated with training. Clients in training are not available for work (and usually do not look for a job) until their training is completed.
The average adjusted duration of training and average payments by program are shown in Table 78 with the resulting costs of income assistance associated with training.
| TABLE 78 Average training time (months) and average income assistance payments (Item A in Table 76) |
||||||||
| Group | Adjusted average duration of training (months) | Average monthly income assistance during training | Total income assistance received during training (Adjusted duration of training multiplied by income assistance during training) | Average monthly income assistance after training | ||||
| a | b | c | a | b | c | d | ||
| Treatment (Taking Charge!) | 4.4 | 3.7 | 0.7 | $904 | $3,978 | $3,345 | $633 | $682 |
| Comparison | 2.7 | 2.4 | 0.3 | $891 | $2,406 | $2,138 | $267 | $650 |
| Control* | 0.0 | 0.0 | 0.0 | $897 | $0 | $0 | $0 | $693 |
| * To support the statistical analysis we defined the intervention for the Controls as 1 month, but in reality the intervention period for these income assistance participants is 0. | ||||||||
Note that we have bolded the data to be included in the benefit-cost calculation (Table 81).
Item B (training allowances) from the benefit-cost framework in Table 76 are modest and summarized in Table 79. Any differences may partly be due to these costs being embedded in the tuition and materials paid by the Program to service providers. As we noted previously in this section, the existence of training costs for the Control Group is curious, as these income assistance clients should not have taken any interventions at all. It is possible that these costs are associated with short workshops and other programs.
The cost of training (item C from Table 76) is available from Table 72.
| TABLE 79 Training allowances (Item B in Table 76) |
|
| Group | Average training allowance during intervention |
| Treatment (Taking Charge!) | $6 |
| Employment Connections | $12 |
| Comparison | $55 |
| Control | $2 |
| Source: Program and SAMIN Extract. | |
Item C, (Cost of training), is the average cost per client, as presented in Table 72. Note we excluded the training allowances paid to avoid double counting.
On the benefits side, item E from the framework is calculated from the regression analysis, where the reduction in benefits due to earning increases by $68 (Taking Charge!), $71 (Comparison), and $26 (Control). Average post-intervention monthly
income rose by $114 (Taking Charge!), $126 (Comparison), and $37 (Control). Note that these are the employment wages while on assistance and do not reflect the incomes clients may earn after coming off assistance (Table 80).72
The average time for a Control Group member to come off assistance is 23 months; for Taking Charge! it is 20; and, for the Comparison Group, it is 16. For the 20 months Taking Charge! clients are expected to be on assistance, government can expect to pay $1,360 less by virtue of the benefit reductions due to earnings over this 20 months (20 times $68). For the Comparison Group government pays $1,136 less and to the Control Group this is $598. (See column E, Table 81.)
Taking Charge! reduces the time on assistance by three months over the Control Group, for an avoided income assistance cost $2,046 (3 times $682) and the avoided costs for the Comparison Group is $4,550 (7 times $650). This supports column F (Reduced time on income assistance) in the benefit cost framework (Table 81).
The tax benefits to government are simply the employment income earned while clients are on assistance times the marginal rate of taxation which we have taken to be 9 percent for this group. Increased incomes for Taking Charge! are $2,280, and the expected taxes from this are $205. For the Comparison Group and the Control Group the tax revenues are $181 and $77 which gives us column G in the benefit cost framework (Table 81).
Now it is possible to combine this information to produce an overall estimate of the benefit-cost ratio for each of the Programs relative to the Control Group.73
The benefit cost ratio for the Control Group is very large, but this is because nothing is spent and a certain percentage of clients returns to work. This type of calculation shows the limits of cost-effectiveness analysis in examining options such as doing nothing. It ignores the future costs of clients coming on income assistance remaining indefinitely on it.
The benefit-cost ratio shows a range from 0.51 to 0.96 for Taking Charge!. In our view, the higher ratio, obtained by calculating the incremental time on assistance as the difference between observed and expected time on assistance is an over-estimate. The lower values ratios probably underestimate the benefit-cost ratio somewhat. The benefit-cost ratios for both Taking Charge! and the Comparison Group are probably closer to 0.6 and 2.0 for Taking Charge! and the Comparison Group, respectively.
The basic cost-effectiveness question is whether program expenditures have value. The ratios are simple to interpret. For every dollar invested in Taking Charge!, the government receives a return of between 51¢ and 96¢.74 For the Comparison Group, this return is between $1.54 and $3.52. Some points emerge from this analysis.
It is always important to realize that some of the Control Group clients return to work, albeit usually in low-wage and insecure occupations. Any intervention must reference this as a baseline.
We have also stressed another point. Limited follow-up is available for the programs. Taking Charge! has more extensive interventions than the other programs. It also offers a wider range of collateral support services and has clients with slightly lower levels of employability than the Comparison Group. The outcomes for the interventions it offers are probably longer term than those offered by the Comparison Group.
That being said, Taking Charge! has both high direct costs per client and has longer periods of benefits paid while in training. These are not offset by reduced income assistance payments, shorter periods over which assistance is paid, and higher taxes. On the other hand, the Comparison Group interventions do offset their costs and have positive benefit-cost ratios. What characterizes the results for Taking Charge! are its high project costs, longer interventions during which income assistance is paid, and a slower re-entry into work. Again, this is the result of offering a wide range of services to a broad group of clients, with many requiring sustained support and showing no employment results during the relatively short follow-up period.
One final point needs to be stressed. The Comparison Group results are heavily influenced by Employment Connections and Opportunities for Employment. These programs focus on job-ready clients who should be expected to require modest interventions to produce good results in a growing economy. Because of the structure of the analysis, we have not been able to compare the net cost effectiveness and benefit-cost ratios for the individual programs that comprise the Comparison Group. It is possible that Community Partnerships and Pathways would show different levels of cost-effectiveness than the Comparison Group as a whole.
This section shows that Taking Charge! is a project that may barely return its investment. Compared to doing nothing (Control Group) and the Comparison Group, especially Employment Connections, it has a low benefit-cost ratio.
Three important facts qualify this conclusion:
|
In this section of the report, we summarize the main findings of the evaluation.
Single parents on income assistance face substantial barriers to economic independence. We can examine economic independence from two perspectives:
Most human resource policies address the supply side, under the assumption that changing the qualifications of income assistance clients and helping them manage personal/family problems will increase their employability. Dealing with supply-side issues is a necessary but not a sufficient condition for increasing the rate of employment among income assistance clients. One of the main findings from this study is that job-ready clients can enter the workforce provided the economy is growing. The fact that up to 25 percent in the Control Group leave income assistance with the assumption they find work within nine months, shows a "natural" process of entry and withdrawal from assistance. Some clients are temporarily on income assistance for a single spell, others cycle in and out, and still others are permanent recipients.
Even the most effective program we studied (Employment Connections) shows a long period of time before the average client comes off income assistance (14 months). Job-ready clients gain employment quickly, but the remaining clients come off assistance slowly. Quick fixes are not available for this group.
With little doubt, the success rates of the last two years will decline. The early successes of labour market interventions are unlikely to be continued as the job-ready pool of income assistance clients diminishes. Should economic growth slow, income assistance clients who have found work may be laid off, and those currently receiving assistance may enter the job market less quickly.
Income assistance programs include clients who have recently come into the program as a result of temporary adversity and those who have received welfare for an extended period. In the impact analysis we found that the total time a client had been on welfare increased the monthly post-intervention income assistance payments and the length of time he or she remained on assistance after leaving a program. This indicates how a more elapsed time on assistance creates greater barriers to employment and training.
The costs of intervention are significant. Including the income assistance payments while clients are receiving training, results in the Program costs ranging between $2,500 and $8,500 per client. Programs that offer collateral supports, such as day-care, typically have even higher costs.
Day care is essential to encouraging single-parent clients on assistance to enter training programs. The day-care services offered by Taking Charge! are important for clients using the computer lab and taking personal development courses at its offices. However, most client training occurs at service providers that often do not have day care associated with the service. To increase access for their clients, programs need to offer more on-site services, such as day-care, throughout the day and on weekends.
All the programs we reviewed encouraged some clients to find employment. Average increases in post-intervention wages ranged from $37 per month for the Control Group to $114 for Taking Charge!, and $126 for Comparison Group clients. Training programs also increased the number of hours worked above the modest increase found for the Control Group.
Correspondingly, the training programs also reduced the time on assistance, increased benefit reductions resulting from employment, and increased the duration of employment. The reduction in benefits paid to income assistance clients ranged from $258 for Taking Charge! clients, to $208 for income assistance Comparison Group clients, to $90 for the Control Group. The reduction in average time on assistance after the intervention over the 23 months for the Control Group, was 20 months for Taking Charge! and 16 months for the Comparison Group.
The interventions we studied have with little doubt had a positive effect on the employability of income assistance clients. Typically, the results for Employment Connections are the most favourable in terms of duration on employment and numbers of clients employed after a specific period of time.
These results offer the following insights into training interventions:
Taking Charge! is a non-profit organization that prides itself on its separateness from government. This is claimed to offer greater responsiveness to clients, the freedom to design programs tailored to income assistance recipient needs, and the ability to attract clients who may be wary of government programs.
The last claim is probably true, but suggestions that Taking Charge! offers more flexibility than government programs are overdrawn. Under the new business planning/performance measurement processes and the devolution of federal training programs, provincial managers have increased authority to design responses for specific clients. Government programs can now respond quickly to client needs and enter into individualized contracts to offer service.75
The federal and provincial governments created Taking Charge!, with considerable publicity, and realigned the funding to training services. This disturbed the existing funding arrangements and ways of doing business. The initial reaction of many community groups, training service providers, and some government departments was a degree of wariness. At the same time, the Board at Taking Charge! set an independent course and purposely designed its programming to run separately from existing services.
This separation has dissipated. Taking Charge! now cooperates more closely with provincial departments and funds training programs offered by both for-profit and nonprofit organizations. At the same time, the initial isolation of Taking Charge! from the existing training providers is unfortunate. One of the reasons Taking Charge! has had only a modest impact on the client group is the delay in developing training services that were integrated with existing programming.
Taking Charge! also became ensnared between two competing objectives. As a Strategic Initiative, a central goal for Taking Charge! was to discover innovative training for single parents on income assistance. At the same time, as a well-funded program created partly by redirecting funding from other programs, Taking Charge! decided it had to meet objectives that would serve 900 clients a year and place 500 into employment. It satisfied the latter goal by becoming a broker and funder of training programs in a conventional sense. It set aside the process of designing and assessing innovative training programs. In this vein, it has not developed an effective tracking process, used databases and information systems to evaluate alternative programs, or experimented with training. Most of the resources have been directed to running conventional contract training, with special emphasis on collateral services for single parents.
Some of the problems encountered by Taking Charge! can be attributed to circumstances outside its control:
A partnership is an association of two independent entities that have mutual goals, that commit joint resources to meet those goals, and that share risk. Taking Charge! staff and management often speak of partnerships, when in fact very few of their relationships are partnerships. The most important relationships are contractual between Taking Charge! and service providers.
Taking Charge! has weak relationships with the traditional nonprofit community. Local community groups told us that Taking Charge! does not figure in their service provision and has not developed partnerships to deliver services to their clients.
Taking Charge! has been unable to develop relationships with the private sector. In part this may be because the Board, management, and staff do not come from a business culture. Quite reasonably, staff and management are client-oriented, but this probably impedes their ability to "connect" with the business community. We found many staff to have unrealistic expectations about the ability of business to hire program graduates in any quantity. The proposed business council has potential for increasing this relationship, but support from government in approaching potential members of this council or even making Board appointments would accelerate this considerably.
Once again, Taking Charge! was impelled to meet quotas for client service and employment. Once the Board made this decision, Taking Charge! became a training-service broker and changed its relationship with the community. As a funder, it found it difficult to develop partnerships with a community that saw it as a source of money.
It is difficult to claim that Taking Charge! offers the taxpayer a return. In relation to the Comparison Group, Taking Charge! is more expensive and less effective. However, this conclusion is based on short follow-up periods and the success that Employment Connections has had by training job-ready clients in a growing economy.
Whether the programs that concentrate on increasing the education of income assistance clients and working on motivation have a long-term payoff will not be apparent for two or three years.
Table 82 responds to each issue and question.
| Table 82 Evaluation Questions |
|
| Relevance/ rationale | |
| 1. What is the legislative base for Taking Charge! - both levels of government? | Taking Charge! is based on agreements signed between Canada and Manitoba and an agreement between Manitoba and Taking Charge!. |
2. Has Taking Charge! met the criteria
established for Strategic Initiatives:
|
As designed, Taking Charge! meets the
requirements of the Strategic Initiatives (SI) in all respects. As executed, important gaps exist between how Taking Charge! operates and the fulfilment of SI objectives. Taking Charge! has not led the development of innovative services, choosing instead to rely on proposals offered by training service providers. Although it is starting to subject service provider's proposals to more scrutiny, Taking Charge! does not set the training agenda. The database created by Taking Charge! can support some evaluation, and while its completeness has improved over the formative evaluation, Taking Charge! management appears not to use the information to plan service delivery. Employment facilitators primarily use it to match client service with provider programs. Taking Charge! has provided a basis for evaluating social reform and assessing the relative merits of different programs and interventions. However, this evaluation has depended on the existence of SAMIN, a provincial information system that tracks Income Assistance recipients. The CAMS database created and maintained by Taking Charge! has limited usefulness for assessing outcomes or evaluating training options. |
| 3. How many target-group members are likely to be in need of Taking Charge!? | Based on current income assistance data, potential caseloads are about 7,300 single parents each month. Based on current caseloads, Taking Charge! serves about 1,200 clients per year or about 100 clients per month. This represents about 1.5 percent of the client potential. |
| 4. To what extent does Taking Charge! reach the intended target group? Do participants represent the target group? If not, for what reasons do discrepancies occur? | Taking Charge!'s clientele represents the target group exactly. All clients are single parents income assistance. To increase its market penetration, Taking Charge! has relied on service providers to recruit, assess, and train clients. E& IA counselors are working more closely with Taking Charge! than previously, especially since the creation of an E& IA office on site. Taking Charge! also uses advertising to raise awareness among the target group. All of these approaches give Taking Charge! a high profile among the target client group. |
| 5. Are the services/interventions provided responsive to, and consistent with participant needs? | Taking Charge! has relied on service
providers to propose interventions. While Taking Charge!
does present ideas to service providers and develop pilot projects, it has
yet to define client needs and call
upon the service providers to meet those needs. Taking Charge! is
rejecting more proposals than in the first years, especially those that do
not lead to employment. In this sense,
it is assuming more control over the training agenda. Taking Charge! has also worked with some service providers to amend existing courses to meet client needs. In areas such as network administration and technical trade courses, service providers have adjusted the pace of the training to increase the likelihood of success for Taking Charge! clients. Training service providers complete 37 percent of assessment and independence planning. They have an obvious incentive to assess clients to fill their available training spaces, as opposed to creating new programs for different clients. Without Taking Charge! doing the assessment and independence planning, employment facilitators are in a weak position to place clients into appropriate programs or to design new training to meet client needs. A service provider will typically want to offer an existing program, rather than assume the costs and risks of developing a new course. |
| 6. What similar services are being provided by other existing | Taking Charge! funds a range of services
similar to projects offered by other programs? provincial
programs. The job search training and job specific training/employment
programs of Employment Connections and Taking
Charge! are similar in structure and approach.
Taking Jobs!, the employment placement service in Taking Charge!, is
essentially the same service provided by
Employment Connections. The work placement process
and the basic education programs are common to Community Partnerships,
Pathways to Success, and Taking Charge!, but
directed to different clients. The Departments of Education and Training and Family Services are now meeting with Taking Charge! more frequently to minimize duplication. However, the fact remains, Taking Charge! offers services that are available in other programs. |
| 7. What gaps in the ongoing/existing array of services/programs are being filled by Taking Charge!? | Taking Charge! offers a supportive environment for single-parents on assistance. As such it is a "welcoming" first step to independence for clients who have low expectations of themselves, weak self-confidence and have spent much of their adult lives in the "system." Its being apart from E& IA and other government agencies may also encourage certain clients to enrol. Because service provider value has not been systematically assessed by Taking Charge! (and other provincial programs), service gaps are difficult to discern. Taking Charge! has not defined a spectrum of services needed to move clients along the employability spectrum. For that matter, the province has also not articulated such a concept (see Section 10). Therefore, it is not reasonable to expect Taking Charge! to be in a position to identify service gaps. In fact, Taking Charge! has adopted the approach of creating new services rapidly to meet possible needs. The Taking Jobs! program is an example. |
| Design and delivery | |
| 1. What are the design and delivery features of the program (i. e., components, activities, and relationships between components and activities)? | Taking Charge! has the flexibility to respond to client needs. It can fund ideas quickly and well, thereby creating a good "test bed" for innovative approaches to employment training for this client group. At the same time, Taking Charge! has not used this potential fully. It relies on service providers to propose courses and does not set the training agenda. Much of the substantive job preparation occurs outside Taking Charge! by service providers. In contrast, the original conception saw Taking Charge! as the primary coordinator of client assessment, training design, and intervention coordinator for programs directed to the target group. Taking Charge! is now one of several interventions funded by the province and has become a funder and a broker of training services. |
| 2. What are the strengths and weaknesses of the program design? | Strengths:
|
| 3. What are the strengths and weaknesses of the pilot-project organizational structure? Are the roles and responsibilities of the various partners and service providers (e. g., delivery agent, management committee, Board members, staff, volunteers) clearly enunciated? | Too much time passed in the initial planning
stages. In part, this was because the creation
of Taking Charge! with significant funding (some diverted from other
programs) prompted rivalry among
various community organizations as well as between Taking Charge!
and some government departments. This time was never made up. Taking Charge! essentially abandoned the pilot-program concept when it decided to meet the original expectations for service delivery and employment success. At this point, it ceded some of its mandate to service providers who were contracted to recruit, assess, and train clients. Too much is left to the service provider organization in the form of recruitment, individualized job planning, assessment, intervention delivery, evaluation, and follow-up. Taking Charge! has not created the evaluation and information structure needed to assess the value offered by service providers. Although employment facilitators are now reviewing proposals more critically than in the early years, independent verification of outcomes is not routine. This is true of all training interventions funded by the Province. Reports offered by service providers need independent verification to confirm outcomes. The strength of a pilot structure is that it tests new approaches. However, most staff are not seconded and quickly come to view the Program as a permanent entity to be funded and defended against all criticism. Understandably, Taking Charge! management decided to meet the employment expectations of the original agreement as opposed to testing innovative training approaches. This is easy to understand in light of the criticism directed at the Program in the first two years that it was slow in starting and funded at the expense of other community organizations and labour market interventions. |
| 4. How are individualized plans with participants developed and how do these plans meet their needs? | Individualized plans are created by Taking Charge! employment facilitators and by service providers. Variation clearly exists in how well these plans are completed and to what extent they meet client needs. The plans create employability assessments that categorize income assistance clients as Level 1, 2, or 3. These assessments are completed by many individuals and are unreliable for placing clients in training. Service providers offered quite sharp criticism of this aspect of Taking Charge!. |
| 5. Does Taking Charge! provide sufficient and appropriate resources (human, financial, physical) to participants, service providers, and employers? | Participants are satisfied with the
counseling process and the supports such as day care,
and the other services offered by Taking Charge!. For the most part, service providers find Taking Charge! supportive. Taking Charge! can fund service providers well. We found training courses as high as $12,000 per graduate and as low as a few hundred dollars. Contact with employers is low outside work placement programs. Employers are generally pleased with these placements, but because of constraints on this survey, we could not compare their perceptions of Taking Charge! with other interventions. |
| 6. To what extent did participants discontinue before their anticipated completion dates? What were the main reasons for discontinuation? | Taking Charge! staff report that training
participants quit programs because of personal
and family barriers. Some clients have unrealistic expectations of
themselves and the training. For other
clients, the idea of training and employment is novel and they
need to adjust to a new lifestyle. Finally, timing is critical. Reports
from clients (follow-up survey)
indicate that training must occur when a client is both willing and
able to take advantage of the intervention.
The quit rate varies by type of intervention, the
job readiness of clients and the nature of personal and family barriers
that impede participation. No one we
interviewed was able to offer a credible estimate of the percentage
of those who quit programs. About 14 percent of Taking Charge! participants
reported leaving the program prior to its conclusion. This compares with
9 percent for Employment Connections and 24
percent for Opportunities for Employment. Unfortunately, our understanding of why clients leave is based on interviews. Taking Charge! records the disposition of clients (completed, terminated, withdrew) on its database with discussion notes, but these are not complete or coded to support systematic reporting. Contract files report graduation rates from 75 percent to 95 percent. A few service providers have had contracts terminated or not renewed because of unacceptable completion rates. The CAMS database is used to record the outcome of training courses, but we found little evidence that Taking Charge! has used this information systematically. Taking Charge! is certainly assessing service proposals more completely now compared with a year ago, and counselors review performance to identify contracts that have high drop-out rates. |
7. To what extent is the community involved
in service delivery and development? What are the linkages, how have they
been developed, and how successfully have they been developed?
|
When government created Taking Charge!, it
created a major change in the funding environment
for training. Community organizations that had received funding from
government, initially viewed Taking Charge! as
a competitor. This changed as Taking Charge!
announced it would accept proposals for funding. Many community-based
organizations then became contractors offering
service for a fee. The service providers now
include non-profit organizations, educational institutions and private
vocational schools/contractors. The
relationship that Taking Charge! has with many community-based
organizations is contractual and symbiotic. The service providers deliver
the programming and Taking Charge!
funds them. In an important sense the voluntary sector and service providers have merged for Taking Charge! At the same time, Taking Charge! has not made many in roads into the mainstream voluntary sector that obtains revenues from traditional sources (private donations). Based on key informant interviews, it is also apparent that its relationships with local organizations are tenuous. Aside from the work placement programs, Taking Charge! has not developed strong relationships with employers. One barrier is that many employers need workers with a grade 12 education, and often Taking Charge! clients do not have this. A GED is equivalent to grade 12 for employers in manufacturing and technology areas. The training literature also strongly confirms the need for a grade 12 education to compete in the "new" economy. |
8. a. What tracking/monitoring mechanisms
have been put in place to collect information on participants and
interventions?
|
Taking Charge! makes follow-up calls, using
volunteers (its own clients) to determine whether
clients need further assistance. Follow-ups to determine the outcomes of
training and to verify employment, create a
difficult problem for the Program. Part of its presentation
is that Taking Charge! operates separately from government. A follow-up
campaign to verify current employment status
resembles the investigation process to eliminate
income assistance fraud. This would impede Taking Charge! in presenting a
"welcoming" environment to clients. Follow-up is integrated with the volunteer program within Taking Charge! and is used to give clients work experience and to improve their phone skills. Taking Charge! also relies on final reports submitted by service providers, often in conjunction with an application for project renewal or the submission of a new project proposal. No third-party verification exists on project outcomes, except for the income assistance client follow-up survey completed as part of the Phase 2 evaluation. The province is presently not performing a follow-up on all training programs although follow-up is maintained on some, such as Opportunities for Employment. For the evaluation, we defined Treatment, Comparison, and Control Groups (see Section 6). |
| 9. Have any operational/legislative/
regulatory constraints been identified that impinge on the ability of the
project (or single parents) to achieve objectives? Are the project design
features (i. e., operational guidelines that define eligibility criteria,
funding limits, etc.) consistent with
the stated objectives of the project?
|
Few operational, regulatory, or legislative
constraints exist in the program. If Taking Charge!
were to be more closely associated with government, it may lose an
important feature that makes it
attractive to clients. Also the follow-up process cannot be investigatory in nature, since that may discourage some clients.
|
| Project Success - Individual Outcomes | |
| 1. To what extent has the project prepared
new participants for achieving self-sufficiency? a. Increased their motivation and self-esteem b. Helped them develop career action plans c. Upgraded their educational skills d. Provided them with occupational skills e. Provided them with pre-employment training f. Provided them with work experience g. Provided them with self-employment/business skills h. Provided them with mentors/role models |
The follow-up survey of income assistance
clients showed that Taking Charge! clients, as
well as clients in other interventions, are satisfied with their programs.
Most clients reported that the programs
in which they participated: increased their motivation
and self-esteem, helped them develop career action plans, and upgraded
their educational skills. It also provided
them with occupational skills, pre-employment training,
work experience, self-employment/business skills, and mentors/role
models.
|
| 2. What is the net impact on the
participants' employability, by type of program/intervention and for
completers/non-completers? a. Improvement in labour market attachment b. Improvement in employment earnings c. Improvement in quality of jobs secured |
Post-intervention employability is not
evaluated by any program or E& IA and we cannot directly
assess employability. In terms of actual employment, 26 percent of the
Control Group have left income
assistance after 9 months; presumably most have found work. About
35 percent of Taking Charge! clients have found work after 9 months
compared to 42 percent of the
Comparison Group. Typical post-intervention earning increases are $114/month for Taking Charge!, $126 for the Comparison Group, and $37 for the Control Group. Most clients who find jobs obtain employment in the service sector, retail, and light manufacturing at wages between $6.50 and $8.00 per hour. From the client survey we found that most who were employed after training were satisfied with the jobs they had obtained. Little variation in satisfaction occurred across the programs. |
| 3. What is the net impact on participants'
self-sufficiency, by type of program/intervention and for completers/
non-completers? a. Change in number and duration of spells on income
assistance? b. Change in amount of income assistance and Employment Insurance received c. Type and amount of income assistance and Employment Insurance received d. Reason for continued dependency on assistance. |
Very few clients in the database received
employment insurance during the study period
and we have not tracked this program among the income assistance clients.
Separating completers and non-completers is also hard for all programs.
This would result in very small sample sizes
and we did not make this distinction. The
data on non-completion are unreliable and based on service provider
reports and follow-up by the programs. Because the effective follow-up ranged from 20 months (for those who completed their intervention in June 1996) to 9 months for those who completed in September 1997, we could not standardize the number of spells or their duration. This has not been tracked. Post-intervention duration on income assistance ranged from an average of 23 months or the Control Group, 20 months for Taking Charge! and 16 months for the Comparison Group. We did not classify the components of income assistance received, since that is not recorded by SAMIN. Continued dependency on income assistance reflects the presence of clients whose educational levels are too low to complete job-oriented training and to compete in the labour market. Unless they have family or personal problems, most job-ready clients (those with grade 12 and some work experience) find work and come off assistance. |
| 4. Has the project brought about any changes in participant's home/family life (e. g., family interrelationships, health status, involvement in other community activities, use of other community services, types of recreation engaged to, etc.)? | The follow-up survey and case studies revealed that the interventions have a beneficial effect on home and family life. We limited the follow-up surveys to specific issues related to clients' abilities to participate in the labour market and further education. Issues such as health status, use of community services, and recreation were set aside by agreement with the Evaluation Steering Committee to focus on the specific labour market outcomes of Taking Charge! |
| 5. To what extent has the project assisted
participants to achieve self-sufficiency? a. What project activities or interventions were most effective and for which type of clients? b. For completers/non-completers? c. For what reasons do some participants remain unemployed and on income support after the program? d. Did Taking Charge! motivate participants to go on to further training? |
Training interventions associated with
immediate job placement and job search have been
the most effective in placing clients of Taking Charge! and the Comparison
Group. Those who are Aboriginal and
have children under 6 years of age face the most significant
barriers to labour market participation. Education appears not to be
related to post-intervention wages,
but work expectations are. Definitions of self-sufficiency varied among key informants. Success ranges from overcoming abuse, to getting a job, to increasing job earnings, to reduced dependency (from 100 percent income assistance to 20 percent with 80 percent employment income). |
| 6. To what extent did the project improve the long-term prospects of participants and their families escaping the cycle of poverty? | With a 9 month follow-up, long-term prospects are impossible to assess. The evaluation literature clearly shows that a 3-year follow-up is needed to determine changes to long-term prospects. Under the argument that any education, training, and job experience is beneficial, the training programs we studied have probably enhanced long-term job prospects. |
| Project Success - Delivery System Outcomes | |
| 1. To what extent has the project succeeded in developing successful partnerships among the various levels of government, employers, and community groups? To what extent has it succeeded in integrating federal/provincial dual services? | If we view partnerships as an agreement
between two entities to achieve a mutual goal and
with each sharing risk to attain that goal, Taking Charge! has not
developed effective partnerships. Those
community organizations that train for a fee are not partners,
they are contractors performing a service. It has not formed partnerships
with the traditional non-profit
community and some local community groups view Taking Charge!
as detached from themselves. Relationships with employers are weak,
although a business advisory council is being
developed. The relationship with the provincial government has improved significantly and is serving to integrate Taking Charge! services with E& IA counseling and Education and Training. Taking Charge! has had little involvement in the devolution of training and labour market services from the federal to the provincial government. It has improved its relationship with the Department of Education and Training. |
| 2. How satisfied are participants with various aspects of the project (e. g., application and selection, services provided, etc.)? | Participants are very satisfied with Taking Charge! and the other interventions in the Comparison Group. |
| 3. To what extent has the project succeeded in removing disincentives to employment and training (e. g., changing income assistance regulations, etc.)? Or alternatively, in increasing incentives (e. g., earned income supplements, day care, nurturing, health benefits, etc.)? | Taking Charge! has had no active role in the Making Welfare Work program. However, because it offers a welcoming and friendly face, it can encourage clients to seek training. The existence of collateral services such as day care is very important to supporting the harder "edge" of welfare reform, such as the imposition of a work expectation. Therefore, a program such as Taking Charge! can be a valuable adjunct to welfare reform, especially for income assistance clients with poor education and job experience. |
| 4. To what extent did participants displace employees already on an employer's workforce? Were these jobs permanent? | With the barriers to the employers survey (see Appendix 3) we were unable to address this issue directly. Given that Manitoba has had a high rate of job creation (7,000 new jobs in June 1998), it is unlikely that Taking Charge! and its approximately 450 graduates who become employed in a year, displace full-time workers. |
| Project Success - Cost-effectiveness | |
| 1. What are the costs of individual components comprising the program? | Taking Charge! spends 60 percent of its annual $5 million budget on training service contracts. Another 19 percent is spent on indirect costs (administration, rent, etc.), and 21 percent is spent on direct client services (computer lab, day care, etc.). |
| 2. What is the per diem cost per participant? Per component? | Taking Charge! cost $3,112 per client, while Employment Connections (about 50 percent for the Comparison Group) costs $1,341. |
| 3. What is the cost per participant, per program completion, per employed participant? | Taking Charge! costs $6,905 per employed client (judged after 9 months), compared with $2,439 for Employment Connections and $4,000 for Opportunities for Employment. |
| 4. What is the net cost or saving due to the difference in subsequent level of income assistance and EI dependency attributable to program participation? | On average, Taking Charge! clients have lower income assistance payments of $258 per month, compared with $208 for the Comparison Group and $90 for the Control Group. A certain number of Control Group clients get jobs without any intervention, and this leads to their income assistance payments being reduced. |
| 5. What is the benefit cost ratio and payback period? | The benefit-cost ratio captures all benefits (net income assistance savings, reduced time on assistance and taxes on earnings ) and all costs (costs of income assistance while on training, cost of the training and special assistance for training). Despite the fact that Taking Charge! clients typically reduce income assistance payments by government more than the Comparison Group (see above), because its training is longer and more costly, it has a benefit-cost ratio between 0.51 and 0.96 (and no computable payback period). The Comparison Group has a benefit-cost ratio of 1.54. Assuming that all benefits are generated within the 16 months average duration on assistance, the payback for the Comparison Group cost is about 9 months. Once again, the Comparison Group programs tend to have a higher proportion of job- ready clients than Taking Charge! which accounts for their higher measured returns. |
| 6. Is the Taking Charge! model a cost- effective way of achieving project objectives? Are there more cost effective methods of achieving the same objectives? How do results compare with those of other programs with similar objectives (e. g., brokered services, collocation)? | In terms of meeting employment for job- ready income assistance clients, Taking Charge! represents poor value compared with programs such as Employment Connections and Opportunities for Employment. Taking Charge! has evolved into a broker of training services, but funds relatively high cost interventions. This conclusion must be qualified short post-intervention by the observation period and the fact that Taking Charge! has a much broader spectrum of clients than those in the Comparison Group. |
| 7. What lessons can be learned from Taking Charge! on interventions to assist the target group? To what extent does it contribute to the development of a policy framework for social security reform? Does Taking Charge! lead to a more efficient delivery of services? To what extent can this experience be extended to/adopted by other jurisdictions? | (See Section 10) |
|
The main findings of this study are clear:
Evidence from the Taking Charge! evaluation shows that short-term training interventions offered to "job-ready" clients can produce positive short-term outcomes in higher wages and departure from income assistance. Our qualification that these results may not be long-term is based on recent evaluations of training programs.76
The evidence from three decades of research on labour market interventions is clear. Short-term training designed to get economically disadvantaged persons into jobs typically does not offer a sustained solution to economic independence. The results in Manitoba may be the coincidence of training delivered to job-ready clients and a rapidly growing economy.
We classify our recommendations into two broad areas:
With the province assuming responsibility for labour market interventions and training, this is an opportune time to recast policy, based on the following recommendations:
Training of income assistance clients should be based on the idea of an employability spectrum.
The idea of employability is abstract. The current assessment process culminates in the assignment of levels of employability and work expectations which serve as the foundation of this process. With the advent of a one-tier approach to income assistance, these assessments will probably be completed with greater consistency. The current assessment questionnaire can probably continue to serve as the basis of this assessment, but several changes are needed:
In other words the employment assessment process should be a dynamic one that traces the progress of clients toward self-reliance. Income assistance clients can be placed on a spectrum based on their employability assessment. Programming can be designed to meet the needs of clients along the spectrum.77
Training interventions need to be aligned with client needs.
Once income assistance clients are placed on the employability spectrum, counselors can assign them to specific interventions. The province should identify specific interventions and the six categories of training used by Taking Charge! and other programs are useful classifications (Table 32).
Implicit in the notion of assigning clients to interventions is the idea that expectations must vary by employability. It is expected that clients who have completed high school, have job experience, and have recently become income assistance clients should return to work quickly if unemployment remains low. These clients only need a brief intervention to help with a job search. Here the sanctions and benefit reductions associated with work expectations are part of the policy mix.
Clients with low education and no job experience will require sustained interventions. Others will be permanent income assistance clients. Some who are disabled or too old to become re-educated will likely not return to work, regardless of the amount of training or level of sanction.
Training providers need to specialize.
Employment Connections is effective because it specializes in job-ready clients. Taking Charge! attempts to meet the needs of a diverse client group through a series of contracts to external training providers. When compared with the performance of Employment Connections, it appears much less cost-effective. Such a comparison is unfair because it does not consider the extent to which educationally disadvantaged income assistance clients move along the employability spectrum when participating in programs such as Taking Charge!
In addition to assigning clients to specific interventions, specialization requires those training providers to serve specific client segments. Some training services should focus on raising the education of "Level 3" clients; others should work on placing clients that are job-ready.
A portfolio of training programs will offer a seamless continuum of services.
Once the province identifies discrete client needs, it can develop specific services. A service such as Employment Connections may serve as the final program, accepting job-ready income assistance clients that have recently come on the welfare rolls, as well as clients who have "graduated" from training and educational improvement programs. Alternatively, the province could contract with private placement companies to assist job ready clients in finding work.
Taking Charge! could be "reinvented" as a program specializing in clients with low education and having barriers to education/employment. The emphasis on a "welcoming" environment, personal development, volunteering, mentorship, and the collateral support services are particularly important to these clients.
An outcome follow-up process will allow government to manage the training programs.
Manitoba Measures requires all government departments and eventually third-party delivery agents to prepare business plans, develop performance measures, and report on plan fulfilment both internally and externally. The province will need an outcome follow-up process to track the cost-effectiveness of alternative interventions and the progress of clients toward economic independence.
Integrated Service Management and the one-tier initiative will result in better tracking of program inputs. In future, the province will have much better information on income assistance clients and the programming they receive. Nevertheless, a key omission is the measurement of outcomes. Without a systematic process of follow-up and outcome verification, the province will never be able to assess cost-effectiveness properly. This evaluation used the available information to the maximum, but we cannot make any inference about employment outcomes of those who disappear from the income assistance rolls.
Training providers such as Taking Charge! are in a poor position to complete such follow-up, because it conflicts with the non-bureaucratic style that is so integral to their service delivery model. Therefore, if follow-up is to be complete, as it must be, the province must undertake it as a core part of its accountability process under Manitoba Measures.
If a program such as Taking Charge! did not exist, it would have to be invented. Taking Charge! offers services to single parents with limited education and job experience. It needs several specific adjustments:
Taking Charge! should focus on clients with the greatest need.
Taking Charge! should focus on personal-development and educational services to help training-ready and multiple-barriered clients. Single parents with low levels of education and work experience can benefit most from the cafeteria-style training, personal-development programs at the head office, and day-care supports. The welcoming atmosphere and mentoring of clients who are making progress are very strong assets that Taking Charge! offers in encouraging income assistance clients to work.
The outcome of these programs is not employment but measurable improvement in employability (through the assessment process), and the increased ability and success clients have in pursuing additional/higher education plus trades training.
Taking Charge! should not offer work placement, job-search assistance, and other labour market skills training. By focusing on high needs clients and moving them to higher education and trades skills, Taking Charge! is alleviated of the responsibility to place clients into jobs. Taking Charge! has not had much success in connecting with private firms and creating work placements. Agencies that specialize in contacting employers and matching trained workers to vacancies do this best.
Board appointees should include some government representatives and exclude contractors.
Taking Charge! should remain an independent, non-profit organization, but with some Board appointments from key government departments and services (Family Services, Education and Training, Industry, Trade and Tourism).
This reconfiguration would increase communication between Taking Charge! and other education/training services, without compromising its ability to create partnerships and joint ventures with other organizations.
Taking Change! should focus its training contracts with a well-defined agenda.
One of the more important developments in the history of Taking Charge! was the expansion of contracting to community-based organizations and other training providers. This changed the relation between Taking Charge! and community groups, requiring management of large-scale contracts, and a level of monitoring that consumed significant resources. It also separated Taking Charge! from local community organizations.
Taking Charge! needs to define specific training goals to meet the needs of client segments. It then must decide whether to meet those needs through training its own staff or by contract. External service providers should be selected on the basis of competitive bids in response to a request for proposals. It is still possible to accept and review unsolicited proposals then Taking Charge! may choose to fund in whole or as a partnership.
Taking Charge! should use a tendering process that responds to the educational and training objectives of its clients. Contracts with service providers should be limited to courses that benefit the clientele. Typically these should be for educational upgrading and be offered by providers with a demonstrated capacity to offer these programs and certify that clients have reached levels to allow them to participate in further skills training.
Evaluating the outcomes of programs to enhance literacy and numeracy skills is more challenging than determining whether a client has found work. If Taking Charge! chooses to focus on high needs clients and offer basic education, it will need to develop measures to track the success of these interventions.
Taking charge ! needs to work is close, partnership with community organizations.
If Taking Charge! were to cease being a major source of contract funding for the training "industry" and non-profit organizations, it could participate in legitimate partnerships with local community organizations that address the needs of economically disadvantaged clients. The process of referral between Taking Charge! and these organizations would improve. Taking Charge! could offer services on location jointly with these agencies. Level 2 and 3 clients would then access the services of both Taking Charge! and the community organizations and probably move along the employability continuum faster.
An initial expectation was that this evaluation would be able to assess the cost-effectiveness of individual program components. For example, it is important to know whether literacy/numeracy training is more effective than to technical education, everything else being equal. Also desirable is the ability to compare the cost-effectiveness of the same intervention offered by different programs and different service providers.
To achieve this level of detail requires two important types of information not available to the evaluation:
It is essential that all programs and service providers classify interventions similarly and maintain detailed information on client attributes. Programs and service providers seek to change their training programs to make them attractive to both clients and funders and classifying their offerings can be challenging.
Further, outcomes need to be tracked in the same way for all participants in all programs. In this evaluation, employment outcomes were inferred from the income assistance database (i.e. SAMIN). Clients who disappeared from the database after the intervention clearly leave social assistance, but it is not possible to differentiate those who have become employed, moved, married, or are in jail.
This evaluation has clearly identified the cost-effectiveness of Taking Charge! relative to the Comparison Group (within the constraints noted above). To compare training outcomes at the intervention or service provider level requires significant additional investment in information systems that record the nature of the intervention and track participants after their involvement. This is a key lesson from this evaluation that could usefully be applied evaluating training programs in general.
Taking Charge! has been a worthwhile experiment. Its experience underscores the importance of specialization and of offering services to economically disadvantaged people. However, by offering a broad spectrum of services, it limits its ability to partner with community organizations. Reconfiguring Taking Charge! will require the province to redefine an overall education and training strategy for income assistance clients.
Taking Charge! should be repositioned and sufficiently funded to deliver education and training programming to those facing more extensive barriers to employment. As with all government training programs, Taking Charge! should only support those clients who are interested in coming off assistance and who do not have severe family and personal problems that impede success. Taking Charge! must remain an education and job-preparation service and not become a one-stop centre to serve all client needs.
The expected outcomes of such a revised program would not be employment, but clients who are able to take further trades and technical training to survive in the modern labour market. The culture of the organization, the background of the staff, and its collateral programming are ideal for supporting these higher-needs clients to start the path toward employment. "Graduates" of Taking Charge! would then move to take higher levels of training and participate in job-readiness programming.
|
|
| Last Modified: 2001-12-19 | Important Notices |
![[To Top]](/web/20060127002936im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}