Formative Evaluation of Manitoba's "Taking Charge!" - March 1998

Formative Evaluation of Manitoba's "Taking Charge!"

View Whole Report

• Introduction
• Project Description
• Project Objectives
• Special Features
• Evaluation Approach
• Key Findings
• Design and Delivery
• Participant Satisfaction
• Information System
• Community Partnerships
• Conclusions

This brief summarizes the findings of a formative evaluation of "Taking Charge!" (TC!), a Strategic Initiative funded jointly by the governments of Canada and of Manitoba. This formative evaluation is the initial phase in a planned two-phase evaluation approach. The evaluation was jointly managed by a committee with representation from the Federal Department of Human Resources Development, the Provincial Departments of Family Services and of Education and Training, and representatives from the Taking Charge! project.

The Strategic Initiatives Program, announced in the Federal Budget of February 1994, is a five-year, Federal-Provincial, cost-shared program designed to test innovative approaches to social security reform. Eligible pilot projects included those intended to improve job opportunities, help individuals adjust to labour market demands, and explore innovative means to effectively and efficiently provide necessary supports to promote economic self-sufficiency. Evaluation of funded projects should provide objective evidence on which approaches work best for specific target groups to enhance their employability and enable their participation in the labour market.

TC! is intended to enhance, through training and/or work experience, the employability of single parents on income assistance. The project has its legislative basis in Manitoba's Employment and Income Assistance Act. It operates as an independent, non-profit corporation with a Board of Directors appointed by the partners to the agreement.

The overall goal of TC! is to test and demonstrate an integrated model for offering services to single parents on income assistance in order to promote their self-sufficiency.

The objectives of the TC! pilot project, as set out in the Canada/Manitoba Contribution Agreement reflect the experimental aspects of Strategic Initiatives, and fall under three broad categories:

Service Delivery Objectives

• to lower the cost of support services by reducing duplication and increasing involvement of the private and voluntary sectors;
• to increase access to and responsiveness of services through federal, provincial, municipal, community and business partnerships;
• to test innovative service delivery mechanisms; and
• to increase awareness and involvement of communities to assist sole parents on income assistance.

Employment Objectives

• to increase employment activity and earning potential of participants;
• to provide necessary supports to reduce barriers to labour market participation;
• to increase labour force attachment among participants reducing their stays on income assistance; and
• to meet employers' needs for job-ready employees.

Social Security Reform Objectives

• to reduce income assistance costs by improving the long-term job prospects for income assistance recipients; and
• to evaluate innovative models to better identify new policy and program directions for social security reform.

The project was designed to respond to the unique identified needs of individual participants. The model embodied several innovative features including: a single location for service coordination; on-site, flexible day care; cafeteria-style training which clients could access according to their needs; and, a broad spectrum of activities ranging from general academic upgrading, parenting techniques, personal development and life management skills, to labour market training and work experience.

The Canada/Manitoba Agreement defines eligible participants as sole parents in receipt of income assistance. The cost for TC! of up to $26.2 million was to be shared by both levels of government. The project planned to assist approximately 900 eligible clients per year with the expectation that of these, 500 individuals would be placed in employment.

The formative evaluation was completed in February 1998 and covered the period of operations from April 1996 to September 1997. The focus was primarily on the design and implementation aspects of the project, and to examine the continuing relevance of the goals and objectives of the project. The evaluation documents program activity, the level of client satisfaction, and perceptions of others on how well the project is functioning to date. As well, the evaluation reviewed the status of program information systems, including the adequacy of data to support a subsequent summative evaluation (phase two) which will report on success outcomes and cost effectiveness.

The formative evaluation incorporated the following methodologies:

• document review including Board minutes, agreements and legislation, Business Plan, Employment and Income Assistance data, as well as statistics and contracts relating to service providers;
• review and analysis of summary statistics from the TC! data base;
• interviews with key informants including TC! Board, management and staff, service providers; and Federal and Provincial Government officials; and
• focus groups and written survey responses involving a cross-section of project participants.

Rationale and Relevance

The TC! project is successfully reaching the intended target group. All clients are single parents on income assistance; the majority are women between 18 and 40 years old, and close to 40 percent of clients are Aboriginal. The project serves about 75 clients per month as planned, which represents just over one percent of the eligible potential target group caseload (estimated around 7000). This indicates strong relevance and a continuing need for specialized services for this clientele.

As designed, TC! met the requirements of the Strategic Initiatives Program with respect to objectives and target groups. Because it operates as an independent agency, the project has sufficient flexibility to respond to client needs and can fund ideas quickly.

Although TC! was announced in September 1994, the complete range of TC! services was available to clients only as of April 1996. This was due to delays in appointing the Board, and due to the planning process required to create the organizational structures, and to develop policies and procedures to support project delivery.

Also, several differences were found between how TC! actually operates and how it was originally conceived.

Independence Planning and Assessment

One major difference is that, in practice, the independence planning model, a cornerstone of the project consisting of a sequence of service flows for assessing clients, creating job plans, and referring clients, did not work out as planned. Instead, in order to meet high volume performance targets, and in response to the job-ready clients, a process of "fast-tracking" evolved. Service providers under contract to TC! increasingly performed the functions of recruiting, screening, assessing and preparing job plans for clients prior to registration with TC! This inverted process compromised the potential for the project to fully explore its alternative one-stop service delivery, among other client support innovations. The unique distinction between TC! and other similar provincial programs disappeared as the project took on the role as mainly a funder of training programs delivered by other parties.

Employability Levels

The original concept of assigning clients to employability levels, (i.e. job-ready, training ready, multi-barriered, etc.), while sound in theory, was initially inconsistently defined and executed. The consensus among those interviewed for the formative evaluation was that client employability needs are perceived as becoming more challenging. If this perception is accurate, in future, comprehensive programming such as TC! may become more crucial in dealing with clients with multiple barriers to self-sufficiency.

The focus groups were generally positive about the TC! project. Participants supported the concept of the project and considered the day care arrangements, wage subsidies, and money for transportation to be important incentives to project participation. Overall, clients expressed satisfaction with training provided, particularly technical programs. Personal development activities were also viewed as a positive approach. Some clients, however, indicated that the quality of training depends on the particular service providers. Others noted the need for improvements in the assessment process and for better access to programs which more closely match individual needs.

Early on, the evaluation identified certain shortcomings in the project information system. TC! made some progress in improving the quality of information put into its database. However, further refinement and more accurate and complete data entry is required to:

• support the development of training plans for clients, and
• assess value for money from the services provided by external parties contracted by the TC! project.

TC! has developed strong partnerships with non-profit community organizations, private vocational trainers, and post-secondary institutions. Community partnership is expressed both through Board membership and through contracts between TC! and independent service providers. The evaluation respondents noted some ambivalence about the close relationship between the TC! Board of Directors and external service providers. While the excellence and professionalism of some trainers were widely acknowledged, concern was expressed about the appearance of a potential conflict of interest arising from this close relationship. Procedures need to be developed to deal with even the appearance of possible conflict. Respondents felt that one way to deal with this issue would be to vary Board membership by increased representation from business, industry associations, and others with no direct financial interest in contracts awarded.

At the time of the evaluation some progress was noted towards forging partnerships with the business sector. The appointment of a manager of business partnership was seen as a positive move towards greater focus on this critical aspect of the project.

The evaluation further identified insufficient service provider follow-up on clients as an important issue for TC! management. Specifically, follow-up and outcome tracking were inconsistently undertaken, and no independent audit of outcomes was provided. Evaluation follow-up surveys of participants in phase 2 will partially address these issues. If this information were to be provided regularly as a standard contract requirement, it could enhance the project's ability to more effectively identify needs, and to more efficiently plan and manage appropriate interventions.

Overall, the model for TC! was deemed to be innovative, comprehensive, and promising as an approach to enhance labour market opportunities for single parents on income assistance. However, the emphasis on client caseload and fast-tracking compromises the potential for adequately serving a broader spectrum of clients and for testing a variety of approaches to address labour market difficulties experienced by single parents on income assistance. More work needs to be done on the development of partnerships with business; on broadening the base of representation on the Board; and on the implementation of a more complete and credible management information system for monitoring participant outcomes and project impacts.