|
|
     |
||
 |
     |
 |
|
Section 4 examines design and delivery issues related to Taking Charge!. First we examine the design features of Taking Charge!, including its organizational structure, client training services and supports, and its philosophy - all aspects intended to make it unique among training programs in Manitoba. Second, we consider client participation and continuation in the Program, and factors that affect the likelihood of participation. This includes an assessment of personal barriers to employment and training and changes in job readiness. Third, we examine Taking Charge!'s information systems - their structure, use and adequacy. This information builds upon findings in the Phase 1 formative evaluation. Finally, we consider operational, legislative, and regulatory constraints to program design and delivery. We examine the structure of Taking Charge!'s Board, the leveling, and employability assessment process, Making Welfare Work and Taking Charge! within the income assistance system. 4.1 Unique Design Features of Taking Charge!Taking Charge! is aimed at single-parents on income assistance. Unique design features include:
4.1.1 Partnering with business and the non-governmental sectorManagement and members of the Board we interviewed emphasized that Taking Charge! offers the unique advantage over traditional government-based service in its ability to form partnerships with business and the non-government sector. With its independent Board as a non-profit corporation, Taking Charge! can enter into contractual relationships with many diverse organizations. Specifically, it can initiate joint ventures with business, the non-governmental sector, and government. 4.1.2 Collateral services and supports are important to the distinctiveness of Taking Charge!Taking Charge! offers a range of collateral services and supports designed to help single parents on income assistance begin training. The ability to tailor service to meet individual clients' needs was cited by management and the Board as a unique feature of the service. Taking Charge! offers additional support to clients while they are involved in the program. Examples include: free day care while the client is taking courses at Taking Charge!, financial assistance for day care when taking courses at a service provider, transportation (bus passes), educational materials, course tuition, and emergency cash to meet contingencies. All Taking Charge! clients receive their normal income assistance support while participating in the program. The importance of these supports was also affirmed by participants in the focus groups completed as part of Phase 1, and by respondents to the Follow-up Survey completed in the summative evaluation. By way of summary:
4.1.3 Clients are assigned their own employment facilitatorEach client is assigned an employment facilitator who helps him/her develop career plans and obtaining pre-employment training, skills development, and job placement. Employment facilitators reported that they routinely help clients with a wide range of training and employment-related problems. Management and Board consider this a key feature of Taking Charge!'s design and delivery model. 4.1.4 Taking Charge! takes a "client friendly" approach to income assistance training"Welcoming" is a word commonly used by management and staff of Taking Charge! to describe how they are different from government. In interviews, most of the management and staff described their approach and the overall atmosphere at Taking Charge! as "client friendly" or "client driven." They contrasted this with government services that they represented as less sensitive to clients and, at times, more "confrontational." 4.2 Client Participation and Continuation in the Program - Factors Affecting the Likelihood of ParticipationClient participation to graduation is an indicator of program effectiveness. Employment facilitators monitor the training service providers and client participation. Low participation (high client drop out) is taken by the Board of Taking Charge! as an important indicator in the decision not to renew a service provider's contract. We examined this issue in two ways:
4.2.1 Personal and family problems overwhelm some clientsWe probed what factors contribute to client drop-outs. Most key informants identified the following factors:
4.2.2 The job readiness of income assistance clients is decliningMeasures of job readiness (or, conversely, barriers to training and employment) include:
Table 25 summarizes the findings from the database we constructed for this study (see Section 6 for details). The number of children less than six years of age has dipped slightly, but both Taking Charge! clients and those within the Control Group are arriving for training with fewer prior jobs and less education. Most interesting is that the Control Group is a population that has spent an increasing percentage of its adult life on income assistance. 
Figures 2 to 6 illustrate the changes in the attributes of the SAMIN extract over the study period. The age of clients has not changed, but fewer are arriving with children less than six years of age. Education has remained stable, but the number of jobs clients have had prior to entering income assistance has declined. Most interesting is that the amount of time clients have spent on income assistance since their 18TH birthday has increased. 
This section provides an overview of the management information environment at Taking Charge! in general and the Client Abilities Management System (CAMS) database in particular. 4.3.1 CAMS has a sound structure but continues to have errors and omissionsBased on interviews with staff and on our own analysis, we conclude that CAMS has improved since the formative evaluation. Employment facilitators told us that they are using the system to identify training opportunities for clients and to match clients to proposed projects. At the same time, based on our most recent extract (July 21, 1998) we believe that the system could still be improved to better support program management and operations. CAMS is a relational database covering many, but not all, aspects of Taking Charge!'s interaction with its clients, service providers, and employers. The database was designed in-house by Taking Charge! staff and is maintained in Microsoft Access 97, a leading end-user database product. As a relational database, CAMS stores data into a series of tables, each containing information about a distinct object on which the program tracks information. Thus in the Taking Charge! database, client information is stored in one table, while information about the program's interaction with its clients is stored in another. Tables are related to each other through the use of common fields (called keys) that allow us to link, for example, a client to his/her interaction with the program. A full description of the database is found in the Phase 1 evaluation. CAMS consists of two components: a set of data tables, plus an application used to populate those tables with data.
Separating the data from the application allows a developer to make and distribute changes to forms and programming code without having to migrate the data from one revision of an application to another. In an environment such as Taking Charge!, where revisions to the database were common, this distinction is consistent with the principles of sound application development. 4.3.2 Information systems at Taking Charge! have three problemsWe have three general concerns about the information systems in Taking Charge!16
Key fields in CAMS have not been defined. As we cited in the formative evaluation, the most serious short-coming is that several important fields have not been controlled, but rather left as open text. Open text fields are extremely difficult to report. For example, the Training Program field in the Training Programs table captures the name of a client's previous training interventions. We found over 1,200 variants on training interventions for clients. We re-coded this information to support the evaluation. Data omissions and poor quality required us to use alternate sources of information. Inconsistent data quality within the Taking Charge! database has the following effects:
The main result of these data issues is weakened confidence in the database's ability to provide accurate statistics about Taking Charge! and its clients. As we have pointed out, Taking Charge!'s management has taken steps in recent months to improve the quality of information in the database. Based on the extract in July, it is clear that follow-up data are being entered. Nevertheless, there are many omissions on key fields such as client attributes, start and end dates of the intervention, and type of intervention. As we discuss elsewhere, Taking Charge! has not recorded outcome data with the precision and completeness needed for this summative evaluation (see Section 4.3.3 below). Therefore, for the measurement of individual impact, we set aside all information on client attributes, education, previous employment, and outcomes. Instead we relied on the SAMIN data and a supplementary data gathering activity for this information.17 This was also a problem for the Comparison Group. Information from Employment Connections, Community Partnerships, and Opportunities for Employment all record data on clients in different formats. Reconciling these various formats was well beyond the resources available for the evaluation, and we simply used SAMIN and EETIPJP data for all the client attributes, education/employment histories and outcomes.18 In an important sense Taking Charge! stumbled into the creation of its management system. It never had the benefit of professional database design and training that would have created an integrated management and evaluation resource tool.19 In hindsight, the expenditures to create such a system were modest in view of the pay-offs to management and evaluation. Routine management information is difficult to locate and verify. At this time, the Board of Directors receives a basic summary of program activity in the form of a "Quarterly Report" organized by fiscal quarter. This report summarizes the types of programs offered plus the numbers of clients attending, an overview of client attributes (visible minority, disabilities, gender, etc.), employment outcomes for graduates, and direct client service activities. Outside of the Phase 1 and the summative evaluation we found no additional in-depth analysis of client involvement by the service provider. It is clear that CAMS, has a limited role in preparing strategic information for management or the operations. We prepared the tables in Section 2 that relate client attributes and offer insights into program activity over time using very simple queries to CAMS, based on our extract. Clearly, the management at Taking Charge! is not sufficiently familiar with CAMS to prepare reports that would assist in planning future services and monitoring current activity. Furthermore, financial information and data on training service provider activity have been maintained in separate databases or in hard copy. This means that preparing reports is complex. For example, our report on training service providers in Section 5 required extensive file review as well as a long verification process with management. Contracting with training service providers is a primary activity for Taking Charge! and reporting on it should be more convenient. We should note that Taking Charge! planned to integrate the financial management and client activity data by December 1998. In the formative evaluation we wrote: As a service planning aid, a database management system would allow employment facilitators to review client needs and project objectives and make a match. Independence planning is currently paper based, when it could be systems based as client attributes and needs are matched to service providers offerings. A database management system would also support the follow-up process. Simple reports could be generated on a recurring basis to flag those that should be re-contacted to establish their current activity. A poor information system removes management from the strategic information needed to plan. A fully relational database, as Taking Charge! essentially has, would allow management to create unique and insightful reports on progress and activity. It would be a critical asset in monitoring the service providers. Rather than our manual review of files, information on average cost, outcomes, and the value for money offered by service providers would be continuously available. This would allow management to speak with greater authority on the program and its activities. CAMS is being upgraded. This should have been an early priority, and not a process coming to fulfilment after three years. The inadequacy of the database system has not only frustrated this evaluation but also has limited management of the project. 4.3.3 Management information system limitations are a serious issue for evaluating income assistance programs in ManitobaThe fact that we did not need this information from Taking Charge! (or any of the Comparison Group Programs) was fortunate. However, this exercise revealed a serious issue in the state of information systems in Manitoba available to track income assistance clients. SAMIN is designed as a financial payments management system. Recently (April and November 1997), SAMIN has been upgraded to track client activity. Additional screens have been introduced that capture education and employment history, permitting counselors to record training intervention, and referral dates of participation in an intervention. At this time, data entry is proceeding to populate the database. The EETIPJP data are all paper-based, although they are now recorded for certain clients in PRA's database. Each training program maintains its own database and approaches to client tracking. Further, when coupled with the variation in employability assessments, these inconsistencies in data management place the province in a poor position to monitor and track initiatives in welfare reform, education and training programs, and services to income assistance clients. The Integrated Case Management initiative is also expected to enhance the province's ability to track program activity. These improvements are important. Nevertheless, little sound information is collected on post-intervention outcomes. We can infer negatively. Clients who remain on income assistance clearly have not been successful in training. Those who are no longer on assistance may have found work, moved, gotten married, etc. It is not possible to determine the employment status of those who disappear from assistance and little can be said definitely about their economic independence. The recent upgrades to SAMIN and Integrated Case Management are all worthwhile activities, but follow-up information on client outcomes will still be needed to assess the value of training. 4.4 Operational, Legislative, Regulatory Constraints in Program Design and DeliveryTaking Charge! faces several operational, legislative, and regulatory constraints that are outside the effectiveness of the implementation of the program (discussed in Section 5). 4.4.1 Structure of Taking Charge! (Board composition)Membership in the Board is by appointment by the Ministers of Family Services (Manitoba) and the Minister of Human Resources Development Canada. Part of the initial start-up delay arose because government delayed the Board appointments. Vacancies are also filled quite slowly with limited representation from the business community.20 4.4.2 The levelling and employability assessment processes require further developmentThe system of "leveling" is central to Taking Charge! and the employment and assistance counseling process. Upon the initial application for income assistance, counselors assign a "level" that summarizes job readiness or employability. Clients may also be assessed for employability by Taking Charge! (and its service providers) as well as by EIA counselors. Everyone uses the same general set of questions to establish the degree to which clients fall into one of three levels as shown in Table 26.
Leveling is common throughout the health and social services system. "Triage" in emergency health care assigns patients to three levels based on the immediacy of care needed. In the home care system, leveling is used to first determine eligibility for home care and then to assign clients to resources. In long-term care, clients are placed into four (soon to be nine) levels indicative of their ability to care for themselves, the level of nursing care required and their potential for harming themselves and others. The leveling process also becomes a mechanism for assigning resources. Each of these leveling systems is based on questionnaires, similar to those used by EIA counselors and Taking Charge! employment facilitators.21 The triage used in emergency health, home care, and long-term care is based on question sets that have been subjected to careful reliability and validity testing. A core set of questions used in both the home care and long-term care questionnaires refers to activities of daily living (ADL). Several ADL measures exist in the literature, and others are being refined through a systematic process of comparing outcomes with the ratings obtained in the questionnaires. The process of comparing outcomes to the levels assigned to clients is fundamental to the integrity of the leveling process. Without this testing, leveling functions neither assign clients to appropriate care, nor allocate resources based on need. The leveling process used by the EIA counselors and Taking Charge! has not been applied with consistency. The fact that levels are incomplete in the Taking Charge! database suggests that they are not seen as crucial planning information. In fact, based on interviews with staff in Family Services and Taking Charge!, although different counselors and facilitators may complete the same form, no assurance exists that this produces consistent results. Although the assignment of levels is partly the result of past education and employment, intangibles such as personal/family problems and "motivation" are factored into a client's level. Clearly, different counselors may rate the same client differently. Another important feature of the employment assessment process is that it should be dynamic, updated as clients proceed through various employment enhancement measures. Any updating that is done presupposes that assessments are completed for everyone. Assessment of the reliability and validity of the employability assessment and leveling process has not been done. Although EIA and Taking Charge! personnel complete similar forms, they are not identical. Coupled with the fact that many people complete the assessment on clients, a high likelihood exists that assessments and leveling vary among counselors. Most respondents we interviewed in government confirmed that this was probable. Further, no assessment appears to have been done of the basic set of questions used to predict whether a client will be successful in a course of training. This is not a criticism of Taking Charge!. Rather, it is a statement about the fragility of the general process for evaluating income assistance clients for job readiness. Employment facilitators emphasized that the employability assessments are not used to create levels. For most, the "level" of a client is quickly determined after a few background questions on work history and education. Facilitators reported that the detailed assessments may be used to identify potential barriers to training and employment and to develop programs. The application of the assessment to this process probably varies among counselors and facilitators. As we show in Section 5, training providers also complete the Taking Charge! assessments. Training providers funded by Taking Charge! are expected to forward client assessments to the Taking Charge! office before accepting clients into the program. Many may complete these assessments in relation to their programs. For example, McKnight and Associates accepts only those clients who it believes will benefit from an intensive work preparation and who will be successful in securing employment. In general, a training contractor may well have different goals than an employment facilitator in assigning a level to a client. The range of people and organizations completing these assessments has the potential to substantially corrupt the employability and leveling process. Each Taking Charge! employment facilitator must identify barriers to training and employment based on individual interpretations. EIA counselors are required to complete employment assessments as a basis for job plans for all clients assigned work expectations. Initially Taking Charge! and EIA processed assessments independently, but they now share employability assessments. A need exists for a reduced-form employability assessment, tested for reliability (controlling inter rater variation) and validity (having the ability to predict outcomes). The province is addressing these issues. All counselors have been trained in completing employability assessments. SAMIN will build a history of each client who is assessed. With the advent of Tier 1, all provincial clients will be assessed by the end of 1998. These are important changes that will improve the province's capacity to evaluate training. 4.4.3 Taking Charge! and Making Welfare WorkTaking Charge! began coincident with a major welfare reform initiative called Making Welfare Work. Although it is not possible to completely disentangle the effects of these two initiatives on the employment of income assistance clients, an operational issue for Taking Charge! is how Making Welfare Work has affected its clients. Also important is the role of Taking Charge! in welfare reform. Key informants made the following points:
4.4.4 Taking Charge!'s role in the "system"Many staff believe that a key element of their success is that Taking Charge! is both apart and seen to be apart from government. This is evident in the Program's focus on being an independent entity and presenting a friendly and welcoming face to income assistance clients. This is distinct from the bureaucratic approach of government, especially the approach of those departments whose function is to verify that clients are eligible to continue receiving income assistance. The outcome of this view may be seen in two ways:
4.5 Strengths and Weaknesses of Program Design and DeliveryStrengths and weakness of Taking Charge!'s design and delivery model are as follows:
| ||||||||||||||||||||||
![[To Top]](/web/20060127003116im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127003116im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127003116im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127003116im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}