|
|
     |
||
 |
     |
 |
|
This section of the report presents the detailed evaluation findings and the study team's assessment of them. The section is subdivided into four parts, which discuss in turn the relevance of the initiative, its design and implementation, its impacts, and the costs as compared to the benefits. Throughout this section, the information is presented on the basis of graduates and exits or participants where appropriate. The text indicates those instances where there are statistically significant differences between groups within these categories, such as NSDP versus WAP or East versus West. 4.1 Relevance Relevance captures a range of questions with respect to the appropriateness of the initiative on a number of different levels: that of the participants, their communities, and NWT and Canada-wide social policy.
4.1.1 IIP in the Context of the Federal Strategic Initiatives Background IIP is one of the programs sponsored under the umbrella of the federal Strategic Initiatives. Key to Strategic Initiatives is the search for new ways to deliver income support and training. Findings As stated in Section 3, the goals of the IIP initiative are:
The IIP initiative helps participants achieve self sufficiency through education, work experience and support services, recognizing full well that for many participants this will be a long-term undertaking. IIP sponsors two types of projects, NSDPs and WAPs. The NSDPs rely extensively on existing ABE curriculum. NSDPs are different from the Colleges' ABE programming in that they include explicitly personal and career counselling, and work placements. The work placements especially are a new design feature for training services as compared to ABE programming of the Colleges. The WAPs are designed individually by the project sponsors and vary considerably. Central to all WAP designs is the importance of work placements, supported by skills-based training and upgrading, personal life management, and career and personal counselling. The WAPs echo the experience with programming for SARs in the NWT under the Canada Assistance Plan, when programming that aimed to provide specific work skills to persons with handicaps was opened up by making all SARs eligible. The design of most WAPs goes beyond the experience of the late 1980s and early 1990s by explicitly incorporating career and personal counselling. Conclusion The IIP initiative is conducted in the context of Strategic Initiatives' search for new ways to provide a social safety net for Canadians. The study team is impressed by the fact that IIP does not limit its definition of "reduced dependency on social assistance" to a narrow job or employment focus. A strict job focus is not meaningful in the context of many communities in the NWT. IIP's goal of generally fostering participants' independence appears to us entirely appropriate in the context of limited job opportunities and considerable social dislocation. IIP introduces as well modest innovations built into the design of the NSDPs and WAPs. They build on existing northern experience and resource materials (ABE and life skills curriculum and a history of work placements under the SAR program) and integrate them in a multi-faceted approach: the NSDPs incorporate work placements and most WAPs classroom training. 4.1.2 Target Group Background The IIP program focuses on persons who are on, or who are eligible to be on, social assistance. Eligibility criteria are stated in Section 3.3. Were these criteria interpreted in a uniform manner? Findings Although the eligibility criteria define potential participants quite precisely, the field work conducted in the context of this evaluation suggests that there are some differences of opinion regarding the target group of the initiative. Key informant interviews and the case studies C described in the Technical Appendix C indicate at least two nuances in the definition of the target group. Some people, and this was perhaps most noticeable in persons involved in NSDPs in the eastern Arctic, see the IIP initiative as a possibility to reach out to persons with very low levels of educational attainment who are poorly covered by other programming. Others want to focus on those SARs who are almost job ready. These nuances are reflected in the evolution of the eligibility criteria that individual projects used, such as easing of the originally hard and fast rule that people with substance abuse problems were not eligible. In its original formulation, this eligibility criteria seriously hampered the recruitment of participants, leading to a more liberal interpretation of eligibility. Changes in eligibility criteria also took place with respect to limiting the project intake to persons who were on social assistance at least three months to the more general criterion of persons who are at risk of long term or recurring unemployment, while still eligible for or on social assistance. Information from the Iqaluit case studies indicates that at least some people think that the program should not have excluded persons who were not on, or not eligible for, social assistance. They suggested that from a community perspective, the impact of the program may have been limited by its eligibility criteria. Conclusion The fairly subtle differences regarding the definition of the target group did not have a significant negative impact on the initiative, although individual projects at times struggled with them. Broadening the criteria with respect to persons with substance abuse problems appears reasonable in cases where recruitment would have been hampered otherwise. Opening projects to persons with substance abuse problems increases the challenges faced by instructors and there is anecdotal evidence from the case studies of some disruptive behaviour in classes. Issues around eligibility and substance abuse underline the need for strong integration of support services. 4.1.3 Participants' Profile Background The question to what extent IIP reaches its target group is, in part, answered by analyzing the participants' profile. This profile should reflect that of the target group. Additional insights into this question can be gained from analyzing the initiative's geographic coverage and target group penetration. The former analysis is presented here and the latter in Section 4.1.4, Project Coverage. Related information on the extent to which the services provided are consistent with the needs of the target group is presented in Section 4.2.2, Appropriateness of Project Components and Section 4.2.3, Satisfaction by Project Component. Findings Table 4.1.1 presents the participant profile information gathered through the surveys administered as part of this evaluation. It indicates that 340 IIP participants (191 graduates and 149 exits) are predominantly young adults with low educational attainment. Specifically:
With respect to the male-female split, the IIP-Year 2 projects reached relatively more women than men. An analysis of participant lists indicates that 55% of participants were female and 45% male.15 In comparison, the gender split of the survey was 38% males and 62% females. This difference may be due to the fact that more men than women were out of their communities and on the land when the surveys were undertaken in July and August. The survey findings suggest that IIP participants, on average, may have had higher education levels than the overall population of social assistance recipients in the NWT. An estimated 50% of those who participated in this evaluation reported that they had a Grade 8 education or less as compared to the approximately 65% of social assistance recipients in the NWT.16 Table 4.1.1 shows as well that 5% of participants (n=18) have a grade 12 education or equivalent. This appears high relative to the general NWT educational attainment, especially recognizing that most projects were located in small aboriginal communities. One reason for the reported result may be that respondents exaggerated their educational attainment. Another reason may relate to the differences in interpretation with respect to the target group, as discussed above, and may reflect a tendency to select the most job-ready candidates from among all eligible persons. A comparison of the profiles of graduates and exits indicates no statistically significant differences between the two groups in respect of their gender, age, marital status, education, and presence of preschool children in their families. Conclusion The findings provided above, including the observation that there are no significant differences between graduates and exits, support the conclusion that the referral and selection processes used by IIP projects worked reasonably well. The resulting group of project participants is reflective of the target group of the program. Table 4.1.1
Notes: Not all respondents answered all questions, resulting in a slight variation in the number of respondents by demographic dimension. All percentages are rounded to the nearest integer. Totals may not add to 100% due to rounding. The Technical Appendix provides participant profiles broken down by WAP and NSDP and East and West. 4.1.4 Project Coverage Background The extent to which IIP reaches its target group has as well geographic and size dimensions. The relevance of the IIP initiative is linked to the number of participants relative to the target population. In addition, in an area as vast as the NWT, the location of projects may give an indication of whether or not the initiative reaches into all regions. Findings As discussed above, the exact number of IIP participants is not known because the initiative has no system in place to track accurately who entered the programs, who graduated, and who left (and when). The number of persons involved with the IIP initiative during its second year is estimated to be in the range of 900 to 950. This equates to roughly 1.5% of the total population of the NWT and some 8% of the total beneficiaries of the social assistance system.17 The IIP initiative has a broad geographic coverage, as shown in Table 4.1.2. The breakdown of IIP projects corresponds roughly with the relative population numbers in the different regions, except that the North Slave Region has relatively fewer projects and the Keewatin relatively more projects than population numbers would indicate.
Table 4.1.2
With respect to community coverage, Year 2 of the IIP initiative reached two-thirds of all NWT communities with populations in excess of 100 (35 out of 53). Two additional findings with respect to the initiative's coverage are:
The former finding suggests that people in the communities recognize a high need for IIP or similar projects. This observation needs to be interpreted in light of the finding, presented above, that some 8% of total social assistance beneficiaries were involved with IIP projects in some way. The latter finding indicates the importance of flexibility on the local level. Conclusion The IIP initiative involves a considerable proportion of the persons who meet the broad eligibility criteria. It furthermore reaches all parts of the NWT and provides projects in two-thirds of all communities with populations in excess of 100. In the opinion of the study team, the coverage of the IIP initiative is extensive.
4.1.5 Exits Background Exit rates are an indication of the "fit" between the project and the needs and aspirations of the participants. Reasons for leaving the projects provide information that is relevant for the day-to-day management of projects. Findings The exact number of persons who entered an IIP project and did not continue to the end is not known. IIP has no system that tracks accurately who participated and who left, an issue that will be discussed further below. Forty-four percent (n=149) of our surveys are exit surveys, suggesting that 44% of participants dropped out. However, this number is influenced by decisions of the study team and community researchers about who to interview. Lists of project participants gathered by the study team suggest an exit rate of 50%. This statistic is derived from start and end date information that was provided on approximately half of the participants. Field work revealed the existence of other, longer lists in the communities, suggesting that the names of at least some exits were lost along the way. This exit information can be contrasted with recent experience of other programs. Nunavut Arctic College experiences ABE exit rates around 35%, recognizing that some persons leave the program to transfer to other programs or for employment.18 The Northern Addiction Worker Training program shows an exit rate of around 30%.19 Exit rates experienced by IIP in Year 1 were similar to those of Year 2. Conclusion Although the program data do not allow for an accurate assessment of exit rates, the available evidence suggests that the exit rate was approximately 40%-50%. This appears high relative to other programs, such as ABE or Northern Addiction Worker Training. The uncertainty associated with the exit rate suggests the need for stronger systems to track participants and management instruments that will make it possible for project managers in the communities and the Colleges to mitigate those circumstances that make people leave the projects. A consistently and timely administered and analyzed exit survey C such as included in the IIP Program Handbook in July 1996 C may provide the necessary information. Leaving an IIP project is not necessarily a failure. IIP's objective is to help persons along the way to reduced dependency on social assistance. In this context, an exit can be considered a success if that person, on balance, had a positive experience. Information on the impacts of the projects on exits, presented in Section 4.3, suggests that many exits indeed benefited from their involvement in the projects. A further analysis of the reasons why people leave the program indicates that employment features among them. In addition, all education and training institutions experience a certain level of exits as students assess their involvement, change classes, schools or study direction, or decide to pursue other activities. No program or initiative could appeal to all persons for which it is intended.
4.1.6 Reasons for Leaving Background The reasons that participants give for leaving provide insight into the weakest aspects of the projects. Findings Table 4.1.3 presents the reasons exits give most frequently for leaving the projects. As with all survey-related information, this table relies on self-assessment of the respondents and cannot be checked against other information. The total in the table does not add to 100% because multiple answers were permitted.
Table 4.1.3
The survey probes as well which parts of the program respondents did not like. Fourteen percent of exits indicate that the project was not challenging enough, while 5% mention each of the following: not enough instruction, instructor too personal, gossip, language of instruction, and other.20 The information presented in the table clearly indicates that child care difficulties feature prominently among reasons to leave the program. As mentioned before in Section 3.3, IIP projects do not provide child care spaces. IIP participants are, however, eligible for child care allowances provided by a different division of ECE. Interviews conducted in the context of the three case studies indicate that the problem with child care allowances relates to their late arrival. If allowances arrive late, the affected participants may not be able to keep up with payments for child care, leading to child care providers quitting. These interviews indicate as well that the number of good child care providers is often insufficient. The other major reason for dropping out of IIP projects relates to the instructor. Twenty-one percent of respondents to the exit survey indicate that instructor-related issues play a role.21 This finding underlines the importance of the instructors in IIP projects. The study team observes that the implementation of IIP is influenced by system-wide deficiencies with respect to instructors: there is no full-time ABE training program (although there is a part-time program), there is no full-time life skills teacher training program; and there is a lack of aboriginal instructors, especially in the west. The experience of the study team suggests that contract instructors often have little or no preparation time or orientation to the project prior to starting work. The evaluation activities uncovered isolated instances of instructors arriving from outside the community on Friday to start teaching on Monday and of instructors displaying considerable insensitivity towards the culture in which they found themselves. IIP recognizes the importance of appropriate orientation and supported Aurora College's request for the professional development workshops in Fort Smith and Yellowknife to support NSDP instructors already in place. The Program Handbook includes instructor orientation among the sponsors' responsibilities. Finally, one of the top six reasons for leaving the project is very positive for the IIP initiative: 13% of exits left the initiative for employment. This finding is further corroborated by the fact that almost 60% of exits indicate that they have worked at a job since their involvement with IIP. Conclusion The major reasons that respondents mention for leaving the projects - child care, personal problems, and instructors - are "old" problems. The experience of the Colleges and other programs has underlined the importance of these factors time and time again. Because it is known that these factors would be crucial, they could have been alleviated within the confines of the IIP initiative. Orientation of the instructors to the projects and timely arrival at the project site should have been the norm for all projects. Child care allowance could be paid out by the projects, with subsequent reimbursement by ECE.
4.1.7 Project Alignment with Community Needs Background Project relevance can be looked at from two perspectives. A program-centred view looks at the alignment of the IIP projects with the IIP guidelines and the alignment of the IIP initiative with the objectives of the Strategic Initiatives. This perspective was examined in Section 4.1.1. It can also be viewed from the perspective of the communities, raising questions about the alignment of the projects with community needs and aspirations. Findings More than four-fifths of the 132 key informants agree with the statement that the projects fit well with the social needs of their communities. Slightly less than three-quarters see the projects as well aligned with the communities' economic needs. The finding that the projects generally fit the community needs is tempered by the fact that only about a third of key informants feel that the communities were sufficiently involved with the project planning. This finding should be placed in the context of the IIP approach to community involvement, which includes advertising for project sponsors and personal contacts of regional ECE officials with groups and organizations who might be interested in IIP. Conclusion The projects are well aligned with the community needs. This alignment is achieved despite relatively limited community involvement with the projects. This observation suggests that additional community involvement could further enhance the relevance of the projects from the perspective of the communities. The planning timelines of future initiatives should be amended to allow this more intense interaction.
4.1.8 IIP as a Stepping Stone Background The time certain nature of IIP underlines the importance of its linkages to other programs or agencies that may be able to work with IIP participants as they make their journey towards lessened reliance on social assistance. Findings Most IIP participants had realistic expectations when they entered the program. Seventy-two percent (n=138) of graduates indicate that they were motivated by a desire to improve their education and 51% (n=98) wanted to increase their job skills. Only 22% of graduates indicate that they thought that they would get a job after the program and 20% state that their desire for a job played a part in their participation with the projects. IIP is a two-year pilot initiative. It focuses on delivering programming within its time frame and contains no provisions for ongoing support or monitoring of participants. Some individual WAPs have a longer time horizon. One project gathered data on and trained people for an evolving renewable resource industry. Another project provides articulation with ongoing trade programming. Most of the WAPs, however, do not provide a formal linkage to further work experience or training programs. In general terms, the low level of economic development in many smaller communities and the retrenchment of the GNWT, the largest employer in the Territories, severely limits the number of training or work placement positions outside the IIP initiative, making the linkage to ongoing work activity-based training tenuous. There are some exceptions, especially for communities in close proximity to existing or planned mine developments or in the context of establishing an infrastructure for Nunavut, where ongoing on-the-job training may be a possibility. The NSDPs link to the future lies mainly in career counselling and career action plans. The graduate survey indicates that 25% of NSDP graduates (n=24) did not get or could not remember getting career counselling, a required part of each project. Seventeen percent (n=17) indicate that they did not receive a career action plan. Those who received the counselling or the career plan express a high level of satisfaction with them. Three quarters (53 out of 71 respondents) express satisfaction with the career counselling and 80% (65 out of 81) with the career action plan. A further link to future training opportunities relates to the increase in educational attainment of NSDP participants. NSDP graduates will receive preference over other students when enrolling in further ABE programming. Conclusion IIP is a two-year initiative and this time frame limits its ability to forge solid linkages with other programs for its graduates. There exists a linkage between NSDPs and ongoing College programming via the normal testing procedures the Colleges use for assessing student attainment levels. IIP increases this linkage by obtaining a preferred entry of its graduates into ABE programming of the Colleges. With respect to the WAPs, the linkage to ongoing work placements or training opportunities is virtually non-existent.
4.1.9 Service Gaps Background The IIP initiative is especially relevant if it fills a gap in government service provision to the communities. The relevance of the project is diminished insofar as its services overlap with those of other programs. Findings In recent years, the funding of training of SARs in the NWT has undergone significant change. In the late 80s and early 90s, CEIC, now HRDC, cost-shared SARs= programming under the Canada Assistance Plan, using funding from Health and Welfare Canada. It was involved as well with an annually negotiated government to government training plan. This latter program targeted a broader group than just SARs. Since then, part of those funds have been diverted into the Pathways strategy, which is now controlled through regional bilateral agreements. The balance of the money is now expended through an annual ECE training plan. IIP is the follow-on program to the SARs program in the late 80s, administered on the federal side by HRDC. The annual ECE training plan, which targets more than just SARs, and IIP form the bulk of programming for SARs. There are some instances in which IIP-funded projects overlap with other programs or training activities. For example, two projects provided training to construction and retail workers who would have received training in the absence of the IIP initiative, although possibly less comprehensively. In another instance IIP participants mixed with the general student population of Aurora campus and took the regular College Office Procedures (COP) program side-by-side with non-IIP participants. This clearly indicates an overlap between an IIP project and regular college programming, with the proviso that the College may not have been able to offer COP at that particular time without the enrollment of IIP participants. In general, there is overlap between programming available to communities. Section 5, Beyond IIP, enumerates 14 different human resources programs and there are overlaps between IIP and some of them. For example, one project came to IIP after applying for Pathways and Building and Learning funding. One NSDP classroom was side by side with a regular ABE program, using essentially the same curriculum and targeting essentially the same persons. As discussed in Section 5, Beyond IIP, potential overlaps, such as between ABE and NSDPs, play a role in allocating different programs and projects to different communities and can result in a community getting a NSDP because it has no regular ABE programming. This may limit the actual overlap of the two programs notwithstanding the fact that they are in many ways similar programs. Conclusion The IIP initiative in part fills the gap that was left by the termination of the Government to Government Training Plan and the SARs program under the Canada Assistance Plan. It follows in the footsteps of the SARs program and shifts the emphasis of the training that was provided under Government to Government Training Plan by including work placements. IIP projects show some overlap with existing programs. The overlap is most clear between the Colleges' NSDPs and their regular ABE programming as evidenced by the fact that NSDPs use regular ABE curriculum. In addition, the educational attainment of many IIP participants, described in Section 4.1.3, is sufficiently high to make them eligible for regular ABE programming. Indeed, the Colleges use IIP as a vehicle to provide upgrading in selected communities and in doing so expanded the ABE offering over what would have been available without the initiative. The reality of often non-continuous ABE programming in many communities means that NSDPs provide ABE-based programming in communities that would not have had any upgrading courses otherwise.
4.2 Project Design and Implementation Having discussed issues related to the relevance of the project, we now turn to related issues of project design and implementation. This section addresses questions with respect to the appropriateness of the IIP initiative's activities to facilitate achievement of its objectives. 4.2.1 Project Needs Assessments and Planning Background The first step towards the creation of an IIP project is to do an assessment of the needs of the community and to plan the activities that would be undertaken to meet those needs. If these activities are not done well, the success of the projects is at risk from the outset. Findings NSDPs are planned by the two Colleges, using their internal mechanisms. Although the Colleges have varying degrees of community involvement built into their way of conducting business, it is the study team's assessment that the NSDPs are essentially planned centrally by staff of the Colleges. They cooperate with officials of ECE, with little or no involvement from H&SS and HRDC. This finding is supported by a review of the Program Planning and Delivery Process for NSDPs described in the Program Handbook, which places the responsibility for the preparation of College proposals with the College headquarters and the headquarters of ECE. WAPs involve community-based organizations, insofar as prospective project sponsors were invited to prepare proposals for funding under the IIP initiative. Project sponsors are involved as well in the implementation of the projects, in cooperation with CSSWs, CDOs, Income Support Workers, and others. It follows that the community involvement in planning and executing WAP projects is greater as compared to NSDPs. Two of 30 WAP project sponsors were not community-based organizations: one was an Edmonton-based construction contractor building a dam for an aboriginal Development Corporation and the NWT Development Corporation and another was a Territory-wide retail organization. Overall only about a third of key informants agree with the statement that local people and organizations were sufficiently involved in the planning of IIP projects. This finding is not surprising given the fact that the Program Handbook does not include communities or community-based organizations in the planning, approval, or delivery of NSDPs. WAPs show more community involvement, but even there the involvement is limited to the project sponsor and does not include the community as a whole. The planning process notwithstanding, the IIP projects generally fit well with the social needs of the communities and to a lesser extent with their economic development needs. These issues are discussed in Section 4.1.7. Project needs assessments are undertaken as part of the WAP proposals submitted by the project sponsors. The Colleges, in cooperation with officials of ECE, do a more or less explicit needs assessment as part of the decision of where to offer NSDPs. The evaluation activities provide only limited indications of the extent to which the Colleges, ECE, or project sponsors use available information on community needs. Labour market statistics for Deh Cho communities have been developed, with other regions forthcoming, and the Department of Economic Development and Tourism and the NWT Statistics Bureau distribute labour market information in various forms. In addition, there are many more general studies that describe the NWT labour market, social conditions, and economic development opportunities and barriers, including Department of Education, Preparing People for Employment in the 1990s, the strategic plans of the Aurora College and Nunavut Arctic College, and IIP Year 1 evaluation. Conclusion The division of roles and responsibilities with respect to needs assessment and planning - as well as for subsequent approval, implementation, and evaluation phases - is well defined for the IIP initiative. The projects generally followed the outlined procedures. The needs assessment and planning parts of the IIP initiative have the result that NSDPs are essentially given to the communities rather than generated by them. WAPs involve project sponsors, most of which were community-based organizations. The study team notes, however, that although the IIP initiative reaches out to the communities in the case of the WAPs, it reaches out to individual organizations and not to the community at large. Only a third of key respondents are satisfied with the level of community involvement and the case studies show a strong desire for involving the communities in their totality in the planning of projects, be it through Band, municipal, or other organizations.
4.2.2 Appropriateness of Project Components Background All IIP projects consist of an intake procedure and a combination of classroom training, work placement, career and personal counselling, and life skills training. The relative importance of the various components differs by project. Findings The selection process for most IIP projects involved a variety of persons, including the CSSW, CDO, the instructor/project coordinator, if in place, and a representative of the project sponsor. Other persons may have been involved as well. Key informant interviews indicate that the referral and selection process was generally successful. About three-fifths of key informants indicate satisfaction with the involvement of CDOs and CSSWs in the selection process and the amount of communication among all persons involved in the selection process. However, the three case studies in Rae Edzo, Inuvik, and Iqaluit provide instances of classes that were disrupted by participants with severe drug and alcohol problems, people participating in work placements that did not interest them, and classes with participants with a very wide range of academic abilities. Although there are no data to substantiate it in the context of this evaluation, the experience of the study team suggests that the selection process may have been hampered by the fragmented nature in which client assessments take place in the communities. Any person who is a client of government services may be assessed by the CSSW, the CDO and the ABE instructor. These various assessments, conducted from different perspectives, may not be shared easily within an inter-agency selection process due to confidentiality concerns. It is unclear how much pre-testing of participants took place at project inception or if pre-tests were used to bring together relatively coherent groups of participants. Evidence from the case studies indicates that there were at least some NSDPs with participants with a wide range of skills and educational background. With respect to project content, the IIP initiative exemplifies a further development of existing training delivery models. Its design forces integration of classroom training and work placements and recognizes the need to support these with life skills training and personal and career counselling. These latter services have been available in most communities, but typically not within a single program. The NSDPs and many of the WAPs make extensive use of existing ABE curriculum. The study team notes that the ABE curriculum has been developed and modified over the years to increase its northern relevance. The projects make extensive use as well of existing life skills curriculum and in some instances, NSDP participants were enrolled in other College-based programs, such as the Community Office Procedures program. The study team is not aware of work placement guidelines, except a short description of roles and responsibilities of workplace hosts in the Program Handbook. Ideally, the work placements should have had some guidelines, including a requirement for work placement host and worker orientations, individualized work placement training goals and strategies, feedback mechanisms, and ongoing participant mentoring and support by the project instructors/ coordinators. Although most participants express satisfaction with the work placements, there were isolated instances of IIP work placements that did not work well. The study team was told about a work placement where participants did not show up at all and of a placement where the host was not informed of when the participant was supposed to come or who in the project to talk to if questions arose. Traditional knowledge/language instruction is an optional component, in part in response to findings of the IIP Year 1 evaluation. The survey results indicate that 44% of graduates (n=84) report that traditional knowledge/language instruction was not a part of their project. Most instruction was given in English and 84% of graduates (n=161) expressed satisfaction with the language in which instruction took place. Impacts and satisfaction levels of participants with the various program segments have a bearing on their appropriateness. These are discussed in section 4.3. Conclusion Key informants generally consider the referral process to be appropriate. As discussed in Section 4.1.3, the profile of participants C an output measure for the referral process C is reflective of the target group. Many IIP projects use existing ABE and life skills training materials. Considering the availability of this material and the efforts that have been expended to make this material more relevant to the north, their use in IIP projects is appropriate. Support materials for the work placement component are not available and the projects were left to fend for themselves in this area. The importance of the work placements as part of the integrated delivery of services inside the IIP initiative suggests that development of such materials would have enhanced this part of the projects. The evidence with respect to the inclusion of traditional knowledge and language is mixed. Graduates are by and large satisfied with the language of instruction, which is English. However, two-fifths of key respondents indicate a dissatisfaction with the involvement of elders in the projects, suggesting that there should be more emphasis on traditional pursuits. IIP leaves the inclusion or exclusion of traditional knowledge and aboriginal language up to the individual projects.
4.2.3 Satisfaction by Project Component Background This evaluation sought input from graduates about their level of satisfaction with various parts of the project in which they were involved. These satisfaction levels give insights into the appropriateness of the programming. Findings Table 4.2.1 presents the percentage of the IIP graduates who received the listed project components and expressed their satisfaction with the respective components. The table, which reflects only the satisfaction scores of those graduates who indicate that the particular component is part of their project, shows very high levels of satisfaction with the work placement, education and skills training, and the language of instruction. Not all graduates indicate that they received all components. However, only 10% of graduates indicate that they did not receive one of the components presented in Table 4.2.1. Table 4.2.1
It is instructive as well to see which components received the lowest level of satisfaction. These are listed in Table 4.2.2, which shows that respondents are less satisfied with the financial and support aspects of their involvement with IIP. It should be noted that the table indicates child care in general. No project provides a child care program; rather participants are eligible to receive child care allowances. These findings generally reflect the information gleaned from 132 key informant interviews. Approximately 90% of key informants feel that the participants gained knowledge and skills and less than half of key informants agree with the statement that the projects helped participants with personal and family needs. Approximately one-third of key informants express as well concern about how the projects provided for the financial needs of the participants and about half question the adequacy of the $10 attendance allowance. Table 4.2.2
WAP participants are more satisfied than NSDP participants with the following aspects of the projects:
These differences are statistically significant. There are some statistically significant differences as well between the satisfaction of participants in eastern as compared to western Arctic IIP projects. Eastern Arctic participants are more satisfied with:
Conclusion Project components generally receive a high satisfaction rating from graduates. The findings suggest that the projects were most successful, as measured by participant satisfaction, in those components that are skills oriented. IIP was less well equipped to deal with the personal aspects of development. Forty-five percent of the 192 surveyed graduates indicate that personal counselling was not part of their project. This finding holds true for both NSDPs and WAPs. The lowest level of satisfaction relates to the financial aspects of the projects. The satisfaction-based assessment of IIP projects tends to confirm the initiatives relative greater success in conveying skills than in dealing with personal issues. The study team is intrigued by the differences that show up between graduates in the eastern and western Arctic. Part of the explanation may lie with the higher incidence of permanent aboriginal ABE instructors in the east as compared to the west. The findings show a higher level of satisfaction with WAPs than NSDPs. This could stem from either their greater emphasis on work placements or their inherent higher level of community involvement. On balance, the evidence suggests that the higher level of community involvement is the most significant contributor to these differences, because:
4.2.4 Alignment of Projects with Project Proposals Background The planning phase leads to project proposals that are evaluated on their merit in the context of IIP guidelines. Beyond the approval, the actual implementation of the projects must be in line with the submitted proposals. Findings Although the evaluation includes three case studies, it is not an evaluation of individual projects and the findings pertaining to the alignment of projects with project proposals are not based on a project-by-project assessment. That said, the evaluation activities, especially the case studies and the 132 key informant interviews, do not flag any real concerns in this area. With the exception of one or two projects, the projects that were looked at in some detail were well aligned with their proposals. The exceptions were modifications to NSDPs in progress, based on the College's perception of the needs of participants. These modifications were, in the view of the study team, appropriate changes in view of local circumstances. Conclusion IIP projects were essentially delivered within the confines of their respective proposals.
4.2.5 Project Monitoring and Control Systems Background Project management requires monitoring and control systems for reasons of accountability and to aid the management of individual projects. Findings The IIP initiative is delivered in a relatively decentralized manner and reporting arrangements are as follows:
Reporting of financial data is supported as well by the central GNWT financial systems which record actual expenditures by the regional ECE offices and payments made to the Colleges. Cash flows of the projects are projected and monitored by IIP personnel in Yellowknife. No detailed analysis or audit of the financial records of the IIP initiative was undertaken as part of this evaluation. Casual observation, however, suggests that the financial administration is acceptable. The organizational structure of IIP, which includes that the GNWT pays the contribution agreements and then invoices HRDC in Yellowknife for the federal portion, provides a strong incentive on the part of ECE to conduct the financial administration in a timely and accurate manner. With respect to the more management-oriented reporting, the study team observes that there is considerable variability in the form, detail, and accuracy in which project information is provided. Much of the reporting is on the level of listing the activities undertaken in a time period. In addition, the decentralized implementation of IIP means that there is no central point where all management information is gathered and analyzed. Nor is there evidence of management information funnelled back to the projects. In general, IIP staff experiences frustration with respect to getting community-based sponsors to submit information as outlined in the contribution agreements. The exception is financial information. All contribution agreements are collected by IIP headquarters and expenditures are monitored there. The IIP initiative lacks an adequate management information system that could be used for the day-to-day management of projects, the overall initiative, and evaluative activities. For example, the study team encountered considerable delay in receiving lists of participants to be used as the basis of its survey activities. The lists, when received, were in a wide variety of formats, and often lacking in very basic information. In almost 50% of cases, it was unclear from the lists if participants were graduates of or exits from their project. This suggests that individual projects could not get information about how their exit rates compared to other IIP projects in the region or NWT-wide. In the end, the study team received lists with a total of 884 participant names. The key reason behind the difference between 688 funded positions and the 884 participant names submitted to the evaluation team lies with the continuous intake of new participants by many IIP projects, especially NSDPs. Continuous intake is in line with the experience of ABE programming in the north and across Canada, and the IIP Program Handbook indicates that projects should have regular intake dates to maintain their maximum client level of participation. The expected and common nature of the continuous participant intake raises the question, however, why a better system of tracking participants was not put in place. In addition, it raises questions about who is a "graduate" of projects. Clearly, when intake is continuous, the training and other program services received by a graduate who started on the first day of the project will be different from that received by somebody who entered halfway through the project. Turning, finally, to evaluation-related information, the projects were asked to deliver a Canada-wide baseline survey that was very extensive and detailed. The baseline survey was poorly received by many participants, reducing their willingness to contribute to further evaluation activities. In addition, the terms of reference for the Year 2 evaluation - a joint federal and territorial activity - were very detailed. The very lengthy questionnaire needed to answer all the detailed questions would have resulted in a poor response rate. Discussion between the study team and the IIP Evaluation Committee reduced the size of the questionnaire by focusing on the key aspects of the program. The relatively small questionnaire notwithstanding, it took considerable effort, including drawing heavily on personal goodwill of study team members with people in the communities to obtain the response rate indicated in Section 2. Conclusion The IIP initiative appears to have had appropriate financial information systems in place. However, its project-level management information systems were inadequate, as evidenced, for example, by the inability of the system to generate participant lists in a timely and accurate manner. With respect to the IIP Year 2 evaluation, the experience of administering the baseline survey instrument and the more tightly focused Year 2 evaluation questionnaire clearly indicates the limitations of surveying in the north. The very small population base relative to the number of government programs, many of which undertake surveys in the event of needs assessments, evaluations and other progress-related activities, reduces the possibilities for very in-depth surveys.
4.2.6 Jurisdictional Issues Background There are a number of agencies involved in IIP, including the ECE, H&SS, and HRDC. This has the potential to raise jurisdictional issues. Findings The IIP initiative was delivered by ECE, with limited involvement of H&SS, especially after the program design phase. The involvement of HRDC has been limited as well, and there do not appear to have been any jurisdictional disputes that had an impact on the projects. Conclusion The implementation of IIP is not impaired by jurisdictional problems between the GNWT and the federal government.
4.2.7 Attendance Allowance Background The IIP initiative provides a $10 per day attendance allowance to participants. This attendance allowance is paid by the project sponsor in addition to the social assistance participants receive from ECE. Attendance allowances are controversial in that they add to the project costs, while it is unknown to what extent they are a motivation to participation. Findings Survey respondents indicate that attendance allowances are not a significant motivation for entering the program or staying with it. Of the 192 graduates and 149 exits surveyed, only 13% of project graduates (n=25) indicate that the attendance allowance was part of their reasons for entering the projects, while only 13% of exits (n=19) cite the attendance allowance as a reason for their decision to leave the project. These findings are further corroborated by the survey result that 68% of graduates (n=130) indicate that they would have entered the program if it did not offer the attendance allowance. These findings suggest that reducing or eliminating the attendance allowance may well be an effective strategy to reduce the program costs, although the numbers indicate as well that such a move would have an impact on who would choose to participate. The question of attendance allowances needs to be seen relative to the financial needs of participants. In this context, the $10 per day attendance allowance (approximately $200 per month) is a significant amount of money for potential participants, many of whom live within very modest means. Attendance allowances can also be seen as a celebration of the involvement of people in IIP projects. A large percentage of IIP participants start from low levels of educational attainment, life skills, and often self esteem. This makes it important that their participation is encouraged and supported in many ways which are meaningful to them. Celebrating their involvement by means of an attendance allowance can be seen as part of this encouragement. The design feature of having the projects pay attendance allowances should be considered in the context of social allowance payments. The study team was told by several respondents that there was a marked difference between the generally positive and respectful atmosphere of their project and the "take-a-number" approach to the payment of their social allowance. These respondents see their IIP project as a haven from the disrespectful world in which they find themselves. The IIP handbook suggests that social allowance cheques are distributed at project sites, but this does not always occur. This anecdotal evidence suggests that IIP projects could be further enhanced by insisting that the social allowance payment be part of the program, thus creating a training and personal development experience for the participants that is considerably more positive than the alternative, i.e., drawing social assistance. Paying social assistance within the framework of the projects would reinforce the positive celebratory aspects of the attendance allowance. Conclusion Although the survey results indicate that the attendance allowances are not a central feature for participants, the study team nevertheless does not draw the conclusion that they should be reduced or removed from the project design. Attendance allowances are both a considerable financial boost for many persons on social assistance and a concrete indication of the value that the system places on participation.
4.3 Impact The study team looks upon IIP's impacts as the most important aspect of the evaluation. Did the IIP projects affect the participants and their communities in a positive manner? Can it be demonstrated that IIP lessened the dependence on social assistance?
4.3.1 Impacts on Participants Background IIP projects deliver a range of services to participants including academic upgrading, life skills training, counselling, and work experience. Did these interventions help the recipients? Findings Information gathered from key informants shows that some 90% of the 132 respondents see the projects as good vehicles for delivering skills training and work experiences. Fewer respondents, slightly less than half, believe that the projects addressed adequately the personal and family needs of the participants. These general observations are supported by the findings of the graduate and exit surveys, which address impacts from the perspective of the participants. Table 4.3.1 provides the percentage of graduates who agreed or strongly agreed with a number of statements that relate to project impacts. The table orders the lists on the five highest scoring impacts among graduates. The table presents as well the perceptions of exits about impacts.
Table 4.3.1
The information in the table indicates that the projects helped people, both exits and graduates, to feel better about themselves and their abilities, to improve their education and to look for further training. Not unexpectedly, graduates consistently show a higher level of agreement with the impact statements posed to them as compared to the exits. The difference was statistically significant for a number of them. It is important as well to identify the lowest scoring impacts, presented in Table 4.3.2. The table shows that, although the projects made an essentially positive impact on the dimensions shown, the lowest scores appear to cluster around personal and cultural impacts. In addition, the impact on job searching skills is rated relatively low. Further insights into the project impacts can be gained from information on the most and least useful parts of the program as identified by respondents. Tables 4.3.3 and 4.3.4 indicate that graduates saw work experience and classroom training - both very skills oriented - as the most useful. They perceived the personal counselling and life skills training - both more person-oriented - as the least useful parts.
Project impact can be measured as well by the activity of the participants since their involvement with the projects. Table 4.3.5 shows that of the graduates, 52% (n=100) have applied for further training and 67% (n=129) for employment. The corresponding numbers for the exits are 42% (n=63) and 65% (n=97), respectively.
Finally, interviews conducted in the context of the three case studies indicate the need for long-term funding for projects such as those sponsored by IIP and this resonates with the experience of the study team. The need for a continuous training and career development path for people is an old theme, but one that has not lost any potency. Many IIP participants will not be able to find appropriate training opportunities in their communities now that the initiative is essentially completed. Conclusion The findings presented above indicate that participants were generally very positive about the impacts of IIP projects with respect to the key objective of the initiative: moving people towards reduced dependence. However, the findings, at least in their absolute numbers, should be interpreted with caution because the information presented relies on self-evaluation and is not based on standardized measurement. Some 90% of key informants indicate satisfaction with project impacts related to skills training and work experience. Only about half of the key informants say that the project addressed adequately the personal and family needs of participants. The survey results support the observations of key informants, with the proviso that participants see high impacts in the area of personal development and relatively low impacts with respect to job search related skills. Taken together, these findings suggest that IIP was relatively more successful in improving skills than in dealing with more personal issues. Participants who remained in their IIP projects experience greater benefits than exits, and a high percentage of both graduates and exits indicate continuing impacts as evidenced by seeking further training or employment. All these are clearly indicators of the success of IIP. The relatively lower impact of IIP projects with respect to personal development as compared to skills acquisition relates in part to the very common practice to use instructors from outside the community or region who work under short term contracts to do most of the instruction. This practice does not facilitate the trust relationship needed before effective counselling can take place. In addition, participants may perceive a conflict between the roles of counsellor and instructor, reducing the likelihood of positive counselling impacts. Finally, it needs to be recognized that many ABE instructors are not trained life skills coaches or counsellors. The fact that graduates see personal counselling as the least useful component may relate to the coordination with other agencies that this activity entails. In general, IIP is stronger where it relies on internal resources than where it provides referrals to other programs or support services. On the positive side, the high satisfaction rating of work placements and classroom training may relate to the widespread notion that taking a training course is "a job". Work placements and the attendance allowance reinforce the "job" aspect of training, and thus strengthen the positive view of short term courses. IIP is a 2-year pilot project and this time frame is too short to assess the likely long-term impact of the initiative on the social assistance dependency of participants. However, the time certain nature of IIP does not reduce the very real possibility that the lack of ongoing employment, education and training opportunities for many former IIP participants may negate over time many of the positive results of the initiative. This possibility was keenly felt by many respondents.
4.3.2 Impacts on Communities Background Communities feel the impact of IIP on a number of different levels. Most importantly, the communities feel an impact because community members gain in education and work experience. Community impacts relate as well to the general outlook of the community on the success of the projects, their contribution to local economic and social development, and the fostering of greater inter-agency collaboration within communities. Findings The 132 key informants are by and large positive about the IIP projects in their communities. Around 90% of key informants feel that the projects fit well with the needs of the communities and that the participants did gain knowledge and skills that they will use in their communities. In addition, about two thirds of key informants report satisfaction with the level of cooperation between all agencies and organizations involved in the projects. In general, it appears that IIP is a positive experience from the perspective of the communities. With respect to the inter-agency cooperation, the study team heard mixed evidence. Most projects experience inter-agency cooperation at the outset because CSSWs, CDOs and project sponsors are all involved with the selection of participants. There is evidence of some IIP committees remaining active throughout the project, but there is evidence as well that many committees became dormant soon after the start of the projects. IIP projects create more linkages between the work and training sectors than many other government services by explicitly including work placements. The study team did not hear of extensive difficulties in finding workplace hosts and about three-quarters of key informants were satisfied with the willingness of local employers to act as hosts and two-thirds with the benefits that the hosts received from participating. That said, less than half of the key informants indicate satisfaction with the level of community and elder involvement in the projects. More community control was a very common theme among aboriginal key informants and many others as well. In addition, only about a quarter of key informants express satisfaction with the number of participants relative to the number of people on social assistance. Another impact on communities relates to the equipment that some projects purchased, especially those projects that had a strong traditional skills component. In this context, IIP increases the resources, such as camping equipment, in some communities. Interviews in the context of the case study of Iqaluit-based projects indicate some uncertainty about the future of such equipment and suggest that it may deteriorate quickly if put in storage. In that case, it would not be seen as a community resource and maintained adequately. However, the relevant contribution agreements spell out that equipment stays with the sponsoring organization. Conclusion IIP is a successful program from the standpoint of the communities. It presents an integrated package of services and forces inter-agency cooperation, at least at the outset. All of these aspects are positive. IIP stimulates some inter-agency cooperation and networks between training providers and workplace hosts. This is meaningful in itself and likely contributes to some positive shift in attitudes of employers towards social assistance recipients. It should be noted, however, that networks and inter-agency cooperation need meaningful work at hand to reward the time and effort it takes to maintain them. In this context it is unclear how long these benefits will remain without other programming that requires and inspires this level of cooperation. The positive community impacts of IIP notwithstanding, only about a third of key informants express satisfaction with the level of community involvement with project planning and less than half say that they are satisfied with the level of community involvement generally. To their credit, IIP projects are generally in line with community needs and the WAPs did reach out to community-based organizations. However, they did not come forth from the communities at large.
4.4 Program Costs and Benefits This final section of the evaluation findings looks at IIP from a financial and cost-benefit standpoint. It looks at the narrow question if the IIP projects have contributed to a reduction in spending on social assistance allowances. 4.4.1 Cost per Participant Background The IIP Year 1 evaluation presents estimates of the expenditure per participant of less than $5,000 on average, ranging from slightly over $6,000 per participant for WAPs and almost $4,300 for NSDPs.22 Although that report cautions with respect to the interpretation of these results, it appears appropriate to generate the same statistics for the second year of the IIP initiative. Findings In principle, the cost per participant is a very straightforward calculation dividing the total project costs C estimated at $4.8 million for the fiscal year 1995/96 and $1.2 million for the fiscal year 1996/97 C by the number of participants in the projects that were active in that period. There are, however, a number of complications, including:
With these caveats in mind, the direct project expenditure per participant is estimated at between $5,800 and $6,100, with the cost per graduate roughly twice as high. These numbers are marginally higher if IIP management costs and some support activities, such as the ABE evaluation, are included. As presented in Table 4.4.1, the cost per participant for NSDPs is higher than for WAPs. Table 4.4.1
The estimates of expenditure per participant for WAPs are in the same order of magnitude as the Year 1 estimate of less than $6,000, especially if IIP head office costs are taken into consideration.23 The estimates of per participant expenditure for NSDPs in year 2 is considerably higher than for year 1. The analysis suggests two reasons for this increase in per participant costs for NSDPs:
Conclusion The findings presented above suggest that the expenditure per participant for WAPs is roughly similar in Year 2 as compared to Year 1. With respect to NSDPs, the expenditures are 50% higher. This increase is in part related to a higher per seat contract cost in Year 2 as compared to Year 1 and a relatively lower participant turnover. These findings should be interpreted with extreme caution due to the considerable uncertainty about the actual number of graduates and participants. The estimated cost per participant as well as the estimate of reduction in social assistance payments, discussed below, must be understood within the context of IIP=s impact on other programming costs. Survey findings suggest that many IIP graduates and exits hope to pursue other training options in the future, which means that participation in IIP may increase the system-wide costs, at least in the short to medium term. Insofar as IIP participants would not have pursued educational opportunities in the absence of the initiative, the success of the project (i.e. further training) makes it more expensive. Anecdotal evidence from the key informant interviews suggests as well that participation in IIP may stimulate demands on other community resources, especially in the area of counselling. If this indeed occurs, it would be another example in which project success would lead to system-wide larger expenses. 4.4.2 Reduction in Social Assistance Draw Background Sections 4.1 through 4.3 of this report provide considerable detail about many of the benefits that the IIP initiative conveys upon participants and their communities. IIP may convey benefits as well on the level of policy making, insofar as the lessons of IIP can inform the future programming. No attempt has been made to quantify these benefits in monetary terms. This section investigates the benefits of IIP in terms of the impact on the amount of social assistance allowances that IIP participants receive. The benefits, thus defined, are then compared with the costs of running the projects. A simple example may clarify the procedure. If an IIP participant, who drew social assistance before and during his or her involvement with IIP, finds a job upon completion of a project - and it is recognized that IIP does not set itself this goal - then the initiative could claim that the reduction in social allowance is a benefit of IIP. Assuming that he or she drew on average $500 per month in social assistance prior to participation in an IIP project, it follows that, all else being equal, the intervention with an average cost of $6,000 would pay for itself in a year. Findings The study team requested information on the amounts of social assistance drawn by 884 participants of Year 2 projects. The social assistance information system was able to match 155 of these names (or 17.5%) with names in its database and provided the social assistance payments to these persons from April 4, 1994 though March 31, 1996. This relatively low level of matches is in part due to:
The analysis of social assistance payment reduction was conducted on these 155 cases. The IIP benefit was determined by subtracting the average social allowance payment after the persons' involvement in an IIP project from their average draw immediately before the project. As shown in Table 4.4.2, this benefit is estimated at between $1.50 and $40 per month per participant.24 To estimate the total benefit, these estimates have been applied to our high estimate of the total number of participants (950) and extended over five years. This latter estimate is arbitrary and for illustrative purposes only. Table 4.4.2 shows the resulting range of benefit-cost ratios of between 0.02 and 0.42, indicating that IIP does not pay for itself in reductions in social allowance payments.25 Table 4.4.2
2. Assumes 950 participants and a 5-year time horizon. 3. Direct project costs only; excluding headquarters and ABE evaluation costs.
These results should be interpreted with extreme caution, mostly because of the assumption that the benefits would last five years, the small sample of participants on which information was available and the very short time that expired since the end of many projects. Given the limited training and job opportunities in many communities, it will take some time before participants who want employment or further training find appropriate opportunities. Another way in which to look at the costs and benefits of IIP is to calculate average reduction of social assistance draw by IIP participants necessary to pay for the cost of the program over a selected time period. Table 4.4.3 presents the results of this calculation and shows that IIP would need to cause a 20% reduction in the social assistance payments to the, say, 950 IIP participants to pay for the $5.5 million cost of the project within 5.4 years.
Table 4.4.3
Conclusion The findings of our analysis suggest that the impact of IIP on social assistance payments is insufficient to pay for the project within a reasonable time period. Even the most optimistic estimate of impacts - an estimate that is based on a small sample of cases - translates into a 12 year repayment period. The study team notes that this very long repayment period cannot be interpreted as a failure of the program. IIP did not set out to move people into jobs; its aims were more modest and refer to moving people along towards reduced dependence on the social safety net.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[To Top]](/web/20060127020057im_/http://www11.hrdc-drhc.gc.ca/edd-img/top.gif){kind=small}
![[Previous Page]](/web/20060127020057im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060127020057im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060127020057im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}