Process Evaluation Ready-to-Learn Project - PEI - September 1995

Process Evaluation Ready-to-Learn Project - PEI

View Whole Report

• Introduction
• Project description
• Evaluation approach
• Key findings

This brief summarizes the evaluation findings of the Ready-to-Learn (RTL) Project, a Strategic Initiative jointly funded by the Governments of Canada and Prince Edward Island. The evaluation was managed by a Federal-Provincial Committee with representation from the PEI Office of Higher Education, Training and Adult Learning and Human Resource Development Canada, PEI Region and Evaluation and Data Development at National Headquarters.

The Ready-to-Learn Project is a four-year Strategic Initiative that offers counselling and training to 120 PEI youth who have low literacy levels and who are receiving unemployment insurance or social assistance.

During the summer of 1994 potential participants were invited to an information session and, if interested, to take a literacy level test. Participants were those meeting the eligibility criteria.

At the same time eighteen individuals receiving Unemployment Insurance (UI) or Social Assistance (SA) and with post-secondary credentials and/or experience in education were hired as 'facilitators', and trained at Holland College to provide literacy and life skills training, monitoring and counselling to the youths.

During the first stage (October to May 1994), community-based literacy training and support was provided by the facilitators in 10 communities to 15 groups of 8 participants. This training phase was targeted to grades 7, 8 and 9 levels. The goal was to raise skill levels in mathematics, reading and writing to allow the participants to enter a college preparatory program in the second year. At the end of the first stage, individuals got subsidized summer work experience for 12 weeks with local employers.

During the second year, individuals who attained grade 10 level entered the academic upgrading phase. Facilitators continued to give social and personal support to the participants. Those who didn't reach this level continued their literacy training in community learning centers and may join the college preparatory program when ready.

During the summer of 1996 the participants again will work in subsidized employment or on job training with local employers to gain experience. In the final year of the project, individuals will take skills training in Charlottetown or Summerside.

Project objectives

The stated objectives of Ready-to-Learn are to:

• test the effect that two years of small group tutoring has on the success of students in literacy training, academic upgrading, and skills training;
• test the effectiveness of having professional counselling help available to students;
• test the success of the social/interpersonal support provided in the small group settings;
• test the linking of community-based literacy training to institutional academic upgrading and skill training in a three year process;
• provide some tutoring and teaching experience to a group of UI/SA recipients; and
• test the success of summer work experience for the participants.

To address the evaluation issues five sources of information were used:

Document review

The major areas covered in the document review were: various statistical data on PEI; literacy programming and curriculum development, CAAT tests, community based training, target population needs, and skills training.

Analysis of administrative data

Data from the baseline survey of applicants, a baseline survey of facilitators, and an exit survey of drop-outs were analyzed to draw profiles of the participants, the non-participants, the facilitators, and drop-outs. An examination and assessment of the completeness of the monitoring system was also performed.

Survey of participants and non-participants

Participants were randomly selected from the target population. The remainders were the non-participants. The progress of both groups, participants and non-participants, will be followed to compare differences between those who receive training and those who do not (e.i. the control group).

The participant survey measured progress and satisfaction with various aspects of the project. Questionnaires were self-administered during class time. Of the 117 participants, 115 completed the questionnaire, for a response rate of 98%. Surveys with 67 non-participants were completed, for a response rate of 57%.

Key informant interviews

A total of 15 interviews were completed to assess how the program was being delivered; to determine informants' understanding of project objectives; to identify any major obstacles to achieving objectives; and to gather suggestions for improvement.

Focus groups

Focus group sessions were held with two groups of facilitators, two groups of participants, and a group of regional literacy experts.

The purpose of the focus groups was to discuss the appropriateness of program targeting, implementation problems; to gain a better appreciation of how RTL carries out its activities; to make an early assessment of satisfaction with services provided; to examine preliminary impacts on motivation and attitudes; and gather suggestions for making the program more successful.

Participant profile

RTL participants are split 50:50 by sex. All participants are age 30 years or younger with an average age of 23 years. About three-quarters are single. About 14% graduated from high school, but 22% never made it to Grade 9.

Most -96%- had worked for pay in the past. Low-skill occupations were the norm for participants, with labourers, sales/service workers and restaurant workers most common. On average, participants worked 9.6 months in the last two years. SA participants spent almost 12 of the 24 months prior to RTL on welfare and UI participants spent 11 months on UI during the same period.

Partnership building

Joint planning and management among officials of different levels of government is a necessary process of the Ready-to-Learn project. Everyone who expressed an opinion said that the relationship between the federal and provincial departments co-managing the project was good to excellent.

Unfortunately, in part due to time constraints when the project began, RTL was unable to build the bridges to ensure the cooperation of organizations such as local literacy organizations, employers and local communities.

Project implementation

Limited time to plan and put the project in place was the primary constraint. The second was legislative constraints, particularly the UI Act. These constraints caused policy makers to change the planned design in the following ways:

1. A significant minority of participants did not meet all eligibility requirements. Many could not read and required remedial reading courses. Others qualified for skills training before RTL;

2. Planned services were not suitable for some individuals. The curriculum was unsuitable for those who could not read, and for those already operating at a grade 10 level. Upgrading in the second year is irrelevant for those with high school; and,

3. The monitoring information was not gathered on a regular basis.

Participants were asked about their satisfaction with various aspects of RTL. Overall, they expressed a high degree of satisfaction with RTL; 46% said they were very satisfied and 48% said they were somewhat satisfied. Most participants (70%) said that they were very satisfied with their decision to take part in RTL. Another 27% said they were somewhat satisfied.

Overall, nearly all participants claimed to be very satisfied (49%) or somewhat satisfied (49%) with the services they had received through the project. Only 2% were not very satisfied.

The large majority of participants said they were satisfied or very satisfied with the help provided by the facilitator, for their level of skills and for their friendliness.

Reactions to career counselling were mixed. Participants who consulted the counsellors were very satisfied (18%) or somewhat satisfied (63%) with the help they received. However, 17 % were not very satisfied, or not at all satisfied (2%). Almost 40% had not talked to the career counsellor.

On average, participants said they were very satisfied with classmates and somewhat satisfied with the support provided by the groups. Students in the focus groups judged the size of the groups as very good and the cohesion and support among the groups as very important. Over the first six months of the program, individuals became significantly less positive in eight areas--mostly towards work and training --and significantly more positive in four areas, most notably in their regard for the education they had received.

Only 18 individuals had left RTL by early May. Facilitators argued that personal counselling and support was the reason most of the individuals stayed in the project. They were confident that they could predict those that were in danger of dropping out through personal counselling interviews.

Facilitators felt that they are gaining valuable experience and confidence in new areas, learning many new skills and developing a greater awareness of the difficulties of participants. They believed to be more employable with the skills and insights they were gaining. The skills also were said to be transferable to training levels such as junior high school.