Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Nova Scotia Compass has been successful in achieving its principal objective: to reduce reliance on social assistance. Use of social assistance fell by 22 to 30 percentage points for participants as compared to non-participants. There is some evidence that the impact was diminishing by the time the survey took place, however, raising questions as to the permanency of the effects. Assuming the initial impacts hold over time, it will take Compass about six years to break even, about the same as similar welfare reform efforts in the United States.

By way of summary and conclusion, the balance of this closing chapter will present our capsulized response to each evaluation question.

RELEVANCE

1. In what way does Nova Scotia Compass reflect the criteria established for Strategic Initiatives (SI)?
- innovation/experimentation potential?
- relevancy to SI objectives?
- evaluation/information potential for social reform, etc.?

Some interviewees questioned the uniqueness of Compass, saying it was similar to what was already being done through the ERCs, that key components of Compass such as the wage subsidy were already in use elsewhere, and that EDO was similar to HRDC's Self Employment Assistance (SEA) Program. On the other hand, several informants believed that the role of the job developers was innovative, and extremely important for the success of Compass. Also mentioned as innovative was the Opportunity Fund.

Compass did address Strategic Initiative priority areas; i.e., employment, learning and education, training, and income support in order to boost employability and lower social costs. And given the wealth of findings that have emerged from this evaluation, Compass realized its extensive information/experimental potential.

2A. To what extent does the project reach the intended target group? Are participants representative of the target group? If not, for what reasons do discrepancies occur?

WEO reached its primary target group - inexperienced youth - but over a fifth of its clients were not in the planned target group, being over 30 years old. This occurred because ERCs had no other suitable options to offer inexperienced clients over age 30.

The program had difficulty reaching the primary designated target group for TTO: job-ready clients from the Family Benefits program. The main problem was getting suitable referrals.

2B. To what extent have SARs accessed employment using the services of the Job Developer but without a wage subsidy? Do Compass participants in subsidized jobs differ from those in unsubsidized jobs?

Most job developers said that they had placed clients in jobs without the wage subsidy, although the numbers were small. Actual numbers were hard to estimate (one job developer guessed 5%, another 20%). The administrative data suggest the lower estimate is more accurate: 103 cases found an unsubsidized job after referral to Compass. It is not possible to say how many of these 103 cases were referred to the employer by a job developer.

All job developers felt there were no systematic differences between clients who received the wage subsidy and those who did not. A comparison using administrative data of Compass participants with unsubsidized clients substantially supports this sentiment.

3. To what extent have the recommendations from the process evaluation been implemented? What changes have taken place as a result of the recommendations?

For several recommendations, actions taken by the time the process evaluation final report was published adequately dealt with the issues of concern and no further progress was needed. Thus, the EDO loan limit was increased to $5,000, the ratio of FB to MSA cases under TTO was amended, and WEO was modified to provide the base minimum wage to participants rather than the $160 allowance. In others the Management Response promised that management would deal with the problem. For the most part this was the case.

PROJECT SUCCESS

4. How satisfied are SAR participants with various aspects of the project? To what extent did participants discontinue before their anticipated completion date? What were the main reasons for discontinuation?

Compass clients were very satisfied with Compass, awarding it an average grade of B +. Moreover, every individual facet of Compass (e.g., the placement, help provided by job developer) rated a B or better, except for guidance on services available after Compass, which rated a B-. Clients were especially happy with the help they received from the job developer.

The discontinuation rate among clients was about 16%, much lower than other welfare reform programs have experienced. The greatest proportion, about 20%, left the program involuntarily: they were laid off or fired by their placement employer. Another 15% found a job with another employer.

5. How satisfied are employer participants with various aspects of the project? Do employers offer employment opportunities to participants upon successful completion of the placement? For what reasons do employers offer or not offer employment?

Employers were very pleased with Compass, assigning it an A - grade on average. They were particularly happy with the service they got from the job developer, and with the level of the wage subsidy. Employers evinced some displeasure with the quality of the employees referred and with employees' work attitudes, grading both aspects a B -.

Over half of the TTO client employers failed to keep their commitment to hire the client once the subsidy expired: only about 45% of TTO clients continued to work with the placement employer after the placement. Between a quarter and a third (depending on the source of evidence) of WEO respondents stayed with the placement employer after the subsidy ended.

About half the TTO employers who did not keep their commitment to hire said there was no position available or no money. Many employers also claimed that a lot of clients had a poor work attitude - which was confirmed by job developer - and just "didn't work out." Employers gave two main reasons for continuing to employ clients: they were good workers, or they were now trained for the job.

6. Has the project brought about any changes in participants' home/family life?

There was no significant difference between participants and non-participants in the change in the level of satisfaction toward social life with friends and relatives, education received, and life in general. There was, however, a modest decline in participants' satisfaction with family life (relative to the change for non-participants) and a modest increase in participants' satisfaction with "the work you have done in your life."

7. To what extent has the project prepared participants for achieving economic self-sufficiency?

1. increased their motivation and self-esteem?
2. assisted in the development of a career action plan?
3. improved their job search skills?
4. upgraded educational skills?
5. provided them with occupational skills?
6. provided them with pre-employment orientation?
7. provided them with work experience?
8. provided them with self-employment/business skills?
9. provided them with mentoring/role models?

Compass participants were satisfied that the program had prepared them well for achieving economic self-sufficiency. All but three of these various aspects were graded B or higher on average. One of the aspects receiving a lower grade - upgraded educational skills - was not an objective of Compass. But helping participants to find a permanent job was, and it was given only a C+ average. Nearly a quarter of the respondents gave Compass a failing mark in this respect, most of whom had not found a permanent job. EDO clients expressed some dissatisfaction with their role model, giving a C + average: a quarter gave an F.

8. To what extent has the project assisted participants in each component of Compass to achieve economic self-sufficiency?

1. What activities/interventions were most effective? For what type of participant? For completers/non-completers?
2. Why do some participants remain unemployed and on income support after the project?
3. To what extent and why do participants remain employed following participation?
4. Did the project motivate participants to go on to further training or education?

There is evidence that the Compass program reduced reliance on social assistance by 22 to 30 percentage points. In part because Compass participation helped qualify clients for UI, the program seemed to increase reliance on UI, at least in the short run. Although descriptive data suggested that Compass may have had an impact on employment earnings of participants, econometric analysis did not support this conclusion. Most of the estimates were positive, but not significantly different from zero. There were probably too few cases with available data on post-program earnings to reach significance.

Job developers pointed to three general factors that keep some participants unemployed after Compass participation. One was characteristics of those participants who remain unemployed. Many were lacking the motivation to work: they simply had a poor attitude towards work. Many who stayed unemployed had barriers that could not be overcome such as lack of child care, and lack of money for job search (e.g., for transportation). Another key obstacle was lack of job search skills. A second general factor was the poor labour market, especially for those with few marketable skills. The third general factor was disincentives built into the social assistance system, especially the FB system.

The Compass program tended to increase participants' proportion of time spent working by more than 10%, decrease their time spent unemployed by about 10%, and decrease or leave unchanged their time spent in school.

9. To what extent are project activities and characteristics related to success? What types of project activities/program components are associated with improved employability and earnings, further education, self-employment, etc.

Impacts for TTO were larger than those for WEO in terms of decrease reliance on SA, increased reliance on UI, and time spent working. Both WEO and TTO had similar impacts on the proportion of the population employed, in the range of 18 to 20 percentage points. When the analysis is restricted to those who completed Compass 10 or more months prior to the survey, however, the estimated impacts of the WEO option were much larger - 26 to 29 percentage points - than those of the TTO option - 13 percentage points. These findings suggest that the WEO option had more lasting effects on the likelihood of employment than was the case with the TTO option.

In general, demographic traits had little influence on the results.

10. To what extent has the EDO succeeded in establishing new business? What is the survival rate of EDO sponsored businesses compared to that of businesses started without the assistance of EDO? Why did some EDO businesses fail? What factors are more likely to contribute to success?

There were 104 EDO clients. About 72% of EDO participants were self-employed at the time of the survey, although half this group was also on social assistance (63% were self-employed in the business they were developing while participating in the Compass Program). Of the 11 non-participants who tried to start their own business without the help of Compass, 55% were self-employed in that business at the time of the survey.

Clients' most often mentioned reason why the business is not operating (or never started) was "no market." Interviewees gave the following reasons for EDO business failures: training wasn't adequate; insufficient funding; insufficient follow-up; lack of markets for product; insufficient motivation; and personal problems.

Two general factors were said to have contributed to the success of EDO businesses. First was the flexibility of the program. For example, "Compass provided consulting assistance of up to $1500 to help clients access professional training -in lieu of Stream 1 as it was not equally accessible across the province." Another example was the increase in loan size to clients to $5000. "If clients who had already received the $2000 came to the office and indicated they were struggling they could re-submit and get an additional amount. There was some flexibility to re-open cases. The loan could not finance losses - only growth and development."

The second factor was the partnership arrangement between the ERCs and the Economic Renewal staff in the Business Service Centres.

11. To what extent are the observed results of EDO attributable to government funding? To what extent would EDO participants have initiated self-employment without the assistance of the initiative? Do non-selected clients go on to establish self-employment?

Most EDO participants needed Compass to get their business off the ground: 82% thought that they would not have been successful in establishing their own business without the help of Compass.

Eleven of the 12 EDO non-participants tried to start their own business without the help of Compass, and 55% were self-employed in that business at the time of the survey.

12. To what extent does the EDO result in direct job creation in addition to the self-employment of the SAR participant?

Three-quarters of businesses started through Compass and still surviving by the time of the survey had hired no other workers. In total, the 26 surviving businesses represented in the survey had hired nine full-time workers and three part-time workers. Generalizing these findings to the EDO population, it is estimated that the EDO program has helped generate up to 30 jobs, most of them full-time.

13. Is activity under Compass incremental (over and above that which employers would have done without program funding? Or is there evidence to indicate that placements would have occurred anyway? Or that participants are placed on jobs that would otherwise have gone to others? Did hiring take place sooner as a result of Compass?

About 60% of employers who would have hired without the subsidy said the subsidy spurred them to hire sooner than they otherwise would have. Half the employers that participated in Compass would have hired someone in the absence of the program. Since there was no significant difference in the number of employees hired through Compass between employers who would have hired and those who would not have, it appears that half the participants in Compass may have displaced others who would have been hired without a subsidy.

PROJECT COST-EFFECTIVENESS

14. Is the pilot project model a cost-effective way of achieving project objectives? Are there more cost-effective methods of achieving the same objectives? How do results compare with those of other programs with similar objectives?

Assuming the annual benefits hold up over time, the Compass Program will take about six years to break even. But this assumption may be optimistic given that the impact analysis found evidence that the initial effects on receipt of passive income assistance dissipate during the period following the program.

Most low-cost American welfare reform programs (with proper evaluations) broke even or produced positive returns to government budgets within five years of starting. On the other hand, higher-cost programs such as Compass, usually take longer to pass the break-even point⁵⁶.

15. What lessons can be learned from this project on interventions to assist the target group? How and to what extent does it contribute to the development of a policy framework for social security reform? Does the project lead to a more efficient delivery of services? To what extent can this project be successfully expanded or replicated in other regions/provinces?

Informants had plenty of advice for setting up a program like Compass, although no single recommendation stood out: none was mentioned more than twice and most had only one advocate. Some of the key lessons are:

• Take account of policy issues affecting client participation which could affect take-up. Most importantly, attend to disincentives to leaving social assistance such as loss of health benefits, and need for day care and transportation.

• Build in local decision-making.

• Don't make the wage subsidy automatic; ensure the employer needs it.

• Ensure adequate time for planning and implementation of the program.

• Ensure that field staff adhere to the eligibility criteria established for the program.

• Set up an effective partnership structure and involve all partners from the outset. Keep all partners well informed about the program and its progress.

• Ensure a good financial management system is put in place for proper monitoring of the program.

Footnotes

56	Friedlander, D. and J.M. Gueron (1992) Are high-cost services more effective than low-cost services? In: C.F. Manski. & I. Garfinkel (Eds.) Evaluating Welfare and Training Programs. Cambridge: Harvard U. Press.