Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

This section presents similarities and differences in the delivery structure and outcomes between the three models investigated. The strengths and weaknesses of each model are discussed. Lastly, the usefulness of each model for achieving specific outcomes is outlined.

4.1 Comparison

This section presents the similarities and differences between the three models examined.

4.1.1 Delivery Structure

Exhibit 21, illustrates the similarities and differences in the delivery structure of each site. The Lethbridge and York GIS models closely resemble each other whereas the Mississauga model is quite different. The Lethbridge and York GIS are about an hour long with an internal staff presenting information to the claimants. There is very little interaction in these groups. The Mississauga GIS is an interactive day long session presented by facilitators from a third-party deliverer.

4.1.2 Cost and Savings Figures

The cost of delivering the GIS in Lethbridge, York and Mississauga (Exhibit 22) was estimated at $60,900, $81,047 and $275,459, respectively for the April 1 to December 31, 1997 period. The lower cost for the Lethbridge GIS is likely due to the fact that they only conduct an average of 8 sessions per month, compared to 60 in York and 25 day-long sessions in Mississauga (the equivalent of 125 one hour sessions).

Exhibit 22
Summary of Cost and Savings Information
(April 1 to December 31, 1997)

Mississauga has slightly higher total and net savings figures than Lethbridge, who is higher than York. Although the total gross and net savings are comparable across all three sites, Lethbridge has a higher savings per scheduled client value than both Mississauga and York. One obvious reason is that the number of clients scheduled is much lower than in Mississauga and York. Other reasons may also be related to the level of intensity associated with the GIS:

• I&C; investigators review client files during a GIS session, which allow them to question any inconsistencies or omissions that might lead to a D&D.; Claimant file reviews are not a part of the GIS at the other two sites;

• Participants are asked to complete and bring a job search record and an availability questionnaire to the session. Those individuals with incomplete or insufficient job search efforts or questionable availability to work are directed to a one-on-one interview that might lead to a D&D.; Although questionnaires are completed in York and Mississauga, neither of them ask for a job search record or ask participants about their availability;

• I&C; investigators are responsible for both conducting the sessions and the investigations, whereas York deliverers are Employment staff and Mississauga contracts the delivery of the GIS to a third party; and

• The Lethbridge office has developed a system to capture GIS savings to ensure that the savings are captured and reported accurately, whereas the other two offices are calculating savings manually or from various systems.

4.1.3 Impact on Clients

Participants in the GIS who were observed while conducting the site visits were administered an exit questionnaire. The questionnaire included questions on perceived usefulness of, and satisfaction with, the session participants had just attended. Presented in Exhibit 23, are the results to the question that asked, "On a scale of 1 to 5, where 1 = "strongly disagree" and 5 = "strongly agree", please tell us how you feel about each of the following statements."

Impacts of the GIS on participants are more pronounced in Mississauga. It had the highest percentage of respondents who agreed with all the statements, except one. Compared to York and Lethbridge clients, those in Mississauga generally reported lower awareness of their EI rights and obligations. This is not a surprising result given that, unlike the sessions researched in Lethbridge and York, EI rights and obligations is not a topic that is covered in detail by the Booster Sessions. However, focus group participants who had attended a Jump Start Session (held prior to the Booster session) usually reported being well aware of their EI rights and obligations.

Impacts on people's behaviour and attitudes as a result of attending a GIS were more difficult to gauge but apparent in some cases. Because of the interactive nature of the session in Mississauga, and its large focus on job loss and job finding, greater types of impacts were noted there. Focus group participants mentioned changing their résumé as a result of the information they received from the session. Others reported that as a result of attending the Booster Session, they were now considering contract work, while some had changed their view about the labour market. There was also an impact on people's knowledge of where to go to obtain more information about the services available to help them find a job, especially in Mississauga. In this location, a number of participants mentioned that they had been using new avenues and services to look for employment since attending the Booster Session.

In all three sites, there were indications that the GIS had had an impact on people's morale. More than in any other place, the Booster Sessions in Mississauga had a positive impact on people's level of morale, self-esteem and self-confidence. The majority of focus group participants confirmed that the Booster Session had given them a lift that made them feel better about themselves and renewed their motivation in their job search. Similar comments were heard in York, where participants indicated that attending the ROS had made them feel better about their situation because they realised that they were not the only ones unemployed. However, comments were also heard to the effect that the session made people feel more alone since they felt that the HRCC would not be able to help them any further.

In rating the sessions, some focus group participants stated that their rating depended upon the objectives of the session. Participants in York stated that if the objective was simply to provide claimants with information on rules and regulations, and inform them of where to look and go for assistance, then it would receive a high rating. Those who were interested in particular programs or specific information found it less useful and rated it lower. In Lethbridge, the perceived usefulness of the session depended on how much participants already knew about EI. For example, first-time claimants reported learning a lot and appreciated the fact that time had been taken to explain EI to them.

Comparisons on impacts between focus group participants who had attended a GIS and those who had not were difficult to make (because of the small number of individuals who attended the non-participant focus groups in Mississauga and York). However, in Mississauga, non-GIS participants generally seemed less aware of the various avenues and possibilities open to them in their search for employment than were participants who had attended a Booster Session. In Lethbridge, survey results indicated that GIS participants appeared to know more than non-participants about the services available to help them find a job.

4.1.4 Satisfaction

HRCC management and staff in all three sites studied noted that they were very satisfied with the results which have been achieved as a consequence of introducing the GIS at their respective HRCCs. Reasons for satisfaction included an improvement in overall claimant knowledge, including better understanding of their rights and obligations, more accurately completed Claimant Reporting Cards and other forms, improved detection of abuse and misuse of EI (e.g., increased disentitlements, voluntary disclosures), improved client motivation to seek re-employment, and the potential to achieve savings. Other cited benefits of the GIS included their ability to offer an opportunity for exchange of information, answer claimants' questions and guide them back on track towards re-employment. As an example of a strong statement of satisfaction, in Mississauga the sessions were reported by the manager as being "the most positive interaction the HRCCs have ever had with their clients." Lethbridge staff see the GIS as "good quality service to clients" which they "can't do without." HRCC managers in both York and Mississauga placed an emphasis on the GIS' use as first and foremost an information and employment intervention - not a control or detection mechanism. In Mississauga, this message has been strongly communicated, internally and externally, in order to overcome the perception that "I&C; is out to get people."

The exit questionnaire that was administered to GIS participants to collect feedback on the session asked the question " On a scale of 1 to 5, where 1 = "very dissatisfied" and 5 = "very satisfied", please rate the following components of the GIS." The results of this question are presented in Exhibit 24.

In general, focus group participants in York and Lethbridge tended to be less satisfied with their participation in the GIS than were those in Mississauga. In Lethbridge, almost all focus group participants stated that the session seemed to have threatening undertones. A number of participants saw the sessions as a "hoop to jump through" in order to maintain receipt of their claims. It is interesting to note that while focus group participants in Mississauga generally expressed higher levels of satisfaction with the GIS, the only area with which survey respondents were noticeably more satisfied was their level of participation.

Note: The total number of respondents is not the same for all of the components rated because not all respondents answered all questions.

Specific areas of satisfaction for participants in Mississauga included the motivation that participants got out of the session as well as specific tools and approaches they could use in their job search. Many participants also expressed high levels of satisfaction with the Booster facilitators, whom they described as knowledgeable, outgoing and energetic, and supportive. This view was different from that expressed in York and Lethbridge. Although no negative comments were made about the facilitators' tone or attitude in York (the facilitators were described as pleasant), participants found the ROS' delivery to be somewhat "mechanical" and "programmed" because it was tightly scripted. In Lethbridge, facilitators were viewed as "threatening", "intimidating", and "condescending." Beyond how the facilitators were viewed in Lethbridge, participants' level of satisfaction with the I&C; GIS session was also impacted by the fact that one I&C; staff was at the back of the room collecting forms and reviewing files. This was sometimes viewed as more intimidating than the moderator. Participants found the calling of the names of persons who were asked to either remain behind or allowed to go to be intimidating as well.

Mississauga participants expressed satisfaction with the ability of facilitators to answer their questions. However, in York and Lethbridge, participants were not as satisfied. In York, questions could only be asked at the end of the session. In Lethbridge, some clients said they did not ask questions because the deliverer appeared rushed and just wanted to get the session over with as fast as possible. In Lethbridge, focus group participants who had used that route were satisfied with the facilitators' ability to answer their questions and indicated that they were friendlier when spoken with one-on-one than when speaking in front of the group.

4.1.5 Achievement of Shaping the Future Objectives

The objectives of the Shaping the Future initiative are the following:

• introduce active interventions to help implement behavioural changes in employees and workers;

• assist individuals to return to work more quickly;

• reduce EI dependency; and

• protect the integrity of the EI account.

Assessing the extent to which the three models are achieving these objectives is difficult to do given the nature of the objectives and of the research questions developed for this study. However, the findings of the monitoring study indicate that of the three models, the Booster Sessions are most effective at having an impact on the behaviour of clients in the area of employment search. However, given the focus of the York and Lethbridge models on EI Rights and Obligations, these two models are more likely to have an impact on behaviours such as the accurate completion of cards or preventing fraudulent use of the system.

The impact of the GIS on assisting individuals to return to work more quickly cannot be assessed with certainty using the findings from this study. However, to the extent that clients in all three sites are provided with information on available programs and services that can help them in their job search, it would be expected that the GIS would have some impact in this area. The Mississauga model, in particular, would be expected to contribute to the achievement of this objective given the site's objectives of motivating clients to maintain their job search and make it as effective as possible. A few instances were indeed given of clients who had taken steps to accelerate their job search or found work as a result of the Booster Sessions.

Assessing the extent to which dependency on EI is reduced is a very complex endeavour that can only be determined with longitudinal data, and is therefore out of the scope of this study.

The findings related to the number of D&Ds; imposed as a result of GIS-related activities clearly indicate that all three models play an important role in protecting the integrity of the EI account.

4.2 Strengths and Weaknesses of the Models

The strengths and weaknesses of the three models in terms of delivery structure are summarised below.

• The most evident strength of all three models is the interaction integrated in the Mississauga model. Clients are very satisfied with their level of participation and the amount of hands-on activities in the Booster Session. Lethbridge's and York's straightforward presentation of information can therefore be considered a weakness. Even though Lethbridge does not read a script to the claimants, there are no group activities and participants are hesitant to ask questions during the presentation.

• Another strength of the Booster Session is that the deliverer has flexibility to better meet client needs. Booster Session deliverers do not follow a script or slide presentation but rather a general outline of the day's agenda, so they are able to answer questions on the spot and address any concerns right away. Lethbridge, and especially York, are concerned about the consistency of the information presented, so the deliverers try to stay as close as possible to the written script.

• On the other hand, the Lethbridge, and especially York model would hold better in the case of an appeal because all clients receive exactly the same information.

• A weakness at the York session is that deliverers present many sessions a month (60 session for two deliverers). This may contribute to fatigue and disinterest among deliverers. Lethbridge only presents eight sessions a month. The rest of the deliverers' time is spent on investigative activities.

• On the other hand, the large number of sessions conducted in York and Mississauga can also be viewed as a strength, because it allows them to see more claimants. York presented GIS to 9,380 claimants, Mississauga conducted the Booster Session with 5,919 whereas Lethbridge only saw 1,218 claimants through the GIS between April and December 1997.

• The scheduling process of the York model is a strength. It schedules and gives claimants information on the GIS when they apply for a claim. This process eliminates the need to manipulate databases to select clients and mail packages out to them.

• Both Mississauga and Lethbridge conduct follow-up with GIS clients after they attend a session. By following up with claimants, staff are better able to help them identify the avenues that can lead to employment. York's weakness is that there is no follow-up work (outside of investigation activities) conducted with GIS participants.

• A weakness witnessed across all three sites is the intimidating tone and lack of information on the letter directing clients to attend.

• Objectives of the GIS are very broadly defined for the three models. Broad objectives make it difficult to assess specific outcomes against these objectives.

• Currently there are no formal mechanisms for client feedback or evaluation of the GIS in York and Lethbridge. This is a weakness because GIS can be improved with client suggestions. While the third-party deliverer asks for client feedback, this feedback is not anonymous, limiting the opportunities to hear honest comments.

• The York GIS model was designed by both I&C; staff and Employment staff. This is a strength because it ensures better coverage of EI initiatives and avoids duplication of efforts. It also strengthens relationships between the two staff groups. Lethbridge, in retrospect, would have liked to have involved Employment staff with the GIS.

• The fact that Mississauga has a third-party delivering the session is also a strength. Mississauga is able to use their resources in other areas and it strengthens relationship between community partners.

The strengths and weaknesses of the three models in terms of cost and savings are as follows.

• A strength of all the models is that they have all achieved savings.

• Lethbridge has achieved higher savings per client and per disentitlement than the other two sites, most likely because of the intensity directed toward investigation activities.

• A weakness of all three sites is that there is insufficient capturing of accurate GIS-related data (number of scheduled clients, direct and indirect savings, etc.), which makes it difficult to evaluate or revise the GIS strategy. The Lethbridge HRCC is the most advanced in this area.

The strengths and weaknesses of the three models in terms of impacts and satisfaction are:

• A strength of all the models is that there is a definite educational impact on GIS participants.

• The GIS has helped deter fraud of the EI system. All three sites have identified hundreds of disentitlement and disqualification cases as a result of the GIS.

• Mississauga has the most pronounced impact on clients.

• Satisfaction of the GIS is high in all three sites.

4.3 Conclusion

The table below provides a starting point to identify which GIS model an HRCC might consider implementing given a set of specific circumstances or desired outcomes.

Situation	Lethbridge	York	Mississauga
Tight labour market, with few opportunities for employment
HRCC has limited resources to deliver the GIS
HRCC has no staff available to deliver GIS
HRCC has to serve a clientele located in remote or rural areas
HRCC has a multitude of partnerships with community deliverers which it wishes to promote
Under-utilization of Employment Benefits and Support Measures by claimants
Suspected high level of fraudulent activity
HRCC has limited access to meeting or delivery space
Desire for improved identification of claimants who would benefit most from Employment Benefits and Support Measures
HRCC desires increased linkages between the activities of I&C; and Employment
Desire for improved ability to detect fraudulent activity
Desire for improved ability to successfully win claimant appeal cases
HRCC desires high savings results