Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

2.1 Background

Employment Insurance is intended to assist unemployed individuals by providing temporary income support, subject to specific rules and regulations, when earnings are interrupted. Claims are sometimes disallowed due to perceived infractions of the rules and regulations and, in those cases, claimants have the right to appeal the rulings. The appeal process is potentially four-tiered beginning with a Board of Referees followed by an Umpire, the Federal Court, and, ultimately, the Supreme Court of Canada. The evaluation pilot project dealt with the initial level of hearings before a Board of Referees.

The project had its genesis in field reports of a substantial increase in the number of rulings being appealed to Boards of Referees and, more specifically, the incidence of third-party representation of appellants. The Commission wished to explore the question of whether or not in-person representation on the part of the Commission would significantly affect the process and outcomes of these hearings before the Board of Referees.

In initiating the study, the Commission took into consideration the existence of a backlog of appeals together with a history of field requests for Commission representation. In order to assess the impacts and effects of Commission representation at Board of Referees appeal hearings, the Commission requested that an evaluation be undertaken. Recognising that a full benefit/cost analysis would not be feasible, the Commission and evaluators decided to proceed with an experimental pilot project. In 1995, Evaluation and Data Development developed an experimental project to be implemented in selected sites in order to test the impact of Commission in-person representation in terms of the appeal process, outcomes, various parties involved, and, to a limited extent, potential costs and benefits. The pilot comprised two distinct phases, a quantitative component based on the development of a statistical data base and a qualitative component based on focus groups, debriefing sessions, and key informant interviews intended to assess the non-quantitative aspects of the appeal procedures.

The purpose of the pilot was to determine the feasibility and efficacy of implementing in-person representation of the Commission in appeal cases where the appellant is represented by a legal/paralegal third party. The evaluation was designed to address a range of issues and questions pertaining to the potential effects, intended and unintended, of in-person representation. The broad study issues encompassed the questions of the positive and negative effects of in-person Commission representation, whether representation would make the appeal process more legalistic, and whether representation would rationalise the appeal process by bringing more information to the Board of Referees and enlightening judgements.

Specifically, the pilot was intended to determine:

• the direct effects Commission representation may have on the outcomes of Board of Referees' hearings;
• the possible costs and benefits of Commission representation, although it was recognised that a limited pilot project would not produce a full benefit/cost analysis;
• whether the presence of a Commission agent would affect the level of understanding of issues and the Commission's position;
• whether the presence of a Commission agent at the hearings would affect the proceedings themselves; and
• whether attendance at the hearings would affect the agents' understanding of the hearing process.

The EI Appeals pilot project featured random assignment of cases to treatment and control groups. With sufficiently large sample sizes, random assignment makes it possible to draw reliable inferences regarding effects on the target population.

In the pilot project, the target population included all appeals with legal/paralegal third party appellant representation in selected sites representing a cross section of the total number of potential sites. The sites originally identified to participate in the pilot project were Bathurst, Longueuil, Laval, Toronto (North York and Newmarket), Winnipeg, and Vancouver. Due to administrative difficulties, Bathurst did not participate and other sites were added, namely, Mississauga, Brampton, Victoria, and Prince George. However, the Prince George site did not submit data. Thus, nine sites effectively participated in the project as implemented.

The reference period was to consist of 12 months starting in late November, 1995. However, the sites that came on stream later, as well as the original sites, needed a longer period to organise the required activities. In an attempt to preserve the integrity of sample sizes and the resulting inferences, time frames were adjusted to ensure a minimum of one-year participation in each site.

Cases entered the sample as soon as the Commission received notification that the appellant was to be represented by a legal/paralegal third party. Considerable flexibility was afforded sites in adapting the implementation to their own requirements and circumstances. Based on a formula applied to the Social Insurance Number of the appellant, cases were assigned to either the treatment group (i.e. in-person Commission representation) or the control group (no in-person Commission representation). The one exception to this methodology was Vancouver, where, because of the logistics regarding the location of HRCC offices and hearing venues, the decision was made to assign cases to treatment and control groups according to the day of the week that the hearing was scheduled.

Other variations in implementation were also evident. For example, some sites chose to implement in-person representation on a voluntary basis, while, in other instances, representation was undertaken on a rotational basis. In Toronto, representation was handled by a specialised EI Appeals unit, while, in Montreal, one agent per site was responsible for in-person representation.

These variations in implementation influenced the observed impacts and made it more difficult to generalise results.

In the pilot project, evaluators collected data that could be used to contrast outcomes for the treatment group with those of the control group. One of the major considerations for the data collection was to not unduly increase the workload of operational staff and other involved parties. Both quantitative data and qualitative data were collected in the course of the investigation.

The quantitative data base comprised objective measures of events, dates, and duration as well as scaled ratings of more subjective measures such as opinions and impressions. These quantitative data were derived from multiple forms and survey questionnaires completed by all parties to the appeal process, including appellants, appellants' representatives, Commission managers and agents, and members of Boards of Referees. Forms were provided to respondents by the Clerks of Boards of Referees and were collected immediately after the hearing in order to minimise problems of non-response and recall.

In addition to the collection of quantitative data through survey questionnaires, evaluators incorporated an extensive qualitative component designed to provide broader and more in-depth insight into the potential impact of Commission in-person representation. This information was based on a series of focus groups, debriefing sessions, and one-on-one key informant interviews conducted in the sites participating in the pilot project. These sessions provided information on the perceptions of participants regarding the advantages and disadvantages of Commission in-person representation. Participants in the qualitative data collection sessions comprised Boards of Referees and their clerks; agents representing the Commission at hearings; agents who had made the original rulings and/or had prepared written submissions to hearings; appellants in both treatment and control groups; appellants' representatives; and CEC/HRCC managers and agents in participating sites.

The quantitative database was analysed, subject to the limitations imposed by sample sizes, in order to ascertain where significant differences occurred between the treatment group and the control group. Information from the qualitative data collection was compiled through detailed summaries of each focus group, debriefing session, and interview. A matrix of issues was prepared and each individual comment was recorded under the appropriate study issue. Finally, with careful attention to gauging the level of consensus and/or contradiction among respondent groups and participating centres, the information in the matrix was synthesised, issue by issue, into a report of the findings.

Findings from both the quantitative and qualitative analyses were examined to determine the degree to which one corroborated the other. Conclusions were based on a synthesis of findings.