Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

Evaluation findings cast some doubt upon the initial impetus for the project, namely, that there had been a "large" increase in the number of appeals and, specifically, the incidence of third-party representation of appellants. Administrative statistics suggest that only about 5 to 10 percent of appealed cases have paralegal claimant representation. After running the pilot in the nine selected sites for one full year, relatively small numbers of appeals were identified belonging to the target population.

Table 1 shows the total number of cases included in the pilot study. The total numbers involved are not sufficiently large to allow for detailed statistical analysis of individual sites. In order to provide reliable information, the minimum number of cases would have to be 80 in each of the treatment and control groups.

Source: EI Appeals Pilot Project Data Base

Table 1 Cases Completed by Site
Site	Treatment	Control	Total
Vancouver	44	38	82
Victoria	6	3	9
Winnipeg	35	30	65
York	12	12	24
Newmarket	10	4	14
Mississauga	12	3	15
Brampton	9	14	23
Laval	11	27	38
Longueuil	52	65	117
Total	191	196	387

For the purpose of analysing the results, data were aggregated into four areas, namely, Vancouver, Winnipeg, Toronto, and Montreal. Overall aggregate data were weighted to reflect the level of response in the individual centres in order to provide a more valid overview of the likely impact of in-person Commission representation.

In order to assess the impact of treatment on the hearing process, a series of questions were developed which rated respondents' opinions according to the following scales:

1.	Atmosphere of hearing	-5 (very tense) to   +5 (very relaxed)
2.	Understanding of Commission's position	0 (not at all) to 10   (very well)
3.	How presence affected understanding of Commission's position	-5 (confused) to   +5 (clarified)
4.	How presence affected understanding of relevant legislation	-5 (confused) to   +5 (clarified)
5.	How presence affected understanding of facts of case	-5 (confused) to   +5 (clarified)
6.	How presence helped focus on issues of case	-5 (confused) to   +5 (clarified)

These factors were addressed by appellants, appellants' representatives, and members of Boards of Referees. Table 2 presents the findings of the data collection.

N/A. Not applicable
* Statistically significant at 95% confidence level.
Source: EI Appeals Pilot Data Base

Table 2 Impact of In-person Commission Representation on Hearing Process
Factor	Scale	Appellants		Appellants' Reps.		BOR Members
		Treat.	Cont.	Treat.	Cont.	Treat.	Cont.
Atmosphere of hearings	-5 to +5	1.8	2.4	2.4*	3.3*	2.7*	3.2*
Understanding of Commission's position	0 to 10	5.8	5.8	8.5	7.3	7.7*	7.2*
Effect of representation on Understanding of Commission's position	-5 to +5	1.1	N/A.	1.6	N/A.	2.6	N/A.
Effect on Understanding of legislation	-5 to +5	0.9	N/A.	0.7	N/A.	2.1	N/A.
Effect on Understanding of Facts of Case	0 to 10	0.9	N/A.	0.7	N/A.	2.2	N/A.
Help to Focus on Issues	-5 to +5	1.0	N/A.	0.8	N/A.	2.4	N/A.

Comparisons between the control and treatment groups revealed that appellants experienced no significant difference in the degree of tension at the hearings or in their level of understanding of the Commission's position as a result of the presence of in-person Commission representation at hearings . In the opinions of appellants' representatives and of BOR members, the presence of an in-person Commission representative resulted in slightly greater levels of tension at board hearings. BOR members also indicated a slightly improved understanding of the Commission's position in hearings with in-person Commission representation.

Among the treatment cases, appellants and their representatives indicated, at best, marginal effects in terms of the level of understanding of the Commission's position and focusing the hearing on the issues of the case. As well, these groups perceived virtually no effects on the understanding of relevant legislation and the facts of the case. In at least two participating sites, appellants' representatives were very knowledgeable about the legislation as they are involved in a very large percentage of the appeals undertaken.

Members of the Boards of Referees saw the effects of Commission in-person representation to be slightly more positive on all factors than did appellants and their representatives.

The evaluation collected data, shown in Table 3, to shed light on the likely impact on resource requirements of Commission in-person representation. The impact was assessed in terms of the duration of hearings as well as the amount of time required for preparation and verification of information.

* Statistically significant at the 95% confidence level.

Table 3 Duration of Hearings, Average Preparation and Verification Time
Duration of Hearings	59 min.*	52 min.*
Preparation and Verification Time	245 min.	235 min.

In cases featuring in-person Commission representation, the average duration of hearings and the average preparation and verification times were only slightly longer than in cases without Commission in-person representation. Analysis showed that the slight increase in hearing duration was statistically significant while the apparently longer preparation and verification time was not. This suggests in-person representation may require more resources although the impact is not likely to be large.

Further evidence was collected from Commission representatives regarding the length of time required of them to prepare for in-person representation and to actually attend the hearings. The average time estimated for Commission representatives' involvement in hearings was close to 2.5 hours. Extrapolating from this, it could be estimated that handling approximately 400 such cases might require approximately .5 of a person/year.

The evaluation sought to ascertain the impact of in-person Commission representation on the actual outcomes or results of hearings. The measures indicative of these outcomes were whether there was a significant difference between control and treatment groups in terms of the number of appeals allowed in whole or in part, as illustrated in Table 4.

Table 4 Proportion of Claims Allowed Relative to Claims Dismissed
Claims Allowed
Totally	44%	42%
Partially	6%	15%
Claims Dismissed	50%	43%

Overall, 50% of the claimants in the treatment group had their appeals dismissed. This compares with 43% in the control group. In the treatment group, 44% of appeals were allowed totally, while 6% of claims were allowed in part. In the non-treatment group, 42% of appeals were allowed totally and 15% were partially allowed. The apparent differences between the control and treatment in the outcomes of hearings were not found to be statistically significant. In other words, outcomes of the hearings were not shown to have been materially affected by the presence of an in-person Commission representative.

Similarly, overall statistics revealed no significant difference in the incidence of adjournments between control and treatment groups. Adjournments with representation occurred in 110 cases and without representation in 111 cases. There was, however, variation among sites with one site in particular experiencing a relatively large proportion of adjournments among control cases relative to treatment cases.

Reasons for the number of adjournments were varied. The Toronto Commission representatives observed that one of the benefits of agent representation was the opportunity to ask for an adjournment. If the claimant had new information relating to the appeal, Commission representatives asked for an adjournment. Evidence from another site indicated that appellant representatives asked for adjournments if their case was scheduled for a particular day of the week when there would be Commission representation. The evidence suggests that appellant representatives preferred appearing before the BOR when the Commission was without representation. Also, appellant representatives are more aware than the average claimant of the advantages of requesting an adjournment. Adjournments provide more preparation time of the case material without negative consequences.

Finally, Commission representatives were canvassed regarding their views regarding the perceived impact of representation and satisfaction with hearing results. Table 5 presents the findings.

Table 5 Opinions of Commission Representatives Regarding Effects of In-Person Commission Representation
Atmosphere of Hearings (Scale -5 to +5)	1.8
Extent representative felt "heard" (Scale 0 to 10)	7.6
Est. impact of representation (Scale ­5 to +5)	2.0
Satisfaction with hearing results (Scale -5 to +5)	1.8

Commission representatives in treatment cases noted a fairly relaxed atmosphere at the hearings with no significant level of tension. Clearly, the representatives felt that they had been effective in communicating their positions before the Boards. They felt that their impact was positive but moderate and they were not overwhelmingly satisfied with the outcomes of the hearings.

Not unexpectedly, opinions and positions on most issues varied widely among the different groups of participants in the focus groups, debriefing sessions and interviews.

Expectations at the outset ranged from ambivalence to optimism. Members of Boards of Referees could not see a compelling motive for instituting the pilot project. Agents and supervisors on the Commission side expected that representation would impact positively on the appeal process and on outcomes.

Claimants indicated that they obtained third-party representation for three main reasons:

• Concern or lack of understanding about their case;
• Complexity of their case; or
• Concern about their ability to communicate in English or French.

Appellant views of the subsequent worth of third-party representation depended upon whether they won or lost their appeal. In at least one site, appellants held the view, corroborated by appellants' representatives, that the Board of Referees had permitted only minimal input from their representatives, preferring to hear from the appellant directly.

The agents who represented the Commission in the appeal process generally believed that Commission representation had influenced the outcomes of at least some cases and the quality of Board decisions. Most other study participants, however, felt that the impact had been relatively minor. A significant degree of consensus was found for the view that the largest impact may reside in the edification and enlightenment of agents participating in the process. The extent to which learning was imparted to other HRCC staff varied greatly as no formal mechanisms were in place to disseminate information. However, representing agents gained familiarity with the decisions process and learned how to better present relevant facts in written submissions. Moreover, in at least one location, agents who appeared before the Board of Referees seemed to have gained credibility which carried over to appeals in which the Commission was not represented in person.

The impact upon the actual outcomes of appeals was difficult to gauge from the qualitative findings. All non-HRDC participants felt that decisions were in no way affected by Commission representation. Conversely, most HRDC staff believed that at least some outcomes were affected and, more frequently, the quality of decisions was improved. These perceived benefits were attributed to the opportunity afforded Commission representatives to challenge statements made by claimants and to keep the Board of Referees focused on salient points of the case and relevant aspects of legislation. The perceived impact on outcomes was not borne out by the quantitative data that failed to detect a significant impact upon the incidence of cases dismissed, adjourned, or claims allowed, wholly or in part.

Commission representatives were able to provide an element of clarification regarding the facts of a case or points of law. Moreover, the consensus view of agents and BOR members suggested that the level of openness and honesty was improved by the presence of Commission representatives.

Generally, in-person Commission representation made the appeal process more formal and often made the atmosphere more stressful and intimidating for appellants.

With regard to impacts upon the duration of the appeal process, frequency of adjournments and the flow and pace of hearings, comments were limited and varied. Impacts on these aspects seemed to depend upon the personalities involved and the nature of the case. In addition, variations in implementation among the various sites may have exerted an impact upon observed results, particularly the numbers of adjournments.

Commission representation did not appear to help claimants gain a better understanding of their case or the Employment Insurance Act.

There appeared to be some question in the minds of appellants and their representatives regarding the objectivity of the Boards of Referees. In some cases, the BOR is housed in the same location as the HRCC, they are paid by the government, and trained by HRDC staff. There was confusion among claimants about the relationship between the Board and HRDC. Claimants did not know who BOR members represent. They did not realise that one BOR member represents workers. In some instances, BOR members were perceived as being ill prepared and the results were seen as being overwhelmingly influenced by the president of the Board.

The evaluation sought to address the issue of potential impacts upon workloads, morale, and the HRCC organisation. These factors were found to depend upon (a) the division of labour within the HRCC and the proximity of agents to the location of the BOR hearings, and (b) the way in which the pilot study was implemented in each site (e.g. whether representation was voluntary or not).

In most locations, the appeal process is handled by Agent IIs, a senior supervisory classification. In locations where all Agent IIs write appeals (and thus all participated in the pilot study as representatives), workloads were reported to be significantly increased with predictable adverse effects on morale and stress within the organisation. Where appeal writing and/or Commission representation were concentrated among a few or a single Agent II, (i.e. dedicated resources), positive impacts were noted on morale, workload and organisational stress.

In locations where agents were both preparing written submissions and representing the Commission at hearings, agents felt that they gained a better understanding of the Board of Referees and that the experience would enhance their ability to write effective submissions. In other locations, there was no formal process to share lessons and observations between Commission representatives and agents preparing written cases.

Participants generally agreed that Commission representation should be based on the complexity and nature of an appeal case as opposed to whether or not the appellant is represented by a third party. The Commission should be represented in cases such as misconduct, collusion between employer and employee, fraud or the suspicion of fraud, and cases involving large sums of money. Cases involving straightforward interpretations of legislation can be handled without in-person representation.

The evaluation addressed the issue of whether or not Commission representation should be adopted as national policy. Responses were sharply divided among the various groups participating in the study. Even those who felt representation should be adopted as policy expressed major qualifiers to their opinion.

Representing agents supported the concept but wanted to see workload issues addressed and the role of Commission representative filled on a voluntary basis. Managers, RHQ staff and supervisors saw some positive impacts but, in some sites, were concerned about impacts upon workload and staff morale.

BOR members were ambivalent but some felt Commission representation had been of some help. In two locations, BOR members would prefer that agents be available when needed. In another site, BOR members thought the policy should be adopted provided that HRDC agents had a clearly defined role and had received appropriate training. BOR clerks were concerned about potential increases to their own workload.

Appellants and their representatives, with some exceptions, were, for the most part, opposed to Commission representation. Those opposed felt it conveyed an unfair advantage to the Commission and would intimidate claimants. For many in these groups, the presence of a Commission representative made the process more tense and confrontational. Claimant representatives were mainly opposed because of the intimidation factor and, during the pilot, they did not see a constructive input from Commission representatives. Moreover, they expressed the view that in-person Commission representation, together with written submissions, gave the Commission two opportunities to make its case, whereas appellants were afforded only one such chance. In-person was considered by some to be redundant.

The overall findings suggest that most claimants would probably be intimidated by a Commission representative at their hearing or, at least, would find the process more stressful. However, the prospect would affect neither the appellant's decision to obtain representation nor the type of representation they would choose.

Participants were asked what changes should be made to current arrangements if Commission representation were to become national policy. Opinion was divided over whether the agent who writes the appeal should also represent the Commission at the hearing. However, other findings suggested that, in any event, representation should be voluntary and involve individuals with the skills and aptitude for the job.

There was agreement that the respective roles of the Commission representatives and BOR members need to be more clearly defined. Agents who are expected to represent the Commission at hearings should receive relevant training. There was strong support for the creation of dedicated resources for the purpose of Commission representation.

No consensus emerged on the question of the appropriate role for the Commission representatives. BOR members and appellants opposed to the idea of Commission representation as policy saw Commission representatives as resource persons to be called upon as needed to provide clarification. HRDC staff, especially representing agents, wanted to play a more active, assertive role in the appeal process.