Evaluation of D2006 - April 2006

This report provides a summary of the results of an evaluation of Transport Canada’s (TC)Direction 2006 Program (D2006). The Departmental Evaluation Services (DES) Branch undertook the evaluation at the request of Funded Partnerships Programs Branch of TC’s Rail Safety Directorate. This evaluation is expected to help inform a decision about Program renewal since the Program is slated to end in 2006.

1.2 Program Profile

The Railway Safety Act (RSA) of 1989 mandated the Federal Government to promote and provide for the safety of the public and personnel, and the protection of property and the environment, in the operation of railways, and encourage the collaboration and participation of interested parties in improving railway safety.

In 1994, the Railway Safety Act Review Committee assessed rail safety in Canada. The Committee concluded that more could be done to reduce fatalities and injuries along Canada’s railways. The Committee subsequently made a number of proposals to the Minister of Transport, including the development of a comprehensive plan to reduce the number of collisions at grade crossings by 50% within 10 years. In 1995, following consultations with the affected parties, the Minister of Transport instructed the Department to create a national program that would achieve a 50% reduction in highway/railway grade crossing collisions as well as trespassing incidents[1] within a 10-year timeframe. Subsequent to further national consultations, D2006 was created with this objective. D2006 is operated on an annual budget of $700,000 not including the cost of one program manager working full-time.

The D2006 Executive Committee is responsible for providing strategic direction for the Program. It is co-chaired by one representative from both TC and the Railway Association of Canada (RAC). The D2006 Executive Committee includes representatives from government, industry, and the voluntary sector. The Executive Committee is organized under the following Key Result Areas (KRAs): communications, outreach, legislation, research, education, and enforcement. Operation Lifesaver (OL) is both the main contributor to D2006’s education KRA and the primary distribution network for materials produced by D2006.

1.3 Program Logic Model

The logic model in Table 1 demonstrates the causal relationships between the Program’s activities and the intended immediate, intermediate, and ultimate outcomes. The activity areas of D2006 impact on two levels of reach: at an immediate level of outcomes it targets the Program’s partner organizations, and at an intermediate level it targets the general public.

D2006 conducts activities in the areas of fostering and maintaining partnerships, enforcement, research, communication, outreach, education and legislation. The intended results of these activities, shown as immediate outcomes in the logic model, are to increase the sharing of information among partners, to enhance the adoption of best practices, to increase D2006 partners’ ability to implement projects, to improve grade crossings based on the research that is conducted, and to enhance the ability of legislators and police officials to effectively address rail safety issues.

The activity areas and the immediate outcomes all contribute toward the achievement of the Program’s intermediate outcomes. These include increasing awareness among the public of the dangers associated with grade crossings and trespassing on railway property, enhancing adoption of safer practices, and reducing grade crossing collisions and trespassing incidents. The target for the Program is a 50% reduction in grade crossing collisions and trespassing incidents by 2006.

These intermediate outcomes contribute towards D2006’s ultimate outcomes of increasing the safety of the railway system and enhancing public confidence in the safety of railways. These, in turn, contribute to Transport Canada’s strategic outcome of a safe and secure transportation system.

2.0 RELEVANCE AND DEMAND

D2006 is consistent with Government of Canada and Transport Canada objectives.

The objective of D2006 “is to reduce grade crossing collisions and trespassing incidents by 50% by the year 2006.” This objective is consistent with the Government of Canada’s commitment to maintaining public safety as well as TC’s specific commitment to rail safety, and in particular increasing safety at grade crossings and reducing trespassing incidents.

The Budget Speech of 2005 states “there is no more fundamental — or important — role for government than protecting its citizens from harm.” Similarly, the Speech from the Throne of 2004 states that “better health for Canadians requires more than timely access to health care. It requires … addressing risk factors …[and] prevention of injury”. In addressing risk factors associated with railways with respect to grade crossings and trespassing, D2006 is consistent with these statements.

One of the strategic outcomes of TC is to support a safe and secure transportation system that contributes to Canada's social development and security objectives. Not only does D2006 conform to this broad view of a safe and secure transportation system but TC also released two policy statements in October of 2000[3] outlining the Department’s continued commitment to rail safety and Canadians. These statements address trespassing and grade crossing safety and specify that the Department will maximize its impact on transportation safety. The focus of these policies is to ensure a proactive approach to railway/road crossing safety and prevention of trespassing through awareness of regulatory requirements, education through guidance, counselling and advice, and compliance monitoring of regulated parties.

D2006 activities are needed but there may not be an ongoing need for the Program.

The activities of D2006 were found to be an important component of rail safety in general but evaluators found there is an overlap between these activities and those of other programs with similar mandates. Hence, evaluators are unsure whether the Program is still necessary.

Two recent studies that examined the United States (U.S.) railway system support the need for public awareness and education activities such as those delivered by D2006. One cites a “continuous need to inform drivers and public of real dangers associated with poor perceptions of dangers of grade crossing and perceptions of where and how fast trains travel.”[4] It examines the underlying problems that lead to these incidents and how they must continually be addressed in order for mitigation strategies to be effective. The second study concludes that “reducing or eliminating activities regarding engineering, education and enforcement would increase collision rates at graded crossings by as much as 17%.”[5] As the research was conducted in the U.S., evaluators cannot apply the 17% finding directly to Canada but the research is evidence of the effectiveness of this type of activity in decreasing trespassing incidents and grade crossing collisions in general.

In interviews conducted with D2006 stakeholders, respondents indicated there was a growing need to ensure the message is delivered to specific target groups such as truck drivers, bus drivers, and children. When asked about future demand, respondents indicated there is an increased demand for outreach/communications oriented activities. Respondents also indicated that engineering may be more in demand in the near future. Stakeholders also answered that the availability of outreach/communications material is greatly enhanced as D2006 is very active in producing up to date materials in both official languages which is a capacity not always enjoyed by not-for-profit safety organizations. Not surprisingly, a majority of respondents also believed that the activities of D2006 have contributed to reducing grade crossing collisions and trespassing incidents and that if such activities were to cease, accidents would increase.

Evaluators have difficulty justifying the need for a separate Program to cover off these activities. Although D2006 was meant to be a comprehensive approach to rail safety, in reality it is primarily a public awareness program. OL’s education activities are identical to those of D2006. In addition, they are recognized partners with both their names appearing on outreach and communication products. Although D2006 was intended to undertake legislation activities, this has not happened for a number of years. Funds and efforts were therefore realigned from the legislation activity to other D2006 activities that would benefit from extra resources. Evaluators note that in future decision making on program renewal, the research activity should be considered. Although research activities related to rail safety are actually undertaken by the TDC[6] and funded by TC, D2006 plays the important role of bringing stakeholders together to identify areas of high-risk where research should be undertaken. Nevertheless, evaluators can find little justification for the existence of several programs delivering similar outcomes.

Partners are already playing an active role in D2006 but provinces could play a larger role.

Partners involved in D2006 play an active role in the operational activities of D2006. The Program was meant to bring interested stakeholders together to ensure resources are directed to the greatest areas of need as identified by the stakeholders. This Program element has been fulfilled as evidenced by stakeholder participation throughout the life of the Program. Stakeholders from industry and from volunteer organizations expressed a strong commitment to Program activities. However, respondents from volunteer organizations also note that these commitments compete with responsibilities owed to their parent organizations. As such, they feel they are doing all they can and cannot commit to more duties with D2006.

There is interview evidence that not all the provinces are involved. When stakeholders were asked about the level of involvement of various stakeholders, the provinces received the lowest ratings. There is also interview evidence that not all the provinces are participating to the same extent in D2006 activities. Participants at Executive Committee meetings determine annual projects based on levels of interest in D2006 activities.

Records of Executive Committee meetings indicate that attendance by provincial representatives has been low but there is anecdotal evidence that the provincial governments may not be able to send their representatives outside their respective provinces due to travel policies. Provinces with no rail lines such as Prince Edward Island and Newfoundland and Labrador would not be expected to participate. On the other hand evaluators also note that shortline railways have been increasing and these rail lines fall under provincial jurisdiction. Consequently, more provincial involvement in safety programs will likely be needed. Taken together, these lines of evidence point towards both an existing and growing need for greater involvement from the provinces in the type of activities undertaken by D2006.

[Page Up]

3.0 PROGRAM SUCCESS AND IMPACTS

D2006 has been successful in cultivating partnerships for the purposes of sharing information and increasing stakeholder ability to implement projects.

Partnerships amongst various stakeholders have been successful and there is a good representation of stakeholders from the public and the private sector. The program has been successful in positively impacting on its first level of reach, that is the program’s partner organizations.

Partnerships are the mainstay of D2006. This is evident both at the level of functional direction and in delivery mechanisms. The D2006 Executive Committee includes representatives from government, industry, and the voluntary sector. The delivery of initiatives is also greatly dependent on partnerships. Partnerships with various media outlets have been instrumental in getting the D2006 rail safety message out to the general public through radio and television public service announcements, billboards, and posters. The Professional Drivers Information Package was produced by the D2006/OL Truck-Rail Working Group, and truck driver associations, groups and trainers were instrumental in the delivery of the package to its intended audience.

According to the interviews, stakeholders clearly believe that D2006 has been successful in fostering and maintaining partnerships. Among all scaled questions, the highest scores were observed in the area of partnerships. All respondents indicated a high level of agreement that partnerships resulted in sharing of information. Respondents felt even more strongly that the activity of fostering and maintaining partnerships increased the ability to implement projects.

While D2006 demonstrates high-level success with respect to partnerships, a number of respondents indicated that there was room for improvement. Suggestions included pursuing increased trade union presence at the Executive Committee level, and increased numbers of partnerships in Western Canada. Additionally, some respondents felt that they did not know about all activities or information that they could take advantage of despite the communications taking place at meetings.

It is unclear whether D2006 activities have raised public awareness and changed their behaviour around railways.

Due to a lack of performance data and attribution studies, evaluators could not determine the extent to which D2006 activities impacted on the second level of reach, the general public. Evaluators were unable to determine whether D2006 increased the targeted audiences’ awareness of dangers associated with grade crossings and trespassing on railway property, enhanced the adoption of safer practices, and contributed to the reduction in collisions and trespassing incidents.

There is little doubt that stakeholders interviewed strongly believe the audiences exposed to D2006 initiatives have increased awareness of grade crossing and trespassing hazards. Similarly, respondents express a strong opinion that D2006 activities are having a positive impact with respect to the adoption of safer practices. A typical comment is: “[D2006] has helped alter driver behavior, especially professional drivers”. However, opinion of this sort needs to be corroborated by other lines of evidence. Evaluators could not find such objective substantiation to show that D2006 activities have resulted in increased awareness and increased adoption of safer practices, and that this has then led to a reduction in the number of grade crossing collisions and trespassing incidents.

Evaluators were provided with a report outlining the impact of the 2004 public service announcement (PSA) radio campaign, entitled Report on Value and Impact of Public Service Radio Announcement on Railway Safety. However, the report did not contain any actual evidence as to the impact of the campaign. Evaluators could not find evidence that the campaign, which was designed to yield over 22,000 airings, resulted in increasing the rail safety awareness of their intended audience. The report merely provided a summary of when the PSAs would likely air and the audience reach of radio stations that had committed to participate in the campaign. Similarly, there was no data on the impact of billboards or transit shelter posters.

Information on the Program’s activities and their outputs is not a substitute for performance data on impacts. The argument that the sheer wealth of activities and materials produced and distributed must have had the desired impact, and that this is demonstrated by the decline in the number of incidents, is not sufficient. Even with extensive output levels, desired results may not be achieved. The campaign geared towards truck drivers to reduce the risk of heavy truck/train collisions at crossings is a case in point. Largely in response to a TSB recommendation[7] that followed the investigation of a collision in 2002 between a CN train and a tractor-trailer, a number of videos, guides and safety quizzes were produced and distributed to truck drivers. In its 2004 assessment of the response to its recommendation, TSB found that “while D2006 [was] a sound initiative, accidents involving heavy vehicles [did] not appear to be diminishing significantly”. The TSB assessment noted that the overall D2006 statistics on such occurrences did not match TSB records and concluded that “while [the] response [was] positive, the reduction of heavy vehicle accidents [had] not advanced sufficiently to reduce the risks of transportation safety”. The trucking example illustrates that outputs may not automatically translate into results even if the proper target audience is being reached. For this reason, it is impossible for evaluators to know whether the extensive communication output of thousands of PSAs has achieved the desired results.

The issues surrounding the availability of performance data aside, given the intertwined relationship between D2006 and OL, it would still be difficult to ascertain what percentage of the reduction in collisions at grade crossings and in trespassing incidents is attributable to D2006. Both D2006 and OL are primarily concerned with promoting public awareness of the appropriate conduct at highway-railway crossings and on railway property. Both programs concentrate on education and public awareness to achieve their objectives, but both also extend into enforcement and engineering domains. In addition, there is a high level of cooperation and complementarity between the two programs. D2006 produces materials that are distributed by OL.

Without a basic measure of activity clearly attributable only to D2006, establishing a statistical correlation between D2006 activities and results is not possible. Studies conducted in the U.S. have successfully established a statistical correlation between the types of public education activities similar to those performed by D2006 or OL and the reduction in grade crossing collisions. For example, one U.S. study using regression analysis found that 15% of the decrease in the number of collisions and 19% of the decrease in the number of fatalities was attributable to the U.S. OL[8] program. The measure of activity used in the U.S. study consisted of the number of OL presentations and special training events, expressed as “a rate per 1000 crossings[9]”. However, in Canada, comparable D2006 events are usually sponsored by both D2006 and OL, just as we find OL and D2006 brands side-by-side on almost all of the materials produced by D2006. For this reason, if a similar study were undertaken in Canada, the results would reflect the combined activities of OL and D2006.

While there have been significant reductions in collisions and incidents, the Program targets have not been reached.

The targets of reducing grade crossing collisions and trespassing incidents from 1996 levels by 50% by 2006 have not yet been met. There were 365 grade crossing collisions in 1996 and 270 in 2005, a reduction of 26%. There were 126 trespassing incidents in 1996 and 83 in 2005, a reduction of 34%. See annex 5 for the TSB data.

Evaluators opted to use a 5-year average to examine the trends in collisions and incidents. See Annex 5 for a full methodology description and tables of the resulting data. Using a 5-year average allowed evaluators to examine the collision and incident trends while trying to minimize the effects of years where there were spikes in collisions and incidents. For example, there were 126 trespassing incidents in 1996 but 65 incidents in 2003 followed by 99 incidents in 2004 and 83 in 2005. Over the course of 10 years such fluctuations make it difficult to determine whether there has been an increasing or decreasing trend in the number of incidents. Using a 5-year average it is possible to graph the resulting data and determine if there has been a more linear trend with respect to grade crossing collisions and trespassing incidents. Additionally, evaluators decided to apply the same method of developing 5-year averages to grade crossing and trespassing fatalities and injuries to determine if there were any trends.

Graph 1, on the following page, shows there has been a decrease in the average number of both grade crossing collisions and trespassing incidents. This represents a decrease of 32% in the average number of grade crossing collisions for the 5-year average leading to 2005 when compared with the 5-year average leading to 1996. For trespassing incidents this represents a 25% decrease for the same time periods.

^{Source: Transport Canada, derived from TSB statistics, see Annex 5}

In order to assess the severity of collisions and incidents, the evaluators looked at the associated 5-year average for injuries and fatalities. Graphs 2 and 3 present fatality and injury data for grade crossing collisions and trespassing incidents. Graph 2 shows that grade crossing injuries were reduced by 30% and fatalities were reduced by 37%. Graph 3 shows that trespassing injuries were reduced by 36% and fatalities were reduced by 6%. In-depth data was not available regarding the causes of injuries and fatalities therefore evaluators could not conduct more detailed analysis concerning problem identification. Note that the definition of an injury was redefined by the TSB in 1992 and the necessary data for a 1992-1996 average for injuries was unavailable.

^{Source: Transport Canada, derived from TSB statistics, see Annex 5}

The targets set for the Program may have been unrealistic.

While the target of a 50% reduction in grade crossing collisions and trespassing incidents has not been reached, evaluators note that this target may have been overly ambitious.

The impact of suicides on trespass prevention is an important factor. While the TSB does not capture data on suicides, it appears that suicides represent a significant portion of trespassing incidents. Most stakeholders interviewed believe that 40 to 50% of trespassing deaths are attributable to suicides. They indicate that preventing suicides is difficult to achieve, with comments like: “a good portion of the accidents are suicides, which are difficult to deal with”, “suicide … is a tough subject matter to manage and influence”. Successful suicide prevention strategies that D2006 can emulate do exist but expectations need to be put in perspective. For example, in their presentation[10] to the D2006 Executive Committee, representatives from the Centre for Suicide Prevention cited a regional program in U.K. that focused on identifying “hot spots” and preventing transitory, passing impulse types of suicide attempts. This program was considered a success, although suicides in that region decreased only by 3 percent as a result. Even if D2006 and its partners had deployed successful suicide prevention measures and achieved comparable results, the contribution to the overall reduction in the number of trespassing incidents would likely have been low. For this reason, new targets should take into account factors such as suicides in case of Program renewal.

Evaluators do not advocate that TC or other stakeholders disregard the problem of suicide in rail safety, but caution that suicides are a larger social problem requiring a different set of approaches.

[Page Up]

4.0 PROGRAM EFFICIENCY AND COST-EFFECTIVENESS

The D2006 partnership activities have some efficient characteristics but overlap and duplication between OL and D2006 have also led to some inefficiencies.

The partnership nature of D2006 demonstrates some efficient characteristics. Program stakeholders undertake Program operations in an efficient manner where all stakeholders can access and disseminate research results, communications materials, etc. Decision-making is rendered more efficient as all stakeholders are at the table and can discuss issues and have direct input to decisions.

Despite the efficiency of partnerships, there is a level of duplication and overlap between D2006 and OL that results in some inefficiencies. A comparison of the D2006 Executive Committee Membership and the OL Advisory Committee Membership shows that 6 of the 14 D2006 Committee members are also on the OL Advisory Committee and attend both sets of meetings at separate times and locations. Some respondents indicated that members would sometimes get confused as to which meeting items were for which program, such was the similarity of agenda items for each meeting. Attending two sets of meetings is not beneficial to all stakeholders and having only one set of meetings would benefit members by reducing costs, travel times, and administration. There were also comments from respondents that indicated they don’t understand the distinction between D2006 and OL and found it difficult to explain the differences between the two programs. Respondents also indicated they participate on a voluntary basis and this creates pressure for some of them as they must also commit to their own workplace activities as well as attend to the activities of both D2006 and OL.

The cost-effectiveness of the Program could not be established.

Evaluators cannot establish if the Program has been cost-effective. The link between the activities of the Program and its results cannot be established, and the Program did not provide annual expenditure data, despite repeated requests.

Even if cost and attributable results information were available, evaluators would be unable to factor in the effect of in-kind contributions as detailed accounts of these contributions are also unavailable. In-kind contributions from stakeholders and partners exist and the Program claims a ratio of 5:1, which would mean that each $1 contributed by TC is matched by $5 in services or funds from stakeholders, but this ratio cannot be verified.

[Page Up]

5.0 RECOMMENDATIONS

The evaluation was conducted to evaluate the success, relevance, efficiency and cost-effectiveness of D2006 to aid in decision-making regarding Program renewal since it is slated to end in 2006. Evaluators believe there may be some value in continuing the activities of D2006 but only if Rail Safety incorporates the following recommendations into new programming:

Create a single program to deliver the activities currently delivered by D2006 and OL that builds on the gains made in fostering and maintaining partnerships with industry, the volunteer sector, and the provinces.
Ensure better substantiation of the Program’s impact claims by implementing a performance measurement strategy including accurate data on TC costs and others’ in-kind contributions.
Review grade crossing collision and trespassing incident reduction targets to ensure they are realistic by taking into consideration factors such as suicides.
Pursue more provincial involvement.

[Page Up]

Annex 1: Program Profile

Objective

The primary objective of D2006 “is to reduce grade crossing collisions and trespassing incidents by 50% by the year 2006”.

Roles and Responsibilities

The design of the Program emphasizes a multitude of public and private sector partnerships and shared decision-making. D2006 is primarily governed through a professional service contract issued by TC to the Railway Association of Canada (RAC). TC budgets a given amount annually for D2006. Nearly ninety percent of this sum is allocated to fulfill the contract with RAC, which, in turn, matches it through in-kind contributions. The contract commits RAC to deliver a number of railway safety-related projects annually.

The roles and responsibilities are determined through a committee structure that provides a framework for dividing work and ensuring that critical tasks are accomplished. The Executive Committee is responsible for the management of the funds as well as providing strategic direction for the Program. It is co-chaired by one representative from both TC and RAC.

D2006 focuses on key result areas, or KRAs, to reach its objectives. Each KRA is composed of activities. These KRAs are:

Education.
Enforcement.
Research.
Legislation.
Communication.
Outreach.

Each KRA is the responsibility of a sub-committee. The lead in each KRA is usually a function of appropriate area of expertise. In recent years, ad hoc committees have been introduced to focus on specific tasks such as data collection or to tackle challenges like suicide prevention.

Resources

TC budgets $700,000 annually for D2006. Of this amount, $625,000 is set aside to fulfill the professional service contract with RAC. The Program administrator invests the remaining $75,000 in various safety awareness initiatives that arise on an ad hoc basis each year.

At the time this report was written, the evaluation team had not received data on the actual figures spent in each year of the Program.

Reach

The Program primarily targets cross-sections of the general public, including school-aged children/youth, pedestrians, motor vehicle operators (truck drivers, snowmobile operators, school bus drivers, emergency responders), road authorities and law enforcement agencies. In addition, through its Outreach KRA, the Program often targets communities and municipalities, as well as specific groups such as the hearing-impaired.

[Page Up]

Annex 2: Evaluation Plan, Methodology and Data Sources

Evaluation Plan

The evaluation team developed an evaluation strategy and selected methods for collecting the data needed to address the evaluation questions.

The evaluation study addressed the following questions to determine the Program’s relevance, success and impacts and cost-effectiveness:

Evaluation Questions - Program Relevance	Indicators	Data Sources
1. Does the Program align with government priorities and advance the strategic objectives of TC?	Level of consistency between Program objectives, governmental priorities, and Departmental strategic objectives	Speech from the Throne TC Business Plan TC Report on Plans and Priorities Straight Ahead
2. Is there an ongoing need for the Program?	Opinion / assessment from stakeholders Accident / incident rate trends The extent to which the Program’s targets are met	Survey data
3. Could other partners, including other levels of government, private and voluntary sectors, play a larger role?	Opinion / assessment from stakeholders	Survey data

Evaluation Questions - Program Success and Impacts	Indicators	Data Sources
3. Is the Program achieving its crossing and trespassing targets?	# and % of grade crossing collisions and trespassing incidents vs. target figure # and % of fatalities that result from grade crossing collisions and trespassing incidents vs. target figure # and % of serious injuries that result from grade crossing collisions and trespassing incidents vs. target figure	Statistical data from Transportation Safety Board (TSB) File review
4. To what extent has the Program resulted in a reduction of grade crossing collisions and trespassing incidents?	Expert opinion / Surveys	Statistical data from Transportation Safety Board (TSB) File review
5. Were there any unintended consequences?	# of unintended consequences Expert opinion	Surveys File review
6. Is the activity of fostering and maintaining partnerships having the desired impacts of sharing of information and the increased ability to implement projects?	# of partnerships Level of leverage achieved through partnerships	Surveys
7. Are the legislation, research, enforcement, communication, outreach and education activities having the desired impacts of increased awareness and enhanced adoption of safer practices?	Level(s) of awareness of all target audiences Expert opinion	Surveys

Evaluation Questions –

Efficiency and Cost-Effectiveness

Indicators

Data Sources

8. Is the current funding approach the most cost-effective method of achieving the Program’s objectives?

Level of leverage achieved
Expert opinion

File review
Survey of subject-matter experts, including TC staff

9. Is there any overlap and/or duplication in the Program design and delivery?

Opinion

Interviews / Surveys

Methodology

1. Document review

The evaluators conducted a review of:

Program documents.
Documents outlining Government of Canada and Transport Canada objectives and priorities.
Research papers and media reports regarding grade crossing safety.

2. Interviews

A master question list (refer to Annex 4), prepared by DES in the form of a matrix of 37 questions, was the basis of interviews held with stakeholders (refer to Annex 3). Some of the interview questions were asked of each type of respondent, while a number of questions were asked of only some respondents as determined by their level of involvement in the Program. Respondents were drawn from the Executive Committee as well as the KRA sub-Committees.

3. Data Analysis

Relevant TSB data was reviewed and analyzed (refer to Annex 5).

Data Sources

D2006 Program Files / Documents/Committee Meeting Files.
Straight Ahead - A Vision for Transportation in Canada.
TDC files.
Centre for Suicide Prevention.
Railway Safety Act.
Stakeholder Interviews.
Railway Association of Canada Annual Reports: 2000, 2001, 2002, 2003, 2004.
Transportation Safety Board of Canada Annual Statistics.
Mok, Shannon. And Savage, Ian Why has Safety Improved at Rail-Highway Grade Crossing?
Northwestern University, 2004.
Savage, Ian, Does Public Education Improve Rail-Highway Crossing Safety? Northwestern University, 2005.
Savage, Ian, Railroad Safety and Public Policy Journal of the Transportation Research Volume, Vol 38(1): 56-63, 1999.

[Page Up]

Annex 3: List of Stakeholders Interviewed

Gary Drouin	Transport Canada D2006 Executive Committee co-chair
Mike Lowenger	Railway Association of Canada (RAC) D2006 Executive Committee co-chair
Mike Cameron	Railway Association of Canada (RAC)
Dan Di Tota	Operation Lifesaver (OL)
Bob Nash	CN Rail
Marc Tessier	VIA Rail
Serge Meloche	CN Police
Gerry Moody	CPR Police
David Ewart	Federation of Canadian Municipalities (FCM)
Jennifer Hall	Safe Kids Canada
Harly Toupin	Saskatchewan Safety Council
Marie Brion	Quebec Safety League
Rob Francis	Government of New Brunswick
Steve Harvey	GoTransit, Ontario

Annex 4: Master Question List for Interviews

Note: x denotes that the question was asked of that type of respondent.

Response Types:

O – Open-ended Questions
Y/N+ – Yes or No Questions along with a prompt for more details
S – Scale Questions

Question	Response Type	Railway companies	Railway Association of Canada	Safety Leagues and Organizations	Province		Program Management /Managers	Rail Police	Federation of Canadian Municipalities	Operation Lifesaver
Background
What is your organization’s role/involvement with D2006?	O	x	x	x	x		x	x	x	x
Why did your organization become involved with D2006?	O			x					x
Demand for D2006
How do you see the demand for D2006 initiatives changing in the next few years?	O	x	x	x			x	x	x	x
Have you seen a change in demand for D2006 initiatives over the past several years?	Y/N+	x	x				x	x		x
Program Renewal/Restructuring
What impact would there be if TC funding was reduced or eliminated?	O	x	x	x	x		x	x	x	x
Could another program assume the mandate of this Program?	O	x	x				x	x	x	x
How do D2006 and OL affect one another? Are they complementary or is there interference?	O	x	x				x	x	x	x
D2006 Problem Identification and D2006 Processes
To what extent are you satisfied with the current D2006 process of identifying problems? (Scaled 1-5)	S	x	x				x	x	x	x
To what extent are you satisfied with the current D2006 process of developing strategies to respond to problems (Scaled 1-5)?	S	x	x				x	x	x	x
To what extent are you satisfied with the current D2006 processes such as Delivery of the resulting initiatives (Scaled 1-5)?	S	x	x				x	x	x	x
Are there any areas where you would like to see processes changed?	O	x	x				x	x	x	x
To what extent do you believe D2006 has the capacity to address identified rail safety challenges?	O	x	x				x	x	x	x
Program Impact / Targets
What impact has D2006 had on Grade crossing safety, intended or otherwise?	O	x	x	x	x		x	x	x	x
What impact has D2006 had on the occurrence of trespassing incidents, intended or otherwise?	O	x	x	x	x		x	x	x	x
What other impacts has D2006 had?	O	x	x	x	x		x	x	x	x
What are some challenges D2006 has faced in achieving its intended goals?	O	x	x				x	x	x	x
Change in Awareness and Adoption of Safer Practices
To what extent do you feel that audiences exposed to D2006 initiatives have increased awareness of grade crossing and trespassing hazards (Scaled 1-5)?	S	x	x	x	x		x	x	x	x
Do you know of any specific mechanisms in place to assess the change in awareness by D2006 audiences?	O	x	x				x			x
To what extent do you feel that audiences exposed to D2006 initiatives adopt safer practices with respect to grade crossing and trespassing hazards (Scaled 1-5)?	S	x	x	x	x		x	x	x	x
Do you know of any specific mechanisms in place to assess the adoption of safer practices by D2006 audiences?	O	x	x				x			x
On a scale of 1-5, are legislation activities resulting in: a) Increased awareness b) Enhanced adoption of safer practices	S	x	x				x	x		x
On a scale of 1-5, are research activities resulting in: a) Increased awareness b) Enhanced adoption of safer practices	S	x	x				x	x		x
On a scale of 1-5, are communication activities resulting in: a) Increased awareness b) Enhanced adoption of safer practices	S	x	x				x	x		x
On a scale of 1-5, are outreach activities resulting in: a) Increased awareness b) Enhanced adoption of safer practices	S	x	x				x	x		x
On a scale of 1-5, are education activities resulting in: a) Increased awareness b) Enhanced adoption of safer practices	S	x	x				x	x		x
Partnerships
To what extent do you think the activity of fostering and maintaining partnerships is having the desired impacts of sharing of information (Scaled 1-5)?	S	x	x				x	x		x
To what extent do you think the activity of fostering and maintaining partnerships is having the desired impacts of increasing the ability to implement projects (Scaled 1-5)?	S	x	x				x	x		x
Are there areas for improvement with respect to the activity of fostering and maintaining relationships?	O	x	x	x	x		x	x	x	x
How would you describe the extent of participation by the private sector (Scaled 1-5)?	S	x	x				x	x		x
How would you describe the extent of participation by the voluntary sector (Scaled 1-5)?	S	x	x				x	x		x
How would you describe the extent of participation by provinces (Scaled 1-5)?	S	x	x				x	x		x
How would you describe the extent of participation by the municipalities (Scaled 1-5)?	S	x	x				x	x		x
To what extent are TC Regions involved in D2006 (Scaled 1-5)? If not involved, why not?	S	x					x
Cost-effectiveness
What is your opinion of the current funding approach used by TC?	O	x	x				x			x
Are you aware of alternative funding arrangements that you believe would be more suitable?	O	x	x				x			x
What actions have you taken to manage future spending pressures? What more can be done?	O	x	x				x
Do you have any comments you wish to add?	O	x	x	x		x	x	x	x	x

Annex 5: Transportation Safety Board Statistics

Data: Data was obtained from the TSB Annual Statistics. The data presented in Table 1 are annual numbers of grade crossing collisions and trespassing incidents. Tables 2 and 3 tabulate the number of injuries and fatalities associated with grade crossings and trespassing.

Definitions: The use of the terms collisions and incidents are unique to the D2006 Program and have different meanings at the TSB. The following definitions represent the meaning of the terms as used by D2006:

Collision: A grade crossing collision is defined as any situation where the rolling stock is involved in a grade crossing collision resulting in death, serious injury or damage to property.

Incident: A trespassing incident is defined as a situation where a person sustains a serious injury or is killed as a result of coming into contact with any part of the rolling stock or its contents while trespassing onto the railway lines.

Injury: A serious injury is an injury that is likely to require admission to a hospital.

Analysis: Evaluators decided to use a 5-year average to examine the trends in collisions and incidents. The intent is to ‘smooth’ out the effect of years where there was a spike in collisions or incidents in order to represent any general trends with respect to grade crossing collisions and trespassing incidents. For example, in Table 1, trespassing incidents in 2004 rose to 99 incidents from 65 in 2003. The use of a 5-year average shows that despite this spike there was still a general decline in trespassing incidents when averaged on a 5-year schedule. A 5-year average is calculated by summing the total for a given year and the 4 previous years together and dividing by 5.

Table 1: Trespassing Incidents & Grade crossing Collisions

Trespassing Incidents				Grade crossing Collisions
Year	Number	5-Year Average	Average	Year	Number	5-Year Average	Average
1992	97			1992	386
1993	103			1993	379
1994	99			1994	391
1995	112			1995	379
1996	126	1992-1996	107.4	1996	365	1992-1996	380.0
1997	98	1993-1997	107.6	1997	307	1993-1997	364.2
1998	78	1994-1998	102.6	1998	273	1994-1998	343.0
1999	95	1995-1999	101.8	1999	283	1995-1999	321.4
2000	79	1996-2000	95.2	2000	263	1996-2000	298.2
2001	80	1997-2001	86.0	2001	278	1997-2001	280.8
2002	73	1998-2002	81.0	2002	261	1998-2002	271.6
2003	65	1999-2003	78.4	2003	250	1999-2003	267.0
2004	99	2000-2004	79.2	2004	237	2000-2004	257.8
2005	83	2001-2005	80.0	2005	270	2001-2005	259.2

Source: Annual figures from The Transportation Safety Board of Canada (TSB)

Table 2: Grade Crossing Injuries and Fatalities

Injuries				Fatalities
Year	Number	5-Year Average	Average	Year	Number	5-Year Average	Average
1992	*			1992	73
1993	80			1993	56
1994	64			1994	54
1995	75			1995	53
1996	69	1992-1996	*	1996	46	1992-1996	56.4
1997	60	1993-1997	69.6	1997	32	1993-1997	48.2
1998	43	1994-1998	62.2	1998	39	1994-1998	44.8
1999	44	1995-1999	58.2	1999	37	1995-1999	41.4
2000	33	1996-2000	49.8	2000	33	1996-2000	37.4
2001	47	1997-2001	45.4	2001	41	1997-2001	36.4
2002	42	1998-2002	41.8	2002	46	1998-2002	39.2
2003	52	1999-2003	43.6	2003	27	1999-2003	36.8
2004	50	2000-2004	44.8	2004	25	2000-2004	34.4
2005	54	2001-2005	49.0	2005	38	2001-2005	35.4

*Due to a 1993 change in the definition of what constitutes an injury, data prior to that year is not available.

Source: Annual figures from The Transportation Safety Board of Canada (TSB)

Table 3: Trespassing Injuries and Fatalities

Injuries				Fatalities
Year	Number	5-Year Average	Average	Year	Number	5-Year Average	Average
1992	*			1992	57
1993	37			1993	58
1994	28			1994	56
1995	41			1995	64
1996	45	1992-1996	*	1996	67	1992-1996	60.4
1997	30	1993-1997	36.2	1997	69	1993-1997	62.8
1998	17	1994-1998	32.2	1998	61	1994-1998	63.4
1999	34	1995-1999	33.4	1999	62	1995-1999	64.6
2000	23	1996-2000	29.8	2000	53	1996-2000	62.4
2001	22	1997-2001	25.2	2001	56	1997-2001	60.2
2002	21	1998-2002	23.4	2002	50	1998-2002	56.4
2003	21	1999-2003	24.2	2003	46	1999-2003	53.4
2004	34	2000-2004	24.2	2004	67	2000-2004	54.4
2005	18	2001-2005	23.2	2005	64	2001-2005	56.6

*Due to a 1993 change in the definition of what constitutes an injury, data prior to that year is not available.

Source: Annual figures from The Transportation Safety Board of Canada (TSB)

[Page Up]

[1] Refer to Annex 5 for definitions of grade crossing collisions and trespassing incidents.

[2] TDC Report TP 13128E, Study of adding reflective materials to crossing signs and posts.

[3]Trespassing Prevention Policy October 2000, Railway/Road Grade Crossing Policy October 2000.

[4] Mok & Savage, Why Has Safety Improved at Rail-Highway Grade Crossings, p. 4.

[5] Savage, Does Public Education Improve Rail-Highway Crossing Safety?, p. 10.

[6] Highway-Railway Grade Crossing Research program.

[7] Transportation Safety Board (TSB) Recommendation R04-02.

[8] Mok & Savage, Why Has Safety Improved at Rail-Highway Grade Crossings, p. 14.

[9] Savage, Ian Does Public Education Improve Rail-Highway Crossing Safety? Evaluating the relationship between Operation Lifesaver and collisions between trains and motor vehicles at public crossings, p. 6.

[10]D2006 Executive Committee meeting, September 27-28, 2004, Calgary.

LIST OF ACRONYMS

EXECUTIVE SUMMARY

1.0 BACKGROUND

1.1 Introduction

1.2 Program Profile

1.3 Program Logic Model

1.4 Evaluation Rationale

1.5 Evaluation Limitations

2.0 RELEVANCE AND DEMAND

3.0 PROGRAM SUCCESS AND IMPACTS

4.0 PROGRAM EFFICIENCY AND COST-EFFECTIVENESS

5.0 RECOMMENDATIONS

Annex 1: Program Profile

Annex 2: Evaluation Plan, Methodology and Data Sources

Annex 3: List of Stakeholders Interviewed

Annex 4: Master Question List for Interviews

Annex 5: Transportation Safety Board Statistics

Average