Canadian Flag Transport Canada / Transports Canada Government of Canada
Common menu bar (access key: M)
Skip to specific page links (access key: 1)
Transport Canada - Corporate Services
* Departmental Evaluation Services
* About Us
* What's New
* Services
* Evaluation Overview
* Results-Based Management Overview
* Evaluation Reports and Other Documents
* Tool Kit
* Contact Us
   
* CS Overview

 

Skip all menus (access key: 2)


AN EVALUATION OF TRANSPORT CANADA'S MOVING
ON SUSTAINABLE TRANSPORTATION PROGRAM - January 13, 2006

[Printable Version]


LIST OF ACRONYMS

EXECUTIVE SUMMARY

1.0 BACKGROUND

1.1 Introduction

1.2 Program Profile

1.3 Program Logic Model

1.4 Study Rationale

1.5 Evaluation Issues

1.6 Methodology and Data Sources

2.0 PROGRAM RELEVANCE

3.0 PROGRAM SUCCESS AND IMPACTS

4.0 EFFICIENCY

5.0 COST-EFFECTIVENESS OF THE PROGRAM

6.0 CONCLUSIONS AND RECOMMENDATIONS

7.0 ANNEXES

Annex 1: MOST Program Eligibility Criteria

Annex 2: MOST Program Evaluation Criteria

Annex 3: Evaluation Framework

Annex 4: List of Key References

Annex 5: Questionnaire for “Successful” MOST Applicants

Annex 6: Questionnaire for “Unsuccessful” MOST Applicants

Annex 7: Interview questions

Annex 8: Case Studies

[page up]


LIST OF ACRONYMS

BSD Bathurst Sustainable Development
DES Departmental Evaluation Services
MOST Moving on Sustainable Transportation
NGO Non-governmental organization
SDS Sustainable Development Strategy
SOV Single-occupant-vehicle
STF Sustainable Transportation Fund
TC Transport Canada
TDM Transportation Demand Management

EXECUTIVE SUMMARY

Background

This report provides the results of an evaluation of Transport Canada’s (TC) Moving on Sustainable Transportation (MOST) program. An evaluation of the MOST Program was previously completed for Phase 1 (1999 to 2001). Therefore, the focus of this evaluation is on Phase 2, which refers to the period 2002 to June 2005, when the evaluation began.

The MOST Program, originally named the Sustainable Transportation Fund (STF), was launched in 1999 as part of the department’s first Sustainable Development Strategy (SDS), a three-year strategy that outlines challenges and commitments through which TC can better integrate sustainable development into its activities. This program supports projects that produce education, awareness, and analytical tools needed to make sustainable transportation a viable option for Canadians. The program is currently in its second phase with $2.5 million in funding to be allocated over the 2002 – 2007 period.

Objectives

The primary objectives of the MOST Program are to:

  • Stimulate the development of innovative tools, approaches and practices for increasing the sustainability of Canada’s transportation system;
  • Realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities; and
  • Provide Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives.

Findings

Relevance

  • The MOST Program is aligned with current government priorities for the environment and contributes to TC’s strategic outcome of an environmentally responsible transportation system. As outlined in TC’s SDS, the MOST Program strives to encourage Canadians to make more sustainable transportation choices.
  • There is an on-going demand for the MOST Program from stakeholders as well as a demand for longer-term funding for sustainable transportation projects.
  • There is a legitimate and necessary role for government in the MOST Program. It is the only Canadian federal contribution program that is dedicated to providing contributions for sustainable transportation projects. Additionally, federal involvement in these types of projects is viewed by MOST funding recipients as a stable source of funding, which has been used as leverage to obtain additional funding from other partners.

Success

  • In Phase 2, the MOST Program provided contributions to 46 projects that involved the development of studies/analyses, tools/practices, demonstration pilot projects, workshops/seminars, and education/outreach programs. These projects addressed a wide range of sustainable transportation issues such as public transit, urban planning and smart growth, and transportation demand management and employer programs.
  • It is difficult to determine the extent to which the MOST Program, as a whole, realizes quantifiable results for TC’s sustainable development priorities. An examination of the three case studies suggests that there are projects that have realized some quantitative environmental and sustainable development results. However, in other cases, the objectives of the projects do not intend to lead to quantitative results.
  • It is difficult to determine the extent to which the MOST Program, as a whole, provides Canadians with practical information and tools for better applying sustainable transportation thinking to their lives. An examination of the case studies suggest that, at a project level, the MOST Program is experiencing some success in achieving outcomes consistent with this program objective. However, it is difficult to assess overall success because it is still too early to report on longer-term results for most of these projects.
  • In varying degrees, all of the actions outlined in the management action plan for the last evaluation were implemented. Some of the actions were only partially implemented because the program did not have the resources to fully carry them out. However, for those actions that were fully implemented, the impact was not as great as expected in most cases.

Efficiency

  • The evaluation found some areas that may need improvement including: the proposal evaluation process, the process for disbursing funds, application and reporting requirements, the timeframe between funding notification and ministerial announcement, and website management. However, stakeholders and MOST Program management perceived the overall program administration to be efficient.

Cost-effectiveness

  • While a full cost-effectiveness analysis could not be done due to limitations in performance reporting, a case can tentatively be made that the MOST Program does provide value for the funding provided. The department is achieving some positive outcomes that contribute to an environmentally responsible transportation system at a low cost.

Conclusions and Key Recommendations

The Evaluation Team concludes that some positive results are being achieved for the investments made by TC and in this context, TC should continue to manage and administer the MOST Program.

Some key recommendations stemming from the evaluation seek to address stakeholder demand for longer-term funding, deal with issues with performance reporting and improve certain aspects of program delivery.

a) Demand for longer-term funding
  • TC should examine the feasibility of modifying the MOST Program’s terms and conditions to provide project funding beyond the current two-year maximum.

b) Performance Reporting

  • MOST Program management should revise the five project categories outlined in the program’s eligibility criteria (see Annex 1) so that they are mutually exclusive. This could facilitate the roll-up of performance data to describe program level results.
  • Given the challenges in ensuring reliability and validity of performance reporting, TC should reassess the program objective to “realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities.”
  • The MOST Program should consider ways to simplify and streamline the reporting of performance data so that it is commensurate with the level of funding allocated, and the scope and complexity of the project. One recommendation would be to create a questionnaire or form that would identify the key indicators that need to be collected and reported on. This questionnaire could be distributed to funding recipients at the start of the project and returned at the conclusion of the project. The reliability and validity of data would be improved, performance data from different projects could be aggregated to describe program level performance, and the process for collecting and reporting on performance data would be simpler for stakeholders.

c) Program Delivery

Communication with Stakeholders

  • Given that previous efforts to gather stakeholder feedback were not very successful, the MOST Program management should consider alternate ways to reach target groups. An adapted version of the questionnaires used as part of this evaluation (see Annex 6 and 7) could be administered to both unsuccessful applicants and funding recipients.

Program Exposure

  • MOST Program management should continue its current efforts in expanding program exposure. However, it should also consider alternative ways to promote the program beyond the website. For example, management could explore opportunities to showcase the program at environmental or transportation events, particularly in underrepresented areas.

Timeliness in Program Delivery

  • MOST Program management should ensure that stated timelines are adhered to in the proposal evaluation process. In cases when delays may occur, management should openly communicate the reasons for the delay.
  • To improve the timeliness in disbursement of funds, the MOST Program management should consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones.
  • Where other TC groups affect the timeliness of program delivery areas, the MOST Program management should enter into discussions with these groups to re-examine the processes associated with these areas and assess options to improve processing time.

Resources for Program Delivery

  • TC should reassess the number of human resources devoted to the administration of the MOST Program and determine if additional resources are required to improve program delivery and to support the implementation of the recommendations of this evaluation.

[page up]


1.0 BACKGROUND

1.1 Introduction

This report provides the results of an evaluation of Transport Canada’s (TC) Moving on Sustainable Transportation (MOST) program. The Departmental Evaluation Services Branch (DES) undertook the evaluation at the request of the Environmental Affairs Directorate of TC’s Programs Group in order to meet the requirements of Treasury Board Secretariat’s Policy on Transfer Payments, in effect since June 2000. The Policy requires that a program evaluation be performed before renewal of any transfer payment program.

1.2 Program Profile

Background

The MOST Program, originally named the Sustainable Transportation Fund (STF), was launched in 1999 as part of the department’s first Sustainable Development Strategy (SDS), a three-year strategy that outlines challenges and commitments in which TC can better integrate sustainable development into its activities. This program supports projects that produce education, awareness, and analytical tools needed to make sustainable transportation a viable option for Canadians. It was initially established as a three-year program with $1.5 million in resources to be allocated to various projects designed to encourage sustainable transportation practices among the Canadian population. In response to an on-going demand, the program was extended to 2007 and was provided with $2.5 million in additional funding to be allocated over the 2002 – 2007 period.

Objectives

The primary objectives of the MOST Program are to:

  • Stimulate the development of innovative tools, approaches and practices for increasing the sustainability of Canada’s transportation system;
  • Realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities; and
  • Provide Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives.

Roles and Responsibilities

The administration of the program is the responsibility of the Urban Transportation Programs group of the Environmental Initiatives Branch at TC. There is one full-time program officer and one manager to oversee the general administration of the program, which includes processing application submissions, coordinating the proposal evaluation process, liaising with stakeholders, and managing contribution agreements. To assist them with web management and financial matters, a web officer and a finance officer are borrowed from other branches in the Programs and Divestiture group, when required.

There is also a MOST Advisory Committee, which is an independent, multi-stakeholder committee that is responsible for reviewing application submissions and allocating contributions. This committee is chaired by TC and consists of up to 10 members, with representatives from other federal government departments, the not-for-profit sector, municipal governments, industry, and academia.

Program Delivery Description

There are two funding rounds per fiscal year with application deadlines of June 1 and December 1. Applicants submit a project proposal that must meet certain eligibility criteria (see Annex 1). Once applications are submitted, the MOST Program Officer conducts a mandatory criteria evaluation, which serves as a pre-screening process to ensure that each application does indeed meet the eligibility criteria. The MOST Program Officer then distributes all of the proposals to the MOST Advisory Committee members.

The MOST Advisory Committee members have a minimum of three weeks to review proposals against the evaluation criteria (see Annex 2). In the Committee’s overall selection, consideration is given not only to the evaluation criteria but also to the regional balance of projects, the variety of initiatives, as well as, a balance among projects that are innovative and those which build on existing initiatives. The Committee uses consensus to make recommendations for funding. Final decision-making authority rests with the Director General of Environmental Affairs.

Applicants are officially notified of funding decisions by email approximately three months after the closing date of each application deadline. As a condition of the notification, successful applicants are not allowed to publicly disclose that they have been allocated funding from the MOST Program until the ministerial announcement. However, during this time, the contribution agreements can be signed and funding recipients may commence their projects.

Funding recipients are required to submit two reports for multi-year projects: a mid-term report and a final report. For projects that are one year or less, only a final report is required. Funding recipients are also required to account for all project expenditures and revenues. Once the final reports are submitted, the MOST Program management assess the final reports and process invoices for payments. Reimbursement from the MOST Program is based on actual project expenditures and actual leveraged funding from sources other than the program.

MOST Program Contributions

Maximum TC funding per individual project is $100,000 over two years with the contribution not exceeding 50 percent of eligible costs. The remaining 50 percent of project costs must be obtained from other sources. Approximately $350,000 may be disbursed per round but the actual disbursement depends on the quality of the projects. Table 1 on the next page displays the actual amount awarded in each round since the launch of the MOST Program.

Table 1: Amount of MOST Funding Awarded by Round

Phase Date Funding Round Amount of Funding Awarded ($)
 1 December 1999 1 $189,500
  June 2000 2 $185,971
  December 2000 3 $310,436
  June 2001 4 $167,408 (+$25,041)
 2 June 2002 5 $350,000
  December 2002 6 $320,000
  June 2003 7 $226,650
  December 2003 8 $266,330
  June 2004 9 $448,194
  December 2004 10 $407,000
  June 2005 11 $350,000

Note: “+” indicates additional money allocated but not spent

1.3 Program Logic Model

Table 2 shows the logic model for the MOST Program. The logic model was originally created as part of a Results-based Management and Accountability Framework (RMAF) in 2001, at the time of the last program renewal.

Table 2: MOST Logic Model

Activity Reach Immediate Outcome  Intermediate Outcome Ultimate Outcome
TC makes contributions to external organizations to conduct sustainable transportation projects (up to 50% of total project cost) Primary
  • Not-for-profit organizations, community groups, organizations/associations for First Nations/Aboriginal groups, education and academic institutions, businesses, and industry/professional associations

Secondary

  • Canadian public, including but not limited to the transportation sector, municipalities, First Nations and Aboriginal Peoples, and educators
Completion of the following types of sustainable transportation projects:
  • Studies or analyses that contribute to a greater understanding of sustainable transportation issues
  • Innovative sustainable transportation tools and practices
  • Demonstration pilots testing new sustainable transportation approaches/alternatives
  • Workshops, strategy sessions and seminars that bring people together in support of new sustainable transportation ideas/ approaches
  • Education and outreach programs which inform the Canadian public
  • Increased awareness and understanding of sustainable transportation issues
  • Increased awareness of the range of solutions available for addressing sustainable transportation challenges
  • Increased application of sustainable transportation practices
  • Promotion of partnerships and alliances to achieve sustainable development results
  • Development and testing of innovative tools, approaches and practices in support of sustainable transportation
  • Quantifiable environmental and sustainable development results
  • An environmentally responsible transportation system that contributes to Canada’s sustainable development objectives
  • Reduction of greenhouse gas emissions and pollution from the transportation sector
  • Prevention and mitigation of environmental damage from transportation activities

1.4 Study Rationale

The study was conducted to assess the program’s relevance, success, efficiency, and whether there are alternative, more cost-effective ways of delivering the program in order to provide input for future decision-making.

An evaluation of the MOST Program was previously completed for Phase 1 (1999 to 2001). Therefore, the focus of this evaluation is on Phase 2, which refers to the period 2002 to June 2005, when the evaluation began. More specifically, the Evaluation Team assessed the results achieved by MOST-funded projects that were funded between Rounds 5 and 9. The projects in Round 10 and Round 11 had not yet begun when the evaluation started.

1.5 Evaluation Issues

The evaluation study addressed the following questions:

A. Program Relevance

  1. What public policy objectives are to be achieved by the MOST Program? How does it align with current government priorities and advance the strategic outcomes of TC?
  2. Is there an on-going demand for the MOST Program and, if so, what will be its magnitude?
  3. Is there a legitimate and necessary role for the government in this program?

B. Program Success and Impacts

  1. To what extent does the MOST Program stimulate the development of innovative tools, approaches, and practices to increase sustainability of Canada’s transportation system?
  2. To what extent does the MOST Program realize quantifiable results on TC’s sustainable development priorities?
  3. To what extent does the MOST Program provide Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives?
  4. Have changes introduced to the MOST Program since the last evaluation had a positive effect?

C. Efficiency

  1. If the program continues, how could its efficiency be improved?

D. Cost-effectiveness

  1. Is the program the most cost-effective means of achieving the intended objectives? How do program delivery costs compare to those in other jurisdictions and the private sector for similar activities and outcomes?

1.6 Methodology and Data Sources

The Evaluation Team developed an evaluation strategy and selected various methods for collecting the data needed to address the evaluation questions. See Annex 3 for the evaluation framework.

Methodology

File/Document review

The Evaluation Team reviewed relevant websites and various documents from the program files including application submissions, final project reports, final report assessments, and MOST Advisory Committee Records of Decisions. A list of key references can be found in Annex 4.

Questionnaire

In July 2005, questionnaires were administered by direct mail to successful and unsuccessful applicants of the MOST Program. (See Annex 5 and 6 for the questionnaires.) Follow-up packages were sent one month later to non-respondents. The primary purpose of the questionnaires was to obtain stakeholder views about the program. The response rate for the unsuccessful applicants’ questionnaire was 31% with a sample size of 22. The response rate for the successful applicants’ questionnaire was 53% with a sample size of 25.

Interviews

The Evaluation Team conducted face-to-face interviews with the MOST Program management and the chair of the MOST Advisory Committee in order to obtain their views about the MOST Program. In addition, the Evaluation Team also conducted telephone interviews with three funding recipients as part of the Case Study method (see below). The list of interview questions can be found in Annex 7.

Case Studies

Case studies were conducted to gain insight about the success of the MOST Program. Given that MOST-funded projects vary widely in scope, case study methodology provides detailed analyses of specific projects. The Evaluation Team used the following criteria to select three case studies:

  • The case was funded in early rounds, such as Round 5 and 6
  • The case had completed its project as stated in its objectives
  • The case involved the development of a tool or an adoption of a sustainable transportation practice
  • The case represented a different region of the country
  • The case fell into different MOST project type categories (i.e. Studies or Analyses, Tools and Practices, Pilot Projects, Workshops or Information Sessions, Education and Outreach)

The three case studies chosen are Bathurst Sustainable Development, Pollution Probe and Smart Growth BC. These case studies are presented in Annex 8.

Limitations

The evaluation relied primarily on performance information provided by funding recipients in their final project reports to evaluate the success of the program. While these documents provided some useful information at the project level, the reliability and validity of the data varied from project to project. As a result, it was difficult to compile results and draw conclusions at the program level. The case study approach was adopted in order to address this limitation.

[page up]


2.0 PROGRAM RELEVANCE

  1. What public policy objectives are to be achieved by the MOST Program? How does it align with current government priorities and advance the strategic outcomes of TC?

Finding: The MOST Program is aligned with current government priorities for the environment and contributes to TC’s strategic outcome of an environmentally responsible transportation system. As outlined in TC’s Sustainable Development Strategy, the MOST Program strives to encourage Canadians to make more sustainable transportation choices.

The MOST Program advances TC’s third strategic outcome of an environmentally responsible transportation system that contributes to Canada’s sustainable development objectives. In TC’s 2005 – 2006 Report on Plans and Priorities, TC commits to continue developing and implementing programs to achieve a more sustainable transportation system. TC recognizes that sustainable transportation is a shared responsibility and commits to working with partners and stakeholders, such as the general public, non-governmental organizations (NGOs), and industry. The MOST Program is an example of a program that contributes to an environmentally responsible transportation system.

The MOST Program is aligned with current government priorities for the environment as outlined in the Speech from the Throne. The design of the MOST Program promotes partnerships in sustainable transportation. This is consistent with the Government of Canada’s commitment to work with partners to build sustainable development systematically. The focus of the MOST Program on innovative projects is consistent with the Government of Canada’s recognition that “human ingenuity will turn increasingly to ways to produce and use energy more cleanly and efficiently; to eliminate toxins from our air, water, and soil, and to build more sustainable communities.” Furthermore, some of the projects funded by the MOST Program have contributed to the Government’s commitment to the Kyoto Accord on climate change.

TC’s 2004 – 2006 SDS is the third successive three-year strategy developed by the department. It outlines seven challenges and 32 specific commitments in which the department can do better to integrate sustainable development into its activities. The seven challenges are:

  1. Encourage Canadians to make more sustainable transportation choices
  2. Enhance innovation and skills development
  3. Increase system efficiency and optimize modal choices
  4. Enhance efficiency of vehicles, fuels, and fuelling infrastructure
  5. Improve performance of carriers and operators
  6. Improve decision-making by government and transportation sector
  7. Improve management of TC’s operations and lands

The MOST Program is identified as one of the commitments to address the first challenge of encouraging Canadians to make more sustainable transportation choices. This is to be done by creating awareness and educating Canadians on the issues, benefits, and trade-offs, as well as practices and choices that individuals can adopt to reduce the adverse impacts of transportation on the environment. It involves partnerships with other federal departments, other levels of government, industry, NGOs, and other stakeholders. Changing behaviour is an essential part of the response to the environmental impact of transportation. All segments of society need to understand the impacts of their transportation behaviour in order to make choices that reduce the adverse effects of transportation on the environment.

Although the MOST Program was identified as a specific commitment to address the first challenge in TC’s SDS, the outcomes of the program also contribute to some of the other SDS challenges. For example, the MOST Program’s outcome of “Development and testing of innovative tools, approaches and practices in support of sustainable transportation” contributes to the second SDS challenge of “Enhancing innovation and skills development”.

  1. Is there an on-going demand for the MOST Program and if so, what will be its magnitude?

Finding: There is an on-going demand for the MOST Program from stakeholders as well as a demand for longer-term funding for sustainable transportation projects.

There is an on-going demand for the MOST Program from stakeholders. Demand for MOST funding as indicated by the number of requests for funding continues to remain strong. Table 3 displays the number of applications submitted in each round. The average number of applications in Phase 1 is 24 and the average number of applications in Phase 2 is 25. This suggests that the number of requests for funding has generally remained consistent.

Table 3: Number of Applications Submitted to the MOST Program

Phase  Date Round Number of Applications
1 December 1999 1 22
  June 2000 2 22
  December 2000 3 29
  June 2001 4 21
2 June 2002 5 31
  December 2002 6 30
  June 2003 7 16
  December 2003 8 18
  June 2004 9 21
  December 2004 10 34
  June 2005 11 22

The demand for the MOST Program is expected to continue. According to questionnaire responses, an overwhelming majority of successful applicants said that they would re-apply for MOST funding (92%). Similarly, a majority of unsuccessful applicants said that they would re apply (65%).

Through their contact with stakeholders, MOST Program management reported that there is a demand for more stable and longer-term funding for on-going projects. The Evaluation Team made the same observation based on the questionnaire responses. At present, the MOST Program provides funding to short-term, innovative projects. Due to the terms and conditions of the program regarding project length, on going projects cannot be funded.

According to MOST Program management, the kinds of projects being submitted for funding have changed over time. There is a shift away from awareness-building type projects to projects aimed at changing behaviour. MOST Program management described this shift as the “natural evolution of a program” where early years are dedicated to building awareness. MOST Program management believe there is now an increased level of understanding among the Canadian public on the topic of sustainable transportation. In this way, the MOST Program management feel that the need for the program has changed. There is less need for projects aimed at educating Canadians about sustainable transportation and more of a need for projects aimed at encouraging Canadians to adopt sustainable modes of transportation.

  1. Is there a legitimate and necessary role for government in this program?

Finding: There is a legitimate and necessary role for government in the MOST Program. It is the only Canadian federal contribution program that is dedicated to providing contributions for sustainable transportation projects. Additionally, federal involvement in these types of projects is viewed by MOST funding recipients as a stable source of funding, which has been used as leverage to obtain additional funding from other partners.

There are a number of other partners typically involved in MOST-funded projects besides TC, including other federal government departments, other levels of government (i.e. local, regional, and provincial), businesses, not-for-profit organizations, foundations and associations, research labs, and academic institutions. Even though there are a number of other partners involved, TC remains a very important player in funding sustainable transportation projects.

MOST is the only Canadian federal contribution program that is dedicated to providing funding for innovative sustainable transportation projects. Other environmental programs at TC do not have the same direct objective. There are programs at other federal departments that may provide some funding to sustainable transportation projects, such as Environment Canada’s Climate Change Action Fund, One Tonne Challenge and EcoAction or Natural Resources Canada’s Fleetsmart Program. However, these programs address the broader issue of climate change or general environmental issues rather than focusing solely on sustainable transportation projects. Some provincial or municipal governments, industry associations, and private foundations may also provide funding to sustainable transportation projects. However, these programs also address broader objectives which are not necessarily focused on sustainable transportation.

In general, contributions from other sources of funding are less than the contributions from the MOST Program. Compared to other sources of funding, the MOST Program provides funding up to 50% of total project costs. Without the federal contribution, smaller organizations would have to find additional sources of funding to make up for the contributions that could have been provided through the MOST Program.

Table 4 suggests that the MOST Program was a primary funding source and did contribute to the achievement of funded projects’ stated objectives. This role of the MOST Program is considered to be very important by successful applicants. In open-ended responses of the questionnaire, the successful applicants speculated that if MOST funding was not available, their projects would not have occurred, the projects would have resulted in delays, or the projects would have been forced to scale back to a more limited focus.

Table 4: Rating of the Role of the MOST Program by Successful Applicants

 

Average score on a 1-5 scale

Statement Agreement with Statement  Importance of Statement Gap
The MOST Program was my main source of funding 3.8 4.8 1.0
Overall, the MOST Program helped to meet my project’s stated objectives 4.6 4.8 0.2

The speculations of the successful applicants are substantiated by the responses from the unsuccessful applicants, in which the majority said that without MOST funding, their projects did not occur. In some cases, unsuccessful applicants were able to find other sources of funding but the focus of the original project changed and they were forced to launch a smaller-scale version of the original project.

Additionally, MOST funding has been used by some not-for-profit organizations to leverage funding from other partners because funding from the federal government is seen as a stable source of funding. This funding arrangement has been vital for smaller organizations, which already have limited resources. It also helps to promote partnerships between other organizations and governments.

MOST stakeholders also believe TC should take a leadership role in sustainable transportation because transportation activities have a wide range of impacts on the environment, including resource use (materials and energy), undesirable residuals (emissions, spills and leaks), and land use, including impacts on wildlife. In particular, the transportation sector has a significant impact on climate change as it is the single largest contributor to greenhouse gas emissions. One stakeholder remarked as follows:

Sustainable transportation should be one of our most pressing environmental, social, and economic priorities and the federal government needs to allocate appropriate resources to address the challenges involved. The MOST Program is an important part of a menu of effective tools for the Federal Government in taking action on sustainable transportation.

[page up]


3.0 PROGRAM SUCCESS AND IMPACTS

  1. To what extent does the MOST Program stimulate the development of innovative tools, approaches, and practices to increase sustainability of Canada’s transportation system?

Finding: In Phase 2, the MOST Program provided contributions to 46 projects that involved the development of studies/analyses, tools/practices, demonstration pilot projects, workshops/seminars, and education/outreach programs. These projects addressed a wide range of sustainable transportation issues, such as public transit, urban planning and smart growth, and transportation demand management and employer programs.

During Phase 2, the MOST Program provided a contribution to 46 projects across the country. The majority of projects funded in Phase 2 originated from British Columbia, Ontario, and Quebec, accounting for 78% of all projects funded. The remaining 22% of projects were initiated in Alberta, New Brunswick, Nova Scotia, and Saskatchewan.

Sustainable transportation is a broad topic and the MOST Program provides contributions for projects that address a wide range of sustainable transportation issues. Given the diversity of MOST-funded projects, three case studies were selected for this evaluation as examples (see Table 5). For more information on these case studies, see Annex 8.

Table 5: Project Descriptions of Case Studies

Project Title Organization Project Description
1. Urban Transportation Project: Addressing Climate Change in the City of Bathurst, New Brunswick – Feasibility Study Bathurst Sustainable Development Bathurst Sustainable Development conducted a study to examine the feasibility of introducing a bus service in the City of Bathurst.
2. S-M-A-R-T Movement Program Pollution Probe Pollution Probe launched a workplace-based trip reduction program directed at reducing employee single-occupant-vehicle trips. This is both an information resource and support service to help guide medium to large organizations that want to reduce employee car trips.
3. Tillicum Burnside Urban Village Community Roundtable Smart Growth BC This project consisted of a community-led visioning design roundtable called, “a charrette” with the ultimate objective to transform the Tillicum Burnside community from an automobile-dominated neighbourhood into a pedestrian and transit oriented urban village setting.

More details on all others projects that have received contributions from the MOST Program can be found on the program’s website, http://www.tc.gc.ca/programs/environment/most/menu.htm

  1. To what extent does the MOST Program realize quantifiable results on TC’s sustainable development priorities?

Finding: It is difficult to determine the extent to which the MOST Program, as a whole, realizes quantifiable results for TC’s sustainable development priorities. An examination of the three case studies suggests that there are projects that have realized some quantitative environmental and sustainable development results. However, in other cases, the objectives of the projects do not intend to lead to quantitative results.

It is difficult to determine the extent to which the MOST Program, as a whole, realizes quantifiable results on TC’s sustainable development priorities. MOST-funded projects vary widely in scope and each project tends to measure and report on different performance indicators and targets. As well, the methods for collecting this performance information are inconsistent. Therefore, the quantitative performance data across all projects is not always comparable, making it difficult to compile all the data and draw conclusions about the kinds of quantifiable results produced by the MOST Program.

MOST Program Managers have indicated that they prefer measures such as, “gas saved”, “kilometres travelled saved”, and “modal shifts before and after a project” to describe quantitative environmental or sustainable transportation results. Of the three case studies, Pollution Probe’s S M-A-R-T Movement Program was the only project to produce this kind of information. In the first phase of its project, Pollution Probe could only conduct follow-up surveys with two out of the five participating companies within the timeframe it had to report to the MOST Program. The results in Table 6 reflect early results on the environmental impact of this project.

Table 6: S-M-A-R-T Follow-up Employee Transportation Survey Results (2003)

  Change in Commuting Mode from Baseline   Average Air Pollutants Emitted (grams)

Energy Intensity

  Transit Carpool Active Tele- work SOV Other Avg distance to work Change in Avg Distance from baseline Per employee /day Change from baseline Per employee per km Change from baseline
Company A 2.6% 4.3%  0.9% - -5.1% -2.5% 26 km -5.22 km 10512.4 -2748.43 202.16 -10.22
Company B  -1% 2% 1% - -2%    26 km +2 km  10896.1 +705.2  209.54 -2.77

Company A experienced a decrease in single-occupant-vehicle (SOV) usage and a decrease in vehicle kilometres travelled. Consequently, this led to a decrease in air pollutants emitted and energy intensity. Similarly, Company B experienced a slight decrease in SOV usage. However, it also experienced a decrease in transit usage. Consequently, this led to an increase in vehicle kilometres travelled and average air pollutants emitted. But, overall energy intensity had decreased.

The provision of these types of quantitative environmental results is not always feasible due to the focus or scope of some projects. For example, in the case of Smart Growth BC’s project, the charrette produced a report with recommendations for changes to make the Tillicum Burnside community more sustainable transportation-friendly. The charrette report itself could not directly lead to environmental or sustainable transportation results. Rather, the recommendations of the charrette would have to have been implemented in order to assess the environmental impact. Smart Growth BC could not be expected to deliver on providing quantitative environmental or sustainable transportation results. Furthermore, the environmental results may not be visible except in the long-term, where the attribution between the charrette report and environmental results would be somewhat weak.

Similarly, Bathurst Sustainable Development (BSD) could not be expected to provide quantitative environmental results because the project was a study to examine the feasibility of implementing an urban transit system. The actual implementation of the bus system itself had not taken place when BSD submitted its final report to the MOST Program. Despite this limitation, BSD did provide some estimates about the environmental impact if a transit system were to be implemented (see Annex 8 for more details).

An examination of the three case studies suggests that some projects can realize direct quantitative environmental or sustainable transportation results, but the objectives of others do not intend to lead to quantitative results.

  1. To what extent does the MOST Program provide Canadians with practical information and tools for better applying sustainable transportation thinking to their lives?

Finding: It is difficult to determine the extent to which the MOST Program, as a whole, provides Canadians with practical information and tools for better applying sustainable transportation thinking to their lives. An examination of the case studies suggests that, at a project level, the MOST Program is experiencing some success in achieving outcomes consistent with this program objective. However, it is difficult to assess overall success because it is still too early to report on longer-term results for most of these projects.

The extent to which the MOST program provides Canadians with practical information and tools for better applying sustainable transportation thinking to their lives can be demonstrated, in part, by the number of projects that are still on-going. Table 7 displays the total number of MOST funded projects in each round and the number of projects that have continued after MOST funding had ended. More than half of the projects in all of the rounds are still on-going after MOST funded ended. It is interesting to note that all of the projects in Round 7 are still on-going.

Table 7: Number of Projects That Have Continued After Funding Ended

Round Number of funded projects Number of Projects that have Continued
 5 12  7 (+2)
6 10 7 (+2) 1 not completed yet
7 7  7
8 8 5
2 not completed yet

Note: “+” indicates the number of additional projects that are possibly on-going

However, this indicator only reveals part of the story since some MOST-funded projects are not intended to continue after MOST funding. All projects that receive contributions from the MOST Program must report on the following:

  • Level of exposure of the project (i.e. Number of individuals and/or organizations exposed to the project’s activities, reports, findings, etc.);
  • Level of increased awareness among the target audience of the project as well as the spin-off awareness of the project to third parties (i.e. through follow-up surveys, third-party interviews, self-reported increase in awareness, etc.);
  • Level of increased awareness of possible solutions available for addressing sustainable transportation challenges; and
  • Extent of adoption and implementation of innovative sustainable transportation activities, tools, and practices.

In general, all of the recipients reported that their projects received exposure in varying ways and that there was increased awareness among the target audience of the project itself as well as increased awareness of possible solutions available for addressing sustainable transportation challenges. The extent of adoption and implementation of innovative sustainable transportation activities, tools, and practices also varied depending on the project scope. Beyond these general observations, it is difficult to aggregate the results in a meaningful way to determine the extent to which the MOST Program, as a whole, provides Canadians in different communities with practical information and tools for better applying sustainable transportation thinking to their lives.

It is easier to observe the extent to which activities, tools, and practices have been implemented at the individual project level. This was examined using our case study approach. A summary of how each project meets this program objective is provided in this section. For more details, see Annex 8.

Case Study #1: Urban Transportation Project: Addressing Climate Change in the city of Bathurst, New Brunswick – Feasibility Study (Bathurst Sustainable Development)

The study concluded there was sufficient ridership demand for a bus service, which has the potential of being financially sustainable within two to three years. BSD estimated that the bus system has the potential to significantly reduce greenhouse gas emissions.

In a follow-up interview, the Evaluation Team learned that BSD is currently assisting the City of Bathurst with a pilot project to test a bus system in the city. Early results reveal a steady increase in ridership levels.

Case Study #2: S-M-A-R-T Movement Program (Pollution Probe)

In the first phase of its project, Pollution Probe recruited five pilot organizations to participate in the S-M-A-R-T Movement Program. In the second phase, they recruited another seven for a total of twelve workplaces. Results of follow-up surveys were conducted and submitted to the MOST Program with participating companies where the program had been operational long enough. These early results revealed that participants generally became more aware about the impact of transportation on the environment and the benefits of sustainable transportation. As shown in Table 6, there was some behavioural change in Company A, which resulted in having a positive impact on the environment, while there was less behavioural change in Company B.

In working with twelve companies, Pollution Probe was not progressing as far along as they would have liked. In its experience, sustaining this type of voluntary project has been challenging for Pollution Probe. In order to be able to implement the employer program, Pollution Probe noted the need to get senior management on board by highlighting the economic benefits of a program. Through its efforts, Pollution Probe has tried to engage senior management of participating companies and has achieved limited success.

Case Study #3: Tillicum Burnside Urban Village Community Roundtable (Smart Growth B.C.)

The project produced a charrette report, which contained recommendations to improve the Tillicum Burnside community including:

  • Improve pedestrian and bicycle circulation and amenities on all street rights of way
  • Implement traffic calming measures (particularly by reducing vehicle traffic lane widths), to discourage speeding and improve the safety and comfort of pedestrian sidewalks and street crossings
  • Utilize zoning and pre-zoning strategies as incentives to owners and developers to invest in building and infrastructure, especially along Gorge Road, Tillicum Road and Burnside Road
  • Reduce the dominant visual and operational impact of vehicles by discouraging large surface parking areas and encouraging shared and underground parking

In a follow-up interview with Smart Growth BC, the Evaluation Team learned that the District of Saanich adopted a Streetscape Action Plan in June 2005, which incorporated many of the charrette report’s recommendations. The District of Saanich has allocated some funding for the first round improvements on Burnside Road and construction is slated to begin in Spring 2006.

The results of the case studies suggest that the MOST Program is experiencing some success in achieving outcomes consistent with this program objective. However, it is difficult to assess overall success because it is still too early to report on longer-term results for most of these projects.

  1. Have changes introduced to the MOST Program since the last evaluation had a positive effect?

Findings: In varying degrees, all of the actions outlined in the management action plan for the last evaluation were implemented. Some of the actions were only partially implemented because the program did not have the resources to fully carry them out. However, for those actions that were fully implemented, the impact was not as great as expected in most cases.

There were three key recommendations in the 2001 evaluation of the MOST Program. The recommendations were offered to assist the program management in expanding program exposure, obtaining better and more timely information on client needs, and implementing more effective results measurement and reporting.

The first recommendation was to enhance the feedback and consultative mechanisms of the MOST Program by doing the following:

  • Making use of future SDS consultation sessions to determine stakeholder preferences for program design and management
  • Enhancing mechanisms to solicit greater feedback on program design and management from applicants, funding recipients, and non-applicants on the target group

Table 8 summarizes the MOST management action plan in response to this recommendation and the impacts of those actions on the program.

Table 8: Summary and Impact of Management Action for Recommendation #1

Management Action Plan Response Action Implemented? Impact
Management said they would use the SDS consultations to gather feedback on the program. Yes. The MOST Program was discussed at some SDS consultation sessions. While stakeholders appreciated the funding opportunities provided through MOST, many found the Program’s relatively small budget and short-term funding problematic. MOST stakeholders believed that TC should consider the feasibility of expanding this funding program by supporting larger and broader initiatives and by providing continued funding to help sustain “proven, successful” programs. No impact. MOST Program management is aware of stakeholder concerns; however, these concerns cannot be fully addressed as they fall outside of the Program’s current objectives.
Management would update the feedback mechanism on the website. Yes. The MOST website was updated to include a feedback mechanism for users to rate each individual web page and to submit any written comments. No impact.
The program had not received any completed feedback forms. Management would conduct a phone survey with funding recipients to solicit feedback on the management and design of the program. Yes, to a small extent. MOST Program Managers believed there was an attempt to conduct phone surveys in 2000 or 2001 but there were no responses. No impact. Phone surveys have not been conducted on an on going basis due to the level of effort required and lack of resources.

The MOST Program management carried out most of the actions as planned in response to the first recommendation. Although the actions taken to gather stakeholder feedback are not having the desired impact, they are not having a negative impact on the program. The MOST Program management should consider alternate ways to reach target groups and solicit feedback. One suggestion would be to adapt the questionnaires used as part of this evaluation (see Annex 6 and 7).

The second key recommendation in the 2001 evaluation of the MOST Program was to promote MOST success stories by:

  • Encouraging funding recipients to disseminate project findings, results, and success stories
  • Continuing to promote actively the MOST Program, particularly to areas under-represented in terms of project submissions

Table 9 summarizes the MOST Program management’s response to this second recommendation and the impact of those actions.

Table 9: Summary and Impact of Management Action for Recommendation #2

Management Action Plan Response Action Implemented? Impact of Action
Management said they were going to modify the program eligibility criteria to include a requirement for the recipient to disseminate any materials of results. Yes. As part of the application submission, applicants must outline their communication plan. Good impact. In general, the projects have implemented their communication plans and are getting good project exposure in their respective communities.
Management said they would highlight projects on the website. Yes, partially. The MOST website includes descriptions of projects which are categorized on the site by subject category or funding round. Little impact. The website is not updated on a regular basis to include final project reports due to limited resources.
Management said they would promote the program to under-represented regions. Yes. MOST Program Managers spoke to regional offices as well as some organizations in under-represented regions. Some impact. There has been an increased in applications from some of the regions where the Program was promoted. However, some areas remain under-represented, including the territories.

The first action described in Table 9 was carried out by management and appears to be having the desired impact. However, the second action is not achieving its full impact. This is partly due to the lack of regular website updates. Given the difficulties in regularly maintaining the website, MOST Program management may wish to consider alternate ways to promote success stories. At the same time, the MOST Program management may wish to consider alternative ways to expand program exposure in order to achieve a regional balance of project submissions.

The third key recommendation in the last evaluation was to promote the use of the results-based approach to program management by:

  • Building the capacity of program staff and funding recipients to demonstrate results through the use of results-based management tools and practices at the project level
  • Updating the MOST performance measures framework to more accurately identify the key results of the program and effective methods for measuring progress in this regard.

Table 10 presents the summary and impact of the management action in response to this recommendation.

Table 10: Summary and Impact of Management Action for Recommendation #3

Management Action Plan Response Action Implemented? Impact of Action
Management said they would work with the Program Evaluation Branch in the development of project planning and reporting sheets, which will help recipients better design quantifiable deliverables and performance indicators. Yes. The Program adopted a “Performance Indicators – Project Planning Table” as part of the application process. This document identifies a project’s objectives and associated performance indicators and targets that will help to determine whether the objectives have been met. Funding recipients are required to report on these indicators in their final project reports. Weak impact. There tends to be some confusion among applicants regarding performance indicators.
Management said they would update the program’s RMAF in order to meet TB requirements. Yes. Minimal impact. MOST Program management did deliver the program as outlined in the RMAF. However, the performance data collected did not provide useful information at the program level.

Of the two actions in Table 10, the most problematic change has been the requirement for recipients to report on performance indicators. The “Performance Indicators – Project Planning Table” document has resulted in much confusion by applicants when they are completing their application forms. The program does offer assistance to help clarify the concepts for applicants but this involves considerable time and effort of the MOST Program Officer.

In addition, the format of this document leads the applicant to develop performance indicators that reflect activities and outputs, rather than outcomes. The applicant may not be aware of this error due to lack of understanding of results-based management concepts. As a result, recipients, who complete these forms as part of their reporting requirements are not providing accurate performance information. Furthermore, the performance information that has been collected is not very useful to describe program level performance.

[page up]


4.0 EFFICIENCY

  1. If the program continues, how could its efficiency be improved?

Finding: The evaluation found some areas that may need improvement including: the proposal evaluation process, the process for disbursing funds, application and reporting requirements, the timeframe between funding notification and ministerial announcement, and website management. However, stakeholders and MOST Program management perceived the overall program administration to be efficient.

Using the questionnaire method, both successful and unsuccessful MOST applicants were asked to provide feedback on various aspects of program delivery. Overall, the MOST Program received average to above average scores on a 1 to 5 agreement and importance scale for nine program delivery aspects. This suggests that these stakeholders are generally satisfied or very satisfied with the delivery of the MOST Program.

Table 11 displays a summary of the ratings from the questionnaires for successful applicants.

Table 11: Rating of Program Delivery by Successful Applicants

   

Average scores on 1–5 scale

  Statement Agreement with Statement Importance of Statement Gap
1. The MOST website gave me the proper information I needed in order to complete the application requirements. 4.8 4.4  0.4
 2. The MOST website is a useful source of information about the program. 4.7 4.4 0.3
3. The mid-term reporting requirements were easy to meet. 4.6 4.3  0.3
4. The MOST Program Staff provided information that was useful to me. 4.5 4.7 0.2
5. The application requirements were clearly described. 4.4 4.8 0.4
6. The final reporting requirements were easy to meet. 4.3 4.6 0.3
 7. The MOST Program Staff responded to my inquiries in a reasonable time. 4.2 4.7 0.5
 8. The process of evaluating proposals was clear and transparent. 4.1 4.6 0.5
9. The proposal evaluation was done in a reasonable time. 3.8 4.6 0.8

Note: n = 25

On average, successful applicants highly rated the usefulness of the MOST website (4.7), especially in order to obtain the proper information to complete application requirements (4.8). Additionally, there was a high rating given to the ease of completing mid-term reporting requirements, where applicable (4.6). Although timeliness of the proposal evaluation received the lowest rating, the satisfaction level is still slightly above average (3.8). These findings suggest that the MOST Program is performing to the satisfaction of successful applicants.

Table 12 displays a summary of the ratings from the questionnaires for the unsuccessful applicants.

Table 12: Rating of Program Delivery by Unsuccessful Applicants

 

Average scores on 1 – 5 scale

  Statement Agreement with Statement Import8of Statement Gap
1. The MOST Program Staff responded to my inquiries in a reasonable time. 4.8 4.5 0.3
2. The MOST Program Staff provided information that was useful to me. 4.7 4.1  0.6
3. The application requirements were clearly described. 4.0 4.7 0.7
 4. The MOST website is a useful source of information about the program. 4.0 4.5 0.5
5. The MOST website gave me the proper information I needed in order to complete the application requirements. 3.8 4.4 0.6
6. The proposal evaluation was done in a reasonable time. 3.0 4.4 1.4
7. The process of evaluating proposals was clear and transparent. 2.9 4.7 1.8

Note: n = 22

On average, the unsuccessful applicants highly rated the timeliness of MOST Program Staff responses to inquiries (4.8) and the usefulness of the information provided by MOST Program Staff (4.7). The program delivery aspects that received the lowest ratings were the timeliness of the proposal evaluation process (3.0) and the clarity and transparency of the proposal evaluation process (2.9). These two aspects also received above average ratings of importance, which results in a large gap. This gap suggests an area for improvement in the MOST Program. Some comments about the proposal evaluation process by unsuccessful applicants are as follows:

Faster evaluation of proposals with faster reply to proposer whether project will be funded.

It is essential to communicate openly the timeframe for the approval process. The extreme time lag in our application process could have jeopardized our funding relationships with 2 other funders.

The notification for the 2005-2006 year came 3 months past stated timeline. This caused some issues re. having to redesign program timelines and deliverables and staff.

Currently, the MOST Program clearly indicates on the website that the expected timeframe for decisions on proposal evaluation is approximately three months after the application submission deadline. According to MOST Program management, the proposal evaluation process has generally been completed within this timeframe throughout Phase 2, with the exception of Round 10, where there was an unusual delay. Although the MOST Program has demonstrated consistent timeliness in carrying out the proposal evaluation process, the comments above suggest that some applicants feel three months is not a reasonable timeframe.

Comments by unsuccessful applicants about the transparency of the proposal evaluation process are as follows:

The evaluation process was a black box and very little feedback was provided about why the project failed – I would have expected more transparency for funding on this scale.

Offer opportunity to present, clarify or answer questions on proposals to help educate, explain or elaborate on project rationale.

The MOST Program management already has a mechanism in place to address the issue of transparency. After receiving notification of decision by email, applicants can telephone the Program Management and request a debrief about the proposal evaluation process.

Aside from the above aspects of program delivery, stakeholders also identified other areas of inefficiency through open-ended responses of the questionnaires. These are as follows: the process for disbursing funds, the application and reporting requirements, the timeframe between funding notification and ministerial announcement, and website management.

a) The Process for Disbursing Funds

Stakeholders expressed some concern that the process for disbursing funds is not done in a timely manner. This is considered to be problematic for not-for-profit organizations, which already have limited resources. Stakeholders would like to see improved turnaround time for signing contribution agreements, processing invoices, and disbursing funds. Below are some responses from successful applicants outlining their concerns.

In the 2003-2004 Program year, there were problems with responses to queries but most importantly to payments…Payments were months late, which caused considerable cash flow issues (and stress) and did impact our reputation in some ways.

We had difficulty in wrapping up the approval for the final payment. MOST took in excess of 150 days to approve and distribute final payment. I hope this is not typical. Non-profits (small) have cash flow requirements that make it difficult to wait until the end of the project before receiving remaining funding (in our case half the awarded amount). I suggest a 75% or 80% up-front payment.

…with respect to not receiving 50% of the funding until the final report was received. As a small non-profit with a small revenue stream, it was difficult to cover all expenses up front before receiving much of the funding. If two or three progress reports could be submitted as the project progressed in order to secure the release of smaller instalments spread over the year, it would have been easier on our treasurer.

MOST Program management is aware of this concern. At present, MOST Program Managers rely on a finance officer borrowed from another group to facilitate this process. Without additional resources, management cannot speed up the administrative process leading up to the disbursement of funds. To address the issue of financial hardship among funding recipients under the current arrangement, management should consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones.

Also, MOST Program management noted that the Finance and Administration Directorate is responsible for actual release of funds. While the Program Management tries to facilitate the process as much as possible, timing of actual disbursement is beyond their control. The MOST Program management should enter into discussions with the Finance and Administrative group to re-examine this process and assess options to improve processing time.

b) Application and Reporting Requirements

Stakeholders noted that application and reporting requirements are onerous, particularly for not-for-profit organizations with limited resources. They are concerned that much time and resources are taken away from the project itself in order to complete applications or collect information for reporting purposes. As some stakeholders noted:

Application is tremendously time-consuming.

Less extensive reporting and application process. Granted, projects have to be accountable for the funds that they receive. We have found that application and reporting time (while necessary) have sometimes compromised project operations.

It is important (and useful for us) to report on the project but a more streamlined reporting mechanism allows for more human resources allocated to the project itself.

The final reporting was too onerous time-consuming needs to be simplified without losing its effectiveness.

MOST Program management should consider ways to simplify the application process and streamline the reporting strategy. The level of reporting should be commensurate with the level of funding allocated and take into consideration the scope and complexity of the project.

c) Delays between notification and announcement

Stakeholders raised some concern about the timeframe between funding notification and ministerial announcement. As one stakeholder noted:

…there was a significant delay in announcing the funding recipients which caused some difficulties with respect to staffing and human resources at [name of organization]. This delay was owing to the Provincial election in Fall 2003 and thus was beyond the control of the MOST program, but this point is simply to stress the importance, especially to smaller non-profits, of making funding announcements on time.

MOST Program management are aware of the lag time between funding notification and ministerial announcement. However, this process is also beyond their control as the timing of the ministerial announcement is coordinated between TC’s Communications Group and the Minister’s Office. Management should advise the Communications Group that the timeliness of ministerial announcement is important.

d) Website Management

The website serves as an important tool not just for completing applications but also to learn about what others are doing in the area of sustainable transportation. Several applicants have raised the following concerns:

A lot of information [on the website] but not user-friendly.

Post [reports for] successful applicants. [The website] is ~2 years out-of-date.

Due to the technical nature of website management and lack of expertise of current MOST Program staff to maintain the website, MOST Program management borrows a web officer from another group to assist them in this area. Given the importance of the website as a communication tool to stakeholders, the MOST Program management may require additional resources to regularly update the website, in particular to post final project reports in a timely manner.

[page up]


5.0 COST-EFFECTIVENESS OF THE PROGRAM

  1. Is the program the most cost-effective means of achieving the intended objectives? How do program delivery costs compare to those in other jurisdictions and the private sector for similar activities and outcomes?

Finding: While a full cost-effectiveness analysis could not be done due to limitations with performance reporting, a case can tentatively be made that the MOST Program provides value for the funding provided. The department is achieving some positive outcomes that contribute to an environmentally responsible transportation system at a low cost.

A full cost-effectiveness analysis could not be done because the Evaluation Team experienced difficulties in selecting a single measure of effectiveness. This is due to limitations with performance reporting by funding recipients as well as the fact that outcomes varied from project to project. As an alternative, the Evaluation Team compared the costs and projected outcomes to provide some indication of the cost-effectiveness of the MOST Program.

Table 13 shows an estimate of the relevant costs in the MOST Program. Even when including resources covered by other budgets, MOST is a low cost program.

Table 13: Annual Cost Estimates for the MOST Program

Cost Component

Cost to Transport Canada

Cost to MOST funding recipients

Overhead
  • Facilities
  • Equipment & Materials
  • Translation
  • $25,000.001
 
Labour
  • 1 MOST Program Officer
  • 10 MOST Advisory Committee Members3
  • 1 Manager
  • 1 part-time Web person
  • 1 part-time finance officer
  •  $62,893.201
  •  $108,480.002
  •  $21,226.082
  •  $8,801.642
  •  $6,880.562
 
MOST contributions
  • up to 50% of all project costs
  • $700,000.001
 
Required inputs from funding recipients
  • MOST recipients must secure other sources of funding (either cash or in-kind)
  • $700,000.00
 
Total Costs  $932,281.48  $700,000.00

1 Costs covered under the MOST Program’s budget
2 Costs covered under another budget
3 This cost is an estimate of the value for the volunteer services provided by the MOST Advisory Committee Members.

Table 14 displays a comparison of the annual costs and projected outcomes using logic and qualitative information. As noted in Question 3, MOST Program is the only Canadian federal contribution program that is dedicated to providing significant contributions for sustainable transportation projects. Since there are no other similar programs in terms of activities and outcomes, the only comparison that could be made is the scenario of providing no funding.

Table 14: Comparison of MOST Program Costs and Projected Outcomes

  Scenario 1 – Status Quo Scenario 2 – Provide no funding
TC’s contribution $932,281.48 $0
Outcomes Client organizations receive necessary funding to initiate innovative sustainable transportation projects. Client organizations would have to search for other sources of funding. This would present a challenge because no other Canadian federal contribution program dedicated specifically for sustainable transportation projects exists.
  Partnerships and alliances between organizations are developed to achieve sustainable development results. Federal government would not be a partner in working towards sustainable development results.
  Increased awareness of sustainable transportation issues among targeted audience of projects. Less awareness of sustainable transportation issues among targeted audience of projects.
  TC demonstrates its commitment to contribute to an environmentally responsible transportation system. TC must rely on other programs to demonstrate its commitment to contribute towards an environmentally responsible transportation system
    Sustainable transportation projects would not take place or would occur on a smaller scale than now.

Table 14 suggests that a case can be tentatively made that the MOST Program does provide value for money. The department is achieving some positive outcomes that contribute to an environmentally responsible transportation system at a low cost.

[page up]


6.0 CONCLUSIONS AND RECOMMENDATIONS

The Evaluation Team concludes that some positive results are being achieved for the investments made by TC and in this context, TC should continue to manage and administer the MOST Program. It is a relevant program for TC to be involved in. The program is successfully meeting its stated objectives at the project level and is generally being delivered by TC in an efficient manner. This program is a cost-effective way for TC to achieve sustainable transportation outcomes.

The section below presents the key recommendations stemming from the evaluation.

Demand for longer-term funding

There is a consistent demand for longer-term funding for sustainable transportation projects by its stakeholders. Through the SDS consultations, responses to questionnaires, and discussions with MOST Program management, stakeholders have requested for this type of program expansion. At present, the terms and conditions of the program outline that funding is dedicated for innovative projects up to a maximum of two years. On-going projects cannot be funded. Therefore, many of the MOST-funded projects are short-term and are accountable to the program for a short timeframe. For several of these projects, not enough time would have passed in order for the project to demonstrate any impact on the environment.

Recommendation:

TC should examine the feasibility of modifying the MOST Program’s terms and conditions to provide project funding beyond the current two-year maximum.

Performance Reporting

There are a few challenges with project performance reporting that made it difficult to evaluate the extent to which the MOST Program was successful in meeting its stated objectives at the program level as well as conduct a full cost-effectiveness analysis.

  1. Lack of Ability to Categorize MOST-Funded Projects

Although there are five project categories outlined in the program’s eligibility criteria (see Annex 1), the majority of MOST-funded projects fall under more than one project category. Since MOST-funded projects vary widely in scope and cannot be grouped into categories, it was difficult to aggregate performance data by project type in order to draw meaningful comparisons at a program level. If distinct project categories are constructed, this could facilitate the roll up of performance data to describe program level results.

Recommendation:
MOST Program management should revise the five project categories outlined in the program’s eligibility criteria so that they are mutually exclusive. This could facilitate the roll-up of performance data to describe program level results.

  1. Extensive Reporting Strategy

Stakeholders have raised some concerns that the reporting strategy is onerous and time consuming. While some stakeholders recognize the value of monitoring performance in demonstrating the progress of their projects, they believe that the current reporting strategy requires too much of their time and resources.

  1. The Reliability and Validity of Performance Reporting

The reliability and validity of the data provided to the MOST Program may be questioned as MOST funding recipients are responsible for collecting and reporting performance information for their projects. The reliability of the data is a concern as some MOST funding recipients, especially not-for-profit organizations, do not have the necessary financial and human resources to collect quality performance data. As well, there is no consistency with regards to the type of performance data that is collected. Each project tends to measure and report on different performance indicators and targets. Furthermore, the methods used to collect data tend to vary, which makes it difficult to compare across projects and aggregate results.

The validity of the data is a concern because much of the performance data that tends to be reported reflects performance indicators for activities and outputs, not environmental or sustainable transportation results. This is understandable given that data for activities and outputs are easier to collect and are more readily available within the short timeframe of a project’s life. Lack of understanding of performance measurement and results-based management concepts by funding recipients may also be an issue.

These limitations make it particularly difficult for the MOST Program to “realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities” – even at the individual project level.

Recommendations:
Given the challenges in ensuring reliability and validity of performance reporting, TC should reassess the program objective to “realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities.”

The MOST Program should consider ways to simplify and streamline the reporting of performance data so that it is commensurate with the level of funding allocated, and the scope and complexity of a project. One recommendation would be to create a questionnaire or form that would identify the key indicators that need to be collected and reported on. This questionnaire could be distributed to funding recipients at the start of the project and returned at the conclusion of the project. The reliability and validity of data would be improved, performance data from different projects could be aggregated to describe program level performance, and the process for collecting and reporting on performance data would be simpler for stakeholders.

 

Program Delivery

There are a few areas of program delivery in which the MOST Program management can improve its performance, including communication with stakeholders, program exposure, resource availability and timeliness of program delivery.

  1. Communication with Stakeholders

The 2001 evaluation recommended that the MOST Program management needed to obtain better and more timely information on client needs. Although the MOST Program attempted to do so by using SDS consultations, conducting telephone follow-up surveys and website surveys to gather feedback, these changes had almost no impact on the program.

Recommendation:

Given that previous efforts to gather stakeholder feedback were not very successful, the MOST Program management should consider alternate ways to reach target groups. An adapted version of the questionnaires used as part of this evaluation (see Annex 6 and 7) could be administered to both unsuccessful applicants and funding recipients.

  1. Program Exposure

Since the last evaluation, MOST Program management attempted to expand program exposure in order to achieve a better regional balance of project submissions and to promote MOST success stories using the website. Although the actions taken to reach underrepresented regions are not having a negative impact, a regional imbalance of application submissions persists. The website is an effective way to promote the MOST Program but management encounters difficulties in maintaining the website on a regular basis.

Recommendation:

MOST Program management should continue its current efforts in expanding program exposure. However, it should also consider alternative ways to promote the program beyond the website. For example, management could explore opportunities to showcase the program at environmental or transportation events, particularly in underrepresented areas.

  1. Timeliness in Program Delivery

Stakeholders raised some concerns about the timeliness of three program delivery areas: proposal evaluation, disbursement of funds, and ministerial announcements of successful projects.

The program’s website clearly states that the expected timeframe for decisions on proposal evaluations is approximately three months after the application submission deadline. The MOST Program should ensure that these timelines are adhered to or revise the standard if it is unrealistic. In cases when delays may occur, the MOST Program management should openly communicate to all applicants explaining the reasons for the delay.

The timeliness in disbursement of funds is important to funding recipients, particularly for not for-profit organizations with limited resources. To improve this process, the MOST Program management may consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones.

The reasons for delays in disbursement of funds as well as the ministerial announcements of successful projects are also often beyond the control of the MOST Program management. TC’s Communications Group is responsible for coordinating the ministerial announcement of projects while TC’s Finance and Administration Directorate is responsible for disbursement of funds. MOST Program management should enter into discussions with these TC groups to re-examine the processes associated with these areas and assess options to improve processing time.

Recommendations:

MOST Program management should ensure that stated timelines are adhered to in the proposal evaluation process. In cases when delays may occur, management should openly communicate the reasons for the delay.

To improve the timeliness in disbursement of funds, the MOST Program management should consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones.

Where other TC groups affect the timeliness of program delivery areas, the MOST Program management should enter into discussions with these groups to re-examine the processes associated with these areas and assess options to improve processing time.

  1. Resources for Program Delivery

TC staff devoted to the MOST Program consists of one full-time program officer and one manager, who also manages other environmental programs. To assist with web management and financial matters, a web officer and a finance officer are borrowed from other branches in TC’s Programs Group. The level of staff time devoted to the MOST Program appears to be insufficient for efficient and timely program delivery. Given that the finance officer and web officer must focus on their primary duties first, the MOST Program management may not be able to improve the processing time leading up to the disbursement of funds or regularly maintain the program’s website.

Recommendation:

TC should reassess the number of human resources devoted to the administration of the MOST Program and determine if additional resources are required to improve program delivery and to support the implementation of the recommendations of this evaluation.

[page up]


7.0 ANNEXES

Annex 1: MOST Program Eligibility Criteria

The following five criteria must be met for projects to be eligible for program funding:

  1. Address at least one of the following MOST Program categories:
  1. Conduct studies or analyses that contribute to a greater understanding of sustainable transportation issues;
  2. Develop innovative sustainable transportation tools or practices;
  3. Undertake demonstration pilot projects that test new sustainable transportation approaches or alternatives;
  4. Conduct workshops, strategy sessions or seminars that bring people together in support of new sustainable transportation ideas or approaches; or,
  5. Deliver education and outreach programs that inform the Canadian public about a sustainable transportation activity.
  1. Target the Canadian Public
  2. Proposals must target the Canadian public including, but not limited to, the general public, the transportation sector, youth, municipalities, First Nations and Aboriginal Peoples, and educators.


  3. Shared Funding through Partnerships
  4. The proposal must obtain a minimum of 50 percent of resources (cash and/or in-kind* from sources other than the Government of Canada, so that the federal contribution does not exceed 50 percent (cash and/or in-kind).

    *Note: Proposals with a combination of in-kind and cash support will be viewed more favourably than those with only in-kind support.


  5. Demonstrate Quantifiable Results:
  6. Proposals must contain quantifiable sustainable transportation targets and performance indicators to measure and report on the environmental and sustainable development impacts expected as a result of the project. Each proposed target must have an associated performance indicator. Qualitative targets and performance indicators can complement the quantifiable ones.


  7. Sharing Results and Program Materials:
  8. Proposals must contain a detailed communication plan for the dissemination of any project-related materials or results that will serve to further the program’s objective of providing Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives.

    The communication plan must include the following details:

    • Method of dissemination (e.g. Internet, mail-outs, newspaper advertisement, workshops, etc);
    • If the Internet is the main method for sharing information, how will the web site be promoted;
    • Expected target audience; and;
    • Expected reach (e.g. number of: web site users, recipients of mail-out materials, participants at a workshop, etc.).

    [page up]


Annex 2: MOST Program Evaluation Criteria

If a project meets the mandatory eligibility criteria, the following evaluation criteria is used to assess the relative strength of each proposal.

Effectiveness in making direct environmental improvements through greater understanding and practical applications of sustainable transportation principles (60%) The degree to which the project proposal:
  1. increases the ability of Canadians to apply concrete and practical sustainable transportation solutions
  2. promotes action by Canadians to adopt sustainable transportation practices in their daily lives
  3. reinforces/strengthens current initiatives and mechanisms addressing sustainable to expand their influence, instill behaviour change and motivate action; and,
  4. provides concrete milestones and expected results within a reasonable timeframe, and provides a process to monitor progress and measure the project’s impacts

 

Innovative solutions (25%) The degree to which the project is replicable and can be applied elsewhere and the extent to which it:a. provides a novel and creative approach for promoting and realizing sustainable transportation; or,b. builds on, rather than duplicates, existing approaches or initiatives.
Experience and competence (15%) The degree to which the project demonstrates a likely chance of success based on:
  1. the proponent’s experience and expertise in similar areas;
  2. the proponent’s demonstrated commitment to the area; and,
  3. sufficient institutional, management/organizational structure and financial and other support to successfully deliver the project.

 

Other criteria

In its overall selection, the MOST Advisory Committee will give consideration to the following (in addition to the above-described criteria) to ensure an assortment of initiatives are funded which produce results supportive of program objectives:

  • regional balance;
  • variety of initiatives; and,
  • balance among projects which are innovative and those which enrich existing initiatives.

[page up]


 

Annex 3: Evaluation Framework

Evaluation Issue Indicator Source Method
Relevance      
  1. What public policy objectives are to be achieved by the MOST Program? How does it align with current government priorities and advance the strategic objectives of TC?
  • The consistency between the MOST Program and federal government and TC’s objectives and strategic priorities
  • The consistency between the MOST Program and the department’s Sustainable Development Strategy
  • Speech from the Throne
  • TC’s Report on Plans and Priorities
  • TC’s Sustainable Development Strategy
  • File/Document Review
  1. Is there an on-going demand for the MOST Program and, if so, what will be its magnitude?
  • Level of demand for the MOST Program among stakeholders
  • Increase in number of requests for MOST funding (using 1999 as a baseline)
  • Stakeholder feedback
  • Program Managers feedback
  • Program Files
  • Stakeholders
  • Program Managers
  • File/Document Review
  • Interviews/ Questionnaires
  1. Is there a legitimate and necessary role for government in this program?
  • The present level of involvement in projects by other levels of government, voluntary sectors and other stakeholders
  • Stakeholder feedback
  • Program Manager feedback
  • Program Files
  • Stakeholders Program Managers
  • File/Document Review
  • Interviews/ Questionnaires Success
  1. To what extent does the MOST Program stimulate the development of innovative tools, approaches, and practices to increase sustainability of Canada’s transportation system?
  • Number and types of projects funded
  • Stakeholder feedback
  • Program Manager feedback
  • Program Files
  • Stakeholders
  • Program Managers
  • File/Document Review
  • Interviews
  • Case Studies
  1. To what extent does the MOST Program realize quantifiable results on TC’s sustainable development priorities?
  • Assessment of quantitative data for different outcomes
  • Program Manager feedback
  • Program Files
  • Stakeholders
  • Program Managers
  • File/Document Review
  • Interviews
  • Case Studies
  1. To what extent does the MOST Program provide Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives?
  • Level of exposure of the project
  • Level of increased awareness among the target audience of the project as well as spin-off awareness
  • Level of increased awareness of possible solutions
  • Extent of adoption and implementation of innovative sustainable transportation activities, tools, and practices
  • Percentage of projects that continue after MOST funding ended
  • Program Files
  • Stakeholders
  • Program Managers
  • File/Document Review
  • Interviews
  • Case Studies
  1. Have changes introduced to the MOST Program since the last evaluation had a positive effect?
  • Extent to which the Program Managers implemented its action plan with success
  • Stakeholders
  • Program Managers
  • Interviews/ Questionnaires Efficiency
  1. If the program continues, how could its efficiency be improved?
  • Stakeholder feedback
  • Program Manager feedback
  • Stakeholders
  • Program Managers
  • Interviews/ Questionnaires Cost-effectiveness
  1. Is the program the most cost-effective means of achieving the intended objectives? How do program delivery costs compare to those in other jurisdictions and the private sector for similar activities and outcomes?
  • Comparison of costs and outcomes of the MOST Program compared to not providing Program
  • Comparison of MOST program with other programs (e.g. in TC’s Environmental Affairs, Environment Canada)
  • Environmental Affairs’ website
  • Environment Canada websites
  • Other programs’ websites
  • Website review
  • Cost-effectiveness modelling

[page up]


Annex 4: List of Key References

Bathurst Sustainable Development Website

Smart Growth BC Website

Pollution Probe Website

Moving on Sustainable Transportation Website
http://www.tc.gc.ca/programs/environment/most/menu.htm

TC’s Sustainable Development Strategy 2004 – 2006
http://www.tc.gc.ca/programs/Environment/SD/menu.htm

TC’s Report on Plans and Priorities 2005 – 2006

Speech from the Throne, October 5, 2004

Consultation Report: Toward Sustainable Transportation – Transport Canada’s Third Sustainable Development Strategy

MOST Evaluation Report (2001)

MOST Management Action Plan (2001)

[page up]


Annex 5: Questionnaire for “Successful” MOST Applicants

MOST Invites Your Feedback!

  1. How did you first learn about the MOST program?
Media Internet
Word of Mouth Government publication
Other (please specify) ___________________________________________
  1. Please circle the number to show to what extent you are in agreement with each statement, and then, what level of importance this aspect of the MOST program has for you.
Statement Agreement Importance
  Strongly Disagree     Strongly Agree Not Important At All      Very Important
The application requirements were clearly described 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
 The process of evaluating proposals was clear and transparent. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The proposal evaluation was done in a reasonable time. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The mid-term reporting requirements were easy to meet. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The final reporting requirements were easy to meet 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST program staff responded to my inquiries in a reasonable time 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST program staff provided information that was useful to me. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST website gave me the proper information I needed in order to complete the application requirements 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST website is a useful source of information about the program 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
  1. What would have been the impact on your project if the MOST funding were not available?






  1. Did your project continue after MOST funding?
Yes No

Please explain.






  1. What other types of funding partners were involved in your MOST-funded project?
    (Check all that apply)
Other federal government departments Provincial governments
Municipal governments Businesses or industries
Not-for-profit organizations Educational and academic institutions
Other (please specify)_______________________________
  1. Are you aware of other programs that provide funding for sustainable transportation projects?
Yes No (please skip to Question 7)

Please identify the funding programs. How do these other funding programs compare to MOST?






  1. Would you apply for MOST funding again?
Yes (please skip to Question 8) No

If no, why not?






  1. Please circle the number to show to what extent you are in agreement with each statement and then, what level of importance of this statement has for you.
Statement Agreement Importance
  Strongly Disagree     Strongly Agree Not Important At All      Very Important
Overall, the MOST program helped to meet my project’s stated objectives 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST program was my main source of funding 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
  1. What changes or improvements could be made to the MOST program?





  1. What was your role in the MOST-funded project?

Project leader

Project team member

Other (please specify) _______________________________

  1. Any other comments?





Thank you for completing this survey!

All information will be kept confidential and protected by the Access to Information Act and the Privacy Act.

[page up]


Annex 6: Questionnaire for “Unsuccessful” MOST Applicants

MOST Invites Your Feedback!

1. How did you first learn about the MOST program?

Media Internet
Word of Mouth Government publication
Other (please specify) ___________________________________________

2. Please circle the number to show to what extent you are in agreement with each statement and then, what level of importance this aspect of the MOST program has for you.

Statement Agreement Importance
  Strongly Disagree     Strongly Agree Not Important At All      Very Important
The application requirements were clearly described. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The process of evaluating proposals was clear and transparent.

 

1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The proposal evaluation was done in a reasonable time.

 

1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST program staff responded to my inquiries in a reasonable time. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST program staff provided information that was useful to me.

 

1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST website gave me the proper information I needed in order to complete the application requirements. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
The MOST website is a useful source of information about the program. 1 2 3 4 5 Don’tKnow N/A 1 2 3 4 5 Don’tKnow N/A
  1. How did the absence of MOST funding impact your proposed project?





  1. Did your project still go ahead without MOST funding?
Yes No
  1. Are you aware of other programs that provide funding for sustainable transportation projects?
Yes No  (please skip to Question 6)

Please identify the funding programs. How do these other funding programs compare to MOST?






  1. Would you apply for MOST funding again?
Yes (please skip to Question 7) No

If no, why not?






  1. What changes or improvements could be made to the MOST program?





  1. Any other comments?





Thank you for completing this survey!

All information will be kept confidential and protected by the Access to Information Act and the Privacy Act.

[page up]


Annex 7: Interview questions

Interview Questions for MOST Program Managers

  1. Please describe the process for administering the MOST Program.
  2. How and to what degree does the MOST Program contribute towards TC’s Sustainable Development Strategy objectives?
  3. What is the current need or demand for the MOST Program? Is there an on-going need or demand for the MOST Program?
  4. What other types of partners are usually involved with the projects funded by MOST?
  5. Is the current cost-sharing funding approach the most cost-effective for achieving the program’s objectives? If not, are there any viable alternatives?
  6. To what extent has the MOST Program been successful in achieving or contributing to its stated objectives?
  7. Have you identified other benefits resulting from the program other than the MOST objectives/
  8. Have you identified any unwanted outcomes resulting from the program?
  9. To what extent did you implement the Management Action Plan from the previous evaluation? What impact did those changes have on the program?
  10. If MOST funding were not available, what would be the impact on the following:
    1. Canadians;
    2. Transport Canada;
    3. TC’s Sustainable Development Strategy?
    4. Government of Canada
    5. Provinces/Territories
  11. Are you aware of any other programs with similar objectives? How do they compare to MOST?
  12. Are there any changes and/or improvements that you would like to see made to the MOST Program?
  13. Do you have any other comments?

Interview Questions for Chair of the MOST Advisory Committee

  1. How was the MOST Advisory Committee originally established? How are members selected?
  2. Describe the process for evaluating MOST proposals.
  3. What works well with the proposal evaluation process?
  4. What could be improved with the proposal evaluation process?
  5. Do you have any other comments?

Interview Questions for Case Study Stakeholders

  1. What progress has been made with the project?
  2. What performance information has been collected so far?
  3. What will success of this project look like? How would you describe the success of the project so far?
  4. How important was MOST funding to the success of this project?
  5. What challenges, if any, are you facing in carrying out this project?
  6. Are you aware of any “spin-off” benefits as a result of this project?
  7. Any other comments?

Annex 8: Case Studies

Three case studies were conducted for this evaluation. The primary objective of the case study approach was to gain insight into the longer-term results of selected MOST-funded projects. The case study approach was also be used to gather information on the following specific evaluation questions, which focuses on the behavioural change and outcome-focused objectives of the program:

  • To what extent does the MOST Program stimulate the development of innovative tools, approaches, and practices to increase sustainability of Canada’s transportation system?
  • To what extent does the MOST Program realize quantifiable results on TC’s sustainable development priorities?
  • To what extent does the MOST Program provide Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives?

Case Study #1: The Urban Transportation Project (Bathurst Sustainable Development)

Background

 In Round 6, the MOST Program awarded $30,000 to Bathurst Sustainable Development (BSD) for a project called, “The Urban Transportation Project: Addressing Climate Change in the City of Bathurst, New Brunswick – Feasibility Study”.

The Bathurst City Council had decided to implement an urban transit system “in order to assist citizens with improving their quality of lives and to provide them with a transportation option other than their personal automobiles as a means of reducing their greenhouse gas emissions.” The Council asked BSD to conduct a study to examine the feasibility of introducing a bus service in the City of Bathurst. The objective of the study was to identify challenges and potential opportunities for implementing an urban transit system in the City of Bathurst, which would include an assessment of citizen needs as well as research into the financial and operational aspects of starting a bus system.

In addition to MOST, BSD had partnerships with the City of Bathurst, the Federation of Canadian Municipalities, SMT-Acadia Bus Lines, Dupon Trolley Industries, Codiac Transit, and Natural Resources Canada. While there are a number of other partners involved, MOST funding was described by BSD as “critical” to the project. “Having Transport Canada as a partner is critical to bringing credibility to the project and encourages the participation of others.”

The feasibility study was completed on April 30, 2004. BSD conducted the following activities as part of the study.

  • Conducted 53 public consultation sessions and a city-wide survey with potential user groups (sample size = 5,775)
  • Hosted an interactive workshop session with commercial business owners/operators
  • Held a live transit demonstration, providing free rides to citizens
  • Worked with partners to select and finalize details of implementing a bus service

In addition to submitting a final report to the MOST Program upon project completion, BSD compiled the findings of its study into a final study report, which is available on its website, http://www.bathurstsustainabledevelopment.com.

Results Achieved

  1. Increased awareness and understanding of sustainable transportation issues and of possible solutions available for addressing sustainable transportation challenges

There was a high level of exposure about the Urban Transportation Project in local media, provincial websites and other environmental websites. Over 500,000 individuals were reached through consultations, surveys, and various other outreach activities. On average, BSD tracked over 1,400 unique visits per month to the project’s website. The BSD report anecdotally observed, “The words, Transit, Sustainable Development and Climate Change have been in our local media and in the conversations of our population more in the past year than in all of the 25 years I have lived in the city.”

  1. Increased application of sustainable transportation practices

The study concluded that there was sufficient ridership demand for a bus service, which has the potential of being financially sustainable within two to three years. The study gathered necessary information that could facilitate the implementation of a bus service. For example, over 21 offers were made by various business to allow bus stop signage and schedules to be posted and for bus passes to be available and sold.

In a follow-up interview, the Evaluation Team learned that BSD is currently assisting the City of Bathurst with the Urban Transit Test Project – a pilot project to test a bus system in the city. This consists of the operation of a fleet of three buses, with 30-minute service around the city to test some of the points and recommendations of the feasibility study.

Furthermore, BSD would like to prepare and distribute a “How to Guide” for small cities wishing to start a financially sustainable public transit bus service. This would document lessons learned from its experience based on the feasibility study and the pilot project.

  1. Quantifiable Sustainable Transportation Results

Using the Greenhouse Gas Emissions Reduction Software designed by Torrie Smith Associates, BSD estimated that the bus system will divert 125,000 auto trips annually. Even taking into account the addition of transit vehicles, BSD estimated there would be significant reductions of various pollutants on a per trip basis as follows:

  • ~100,000 kilograms of carbon dioxide
  • ~10,000 kilograms of carbon monoxide
  • ~1,000 kilograms of NOX
  • ~500 kilograms of VOCx
The BSD’s final report suggested that the implementation of the transit bus service would allow individual citizens who use the transit service daily, instead of driving their personal automobiles, to reach 50-70% of their One Tonne Challenge goal of reducing their personal greenhouse gas emissions by one tonne annually.

BSD launched a pilot test of an urban transit system in May 2005. Although it is still too early to measure the environmental impact, performance information is being collected, such as ridership counts shown below.

Urban Transit Test Project Rider Counts (2005)

Month Number of Riders
June* 5821
July 1998
August 2375
September 3333

*Free transit was provided

In general, BSD has achieved positive early outcomes with this project. Since MOST funding ended, BSD was able to continue its efforts based on the feasibility study and is well on its way to providing Canadians in the City of Bathurst with the capacity to adopt more sustainable modes of transportation.

Case Study #2: S-M-A-R-T Movement Program (Pollution Probe)

Background

The MOST Program awarded $30,000 in Round 5 and $52,900 in Round 8 to Pollution Probe to pilot a two-phased project called, “S-M-A-R-T Movement”.

S-M-A-R-T Movement is a workplace-based trip reduction program that is directed at reducing employee single-occupant vehicle (SOV) trips. It is both an information resource and support service to help guide medium to large organizations that want to reduce employee car trips. S M A-R-T Movement aims to demonstrate to employers and employees that reducing SOV trips and vehicle kilometres travelled will save money and time, reduce stress and absenteeism, increase productivity, and address a number of broader issues such as traffic congestion, air quality, climate change, and other problems associated with urban sprawl. On a broader scale, S-M-A-R-T Movement aimed not only to achieve behavioural change amongst its participants, but also to strengthen support for the emerging approach of transportation demand management (TDM) in order to further its adoption throughout Canada.

In the first phase, the trip reduction program was initially directed at a selection of five pilot organizations in the Greater Toronto Area for the first year and 20 organizations in the Greater Toronto Area by 2005, with the intention of delivering the program across the country in other large urban centres in Canada. In the second phase, MOST funds were directed to the on-going support of the program.

According to stakeholder interviews, MOST Program was absolutely “critical” to the success of this project. Compared to other federal departments, TC is the only one with federal resources. In addition to MOST, other funding and service partners included the following:

  • Climate Change Action Fund
  • Laidlaw Foundation
  • City of Toronto
  • Clean Air Champions
  • Teletrips
  • Carpooltool.com
  • City of Toronto Bicycle Promotions and the Bicycle User Group Network
  • 20/20 The Way to Clean Air
  • One Tonne Toronto
  • Association of Commuter Transportation Canada
  • Sustainable Transportation Education Program Phase 2 (STEP 2) Advisory Committee (Green Communities Association – North Toronto Green Community)
  • Climate-Air Connections Hub (Clean Air Foundation)
  • Clean Air Sudbury
  • Smart Commute (Including Smart Commute Black Creek, Smart Commute Toronto and the Town of Markham)

Pollution Probe submitted a final project report to MOST in June 2005 and the S M A R T Movement Program is still on-going.

In both phases, Pollution Probe engaged in the following activities to launch the project:

  • Recruitment of participating organizations and partnership building
  • Program Implementation
    • Communications and Outreach
    • Evaluation (Surveys and Site Assessments)
    • Trip Reduction Plans
    • Workplace Implementation
    • Business Case Development

Results Achieved

  1. Promotion of partnerships and alliances to achieve sustainable development results

In the first phase, Pollution Probe recruited 5 pilot organizations to participate in the S-M-A-R-T Movement Program. In the second phase, they recruited another 7 for a total of 12 workplaces. Pollution Probe continues to engage a variety of project partners in S-M-A-R-T Movement. It was found that working in partnerships where possible allows for more efficient use of resources and builds on each the strengths of others. Organizational partnerships have been maintained with Clean Air Champions, Teletrips, the Town of Markham, Carpooltool.com, City of Toronto Bicycle Promotions and the BUG Network. These partnerships involved co-promotion through materials and events to S-M-A-R-T Movement participating workplaces as well as strategic discussions and information sharing about each other’s projects where appropriate. Pollution Probe’s S-M-A-R-T Movement has also partnered with a number of organizations on new projects that are mutually beneficial. In particular, Pollution Probe has made great strides in the further development of partnerships with the Greater Toronto Area and Hamilton Smart Commute Initiative.

  1. Increased awareness and understanding of sustainable transportation issues and of possible solutions available for addressing sustainable tranbrsportation challenges

In the first phase, follow-up surveys were conducted with participating companies. According to survey results, 68% of respondents felt they were more or significantly more aware of the benefits of sustainable transportation, transportation issues linked to smog/air quality, and transportation impacts linked to climate change and more or significantly more concerned with the health impacts/benefits of sustainable transportation. As well, there was an increase in employee knowledge of the S-M-A-R-T Movement program. Forty-six percent of respondents in one company and 82% of respondents in another company reported knowing about the program.

In the second phase, a follow-up survey with one company revealed the following results:

  • 57% agreed/strongly agreed that they are more aware of the benefits of sustainable transportation
  • 57% more aware of the transportation issues linked to smog/air quality
  • 47% more aware of the transportation impacts linked to climate change
  • 61% more aware of the cost savings incurred by using sustainable transportation
  • 50% more concerned about the health impacts and benefits of sustainable transportation
  • 21% try to leave my car at home more often for short trips (<5 km one way)

These results suggest that the S-M-A-R-T Movement program led to increased awareness and understanding of sustainable transportation issues by its participants as well as possible solutions available for addressing sustainable transportation challenges.

  1. Quantifiable Sustainable Transportation Results

The following are the results of the two workplaces where the program had been operational long enough to warrant a follow-up survey conducted in Phase 1.

S-M-A-R-T Follow-up Employee Transportation Survey Results (2003)

  Change in Commuting Mode from Baseline   Average Air Pollutants Emitted (grams)  Energy Intensity
  Transit Carpool Active Tele- work SOV  Other Avg distance to work Change in Avg Distance from baseline Per employee /day Change from baseline Per employee per km Change from baseline
Company A  2.6% 4.3% 0.9% - -5.1% -2.5% 26 km -5.22 km 10512.4 -2748.43 202.16 -10.22
Company B -1% 2% 1% - -2%    26 km +2 km 10896.1 +705.2 209.54 -2.77

The results in the above table reveal that Company A experienced a decrease in single-occupant-vehicle (SOV) usage and a decrease in vehicle kilometres travelled. Consequently, this led to a decrease in air pollutants emitted and energy intensity. Similarly, Company B also experienced a slight decrease in SOV usage. However, it also experienced a decrease in transit usage. Consequently, this led to an increase in vehicle kilometres and average air pollutants emitted. But, overall energy intensity had decreased.

  1. Increased application of sustainable transportation practices

In working with 12 companies, Pollution Probe was not progressing as far along as they would have liked. In its experience, sustaining this type of voluntary project has been challenging for Pollution Probe. In order to be able to implement the employer program, Pollution Probe noted the need to get senior management on board by highlighting the economic benefits of a program. Through its efforts, Pollution Probe has tried to engage senior management of participating companies and has achieved limited success. Given the operational difficulties with delivering this program, Pollution Probe has transferred its implementation to other organizations, including SMART Commute, 20/20, and Clean Air Foundation.

In general, Pollution Probe has achieved some positive immediate outcomes with this project. Although it encountered some challenges with on-going implementation, Pollution Probe continues to take steps towards the advancement of the TDM agenda. Pollution Probe felt the need to step back from implementation of this program in order to conduct some analysis. Next steps for Pollution Probe could include the completion of a report of case studies that documents its experiences with program execution and results in terms of modal shift, emission reductions, costs and savings. This shift in focus reveals that even though MOST funding ended for this project, Pollution Probe is continuing its efforts based on the S-M-A-R-T Movement Program to significantly expand knowledge regarding TDM in the Canadian context and to explore the need for TDM supportive policies on a broader regional and national basis.

Case Study #3: Tillicum Burnside Urban Village Community Roundtable (Smart Growth B.C.)

Background

In Round 5, the MOST Program awarded $20,000 to Smart Growth B.C. for a project called, “Tillicum Burnside Urban Village Community Roundtable”.

The project took place in the Gorge Tillicum neighbourhood in Saanich, B.C. The intersection of Tillicum Road and Burnside Road forms the centre of the neighbourhood. Both Tillicum and Burnside Roads consist of four wide lanes each from the TransCanada Highway to Craigflower Street. The rationale for the project grew out of concern that the neighbourhood had become an unsafe area. Both roads carry large automobile traffic volumes and the area lacks facilities for other users, such as pedestrians, cyclists or transit riders. When Smart Growth B.C. submitted its application to the MOST Program, it reported a vehicle count of approximately 23,000 per day on Tillicum Road and 22,500 vehicles per day on Burnside Road. It also reported that two deaths had occurred at this intersection and 500 motor vehicle accidents had taken place during a two-year period before its application.

To address these issues, Smart Growth B.C. engaged in a community-led visioning and design roundtable called, “a charrette”, with the ultimate objective to revise the automobile-dominated neighbourhood into a pedestrian and transit-oriented urban village setting.

In addition to MOST, Smart Growth BC had partnerships with the Gorge Tillicum Community Association, D’Ambrosio Architecture and Urban Design, the District of Saanich, and UBC James Taylor Chair. According to stakeholder interviews, MOST Program was “crucial” otherwise, the project would not have taken place. While they did receive some in-kind support, the architecture firm, which facilitated the charrette process, did require some payment. MOST funding was used to cover this cost.

The project was completed in March 2004. Smart Growth BC conducted the following activities as part of its project:

  • Created an advisory committee for the charrette process
  • Completed background research and preparation
  • Conducted 2 public information workshops
  • Selected a multi-stakeholder roundtable team
  • Hosted the charrette event
  • Promoted the charrette results to the District of Saanich

In addition to submitting a final report to the MOST Program upon project completion, Smart Growth BC compiled the findings and recommendations of its projects into a final report, which is available on the Smart Growth BC website, http://www.smartgrowth.bc.ca

Results Achieved

Based on information in Smart Growth BC’s final report submitted to the MOST Program, the project achieve the following results:

  1. Increased awareness and understanding of sustainable transportation issues and of possible solutions available for addressing sustainable transportation challenges

The projec400s given a fair amount of local exposure. Public information sessions and the charrette event were promoted in local media and Smart Growth BC’s listserv. Over 1,295 people were reached directly and 32,900 people were indirectly reached through the media. The charrette roundtable consisted of 20 people presenting a variety of stakeholders, including the District of Saanich staff, residents, property owners, seniors, youth, and developers. Following the session, charrette participants reported having an increased awareness about sustainable transportation issues and supportive land uses. In addition, Smart Growth BC reported an increase in awareness of innovative solutions. The District of Saanich staff and elected officials were exposed to solutions agreed upon by local residents, property owners, and other stakeholders, such as reduced lane widths and improved facilities for pedestrians and cyclists.

  1. Promotion of partnerships and alliances to achieve sustainable development results

This project facilitated relationship building in the Tillicum community through the charrette process and these relationships continue today. In its final report, Smart Growth BC noted, “By bringing together a variety of stakeholders (including residents, government, property owners, and developers), the process allows these disparate groups to work through competing interests to arrive at common solutions. These solutions addressed transportation issues in the study corridors, and also addressed land use changes that will support more sustainable forms of transportation.”

  1. Extent of adoption and implementation of innovative sustainable transportation activities and tools and practices

The charrette report contained graphic and visual recommendations, such as development concepts for key sites and proposed street sections. In addition, a variety of principles, goals and directives were generated to guide development in the Tillicum Burnside areas including:

  • Improve pedestrian and bicycle circulation and amenities on all street rights of way
  • Implement traffic calming measures (particularly by reducing vehicle traffic lane widths), to discourage speeding and improve the safety and comfort of pedestrian sidewalks and street crossings
  • Utilize zoning and pre-zoning strategies as incentives to owners and developers to invest in building and infrastructure, especially along Gorge Road, Tillicum Road, and Burnside Road
  • Reduce the dominant visual and operational impact of vehicles by discouraging large surface parking areas and encouraging shared and underground parking

In June 2005, the District of Saanich adopted a Streetscape Action Plan, which incorporated many of the charrette report’s recommendations. Currently, the District of Saanich allocated some funding for the first round improvements on Burnside Road. These improvements include: landscape medians, widening of sidewalks, bicycle lanes on both sides of the road and urban furniture. The District of Saanich is currently looking for funding to improve Tillicum Road and the intersection of both roads. Construction with transportation improvements is slated to begin in Spring 2006.

  1. Increased application of sustainable transportation practice

Smart Growth BC has promoted the success of the charrette process in the District of Saanich through its website and its work with other communities. The project is showcased as a model for other communities facing transportation issues. The charrette roundtable has been adopted by other communities and modified to meet their needs. For example, Smart Growth BC reported that a community on Cortez Island was engaging in a charrette in September 2005.

  1. Quantifiable Sustainable Transportation Results

No quantifiable sustainable transportation results are available at this time as the recommendations have not been implemented yet. In its report, Smart Growth BC reported, “When the recommendations of the charrette report are implemented, particularly including an increase in residential and commercial densities, improvements to pedestrian, cycling, and transit facilities, and traffic claming measures, a decrease in greenhouse gas emissions can be expected. Residents and employees in the area will have a much greater range of viable and attractive transportation options for accomplishing their daily trips, including the ability to walk to many more of their daily needs and easy accessibility to transit and cycling”.

Currently, Smart Growth BC’s role has changed with respect to this project. It now serves an advisory role to the District of Saanich when required. Smart Growth BC has no plans or expertise, nor were they expected by the District of Saanich to monitor environmental impact once recommendations were adopted.

Smart Growth BC reported that “the charrette process was tremendously effective in generating a set of design recommendations, including revised street sections, improved pedestrian, cycling, and transit facilities, and supportive land use changes.” This project has demonstrated positive early outcomes. While it is still to early to report on longer-term results, the District of Saanich’s acceptance of an action plan based on the charrette recommendations suggests that this project is headed in the right direction towards creating capacity for Canadians to apply sustainable transportation thinking to their daily lives.


Last updated: Top of Page Important Notices