![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
|
![]() |
![]() |
|
![]()
|
![]() |
3.0 PROGRAM SUCCESS AND IMPACTS 5.0 COST-EFFECTIVENESS OF THE PROGRAM 6.0 CONCLUSIONS AND RECOMMENDATIONS
Background This report provides the results of an evaluation of Transport Canada’s (TC) Moving on Sustainable Transportation (MOST) program. An evaluation of the MOST Program was previously completed for Phase 1 (1999 to 2001). Therefore, the focus of this evaluation is on Phase 2, which refers to the period 2002 to June 2005, when the evaluation began. The MOST Program, originally named the Sustainable Transportation Fund (STF), was launched in 1999 as part of the department’s first Sustainable Development Strategy (SDS), a three-year strategy that outlines challenges and commitments through which TC can better integrate sustainable development into its activities. This program supports projects that produce education, awareness, and analytical tools needed to make sustainable transportation a viable option for Canadians. The program is currently in its second phase with $2.5 million in funding to be allocated over the 2002 – 2007 period. Objectives The primary objectives of the MOST Program are to:
Findings Relevance
Success
Efficiency
Cost-effectiveness
Conclusions and Key Recommendations The Evaluation Team concludes that some positive results are being achieved for the investments made by TC and in this context, TC should continue to manage and administer the MOST Program. Some key recommendations stemming from the evaluation seek to address stakeholder demand for longer-term funding, deal with issues with performance reporting and improve certain aspects of program delivery. a) Demand for longer-term funding
b) Performance Reporting
c) Program Delivery Communication with Stakeholders
Program Exposure
Timeliness in Program Delivery
Resources for Program Delivery
1.0 BACKGROUNDThis report provides the results of an evaluation of Transport Canada’s (TC) Moving on Sustainable Transportation (MOST) program. The Departmental Evaluation Services Branch (DES) undertook the evaluation at the request of the Environmental Affairs Directorate of TC’s Programs Group in order to meet the requirements of Treasury Board Secretariat’s Policy on Transfer Payments, in effect since June 2000. The Policy requires that a program evaluation be performed before renewal of any transfer payment program. Background The MOST Program, originally named the Sustainable Transportation Fund (STF), was launched in 1999 as part of the department’s first Sustainable Development Strategy (SDS), a three-year strategy that outlines challenges and commitments in which TC can better integrate sustainable development into its activities. This program supports projects that produce education, awareness, and analytical tools needed to make sustainable transportation a viable option for Canadians. It was initially established as a three-year program with $1.5 million in resources to be allocated to various projects designed to encourage sustainable transportation practices among the Canadian population. In response to an on-going demand, the program was extended to 2007 and was provided with $2.5 million in additional funding to be allocated over the 2002 – 2007 period. Objectives The primary objectives of the MOST Program are to:
Roles and Responsibilities The administration of the program is the responsibility of the Urban Transportation Programs group of the Environmental Initiatives Branch at TC. There is one full-time program officer and one manager to oversee the general administration of the program, which includes processing application submissions, coordinating the proposal evaluation process, liaising with stakeholders, and managing contribution agreements. To assist them with web management and financial matters, a web officer and a finance officer are borrowed from other branches in the Programs and Divestiture group, when required. There is also a MOST Advisory Committee, which is an independent, multi-stakeholder committee that is responsible for reviewing application submissions and allocating contributions. This committee is chaired by TC and consists of up to 10 members, with representatives from other federal government departments, the not-for-profit sector, municipal governments, industry, and academia. Program Delivery Description There are two funding rounds per fiscal year with application deadlines of June 1 and December 1. Applicants submit a project proposal that must meet certain eligibility criteria (see Annex 1). Once applications are submitted, the MOST Program Officer conducts a mandatory criteria evaluation, which serves as a pre-screening process to ensure that each application does indeed meet the eligibility criteria. The MOST Program Officer then distributes all of the proposals to the MOST Advisory Committee members. The MOST Advisory Committee members have a minimum of three weeks to review proposals against the evaluation criteria (see Annex 2). In the Committee’s overall selection, consideration is given not only to the evaluation criteria but also to the regional balance of projects, the variety of initiatives, as well as, a balance among projects that are innovative and those which build on existing initiatives. The Committee uses consensus to make recommendations for funding. Final decision-making authority rests with the Director General of Environmental Affairs. Applicants are officially notified of funding decisions by email approximately three months after the closing date of each application deadline. As a condition of the notification, successful applicants are not allowed to publicly disclose that they have been allocated funding from the MOST Program until the ministerial announcement. However, during this time, the contribution agreements can be signed and funding recipients may commence their projects. Funding recipients are required to submit two reports for multi-year projects: a mid-term report and a final report. For projects that are one year or less, only a final report is required. Funding recipients are also required to account for all project expenditures and revenues. Once the final reports are submitted, the MOST Program management assess the final reports and process invoices for payments. Reimbursement from the MOST Program is based on actual project expenditures and actual leveraged funding from sources other than the program. MOST Program Contributions Maximum TC funding per individual project is $100,000 over two years with the contribution not exceeding 50 percent of eligible costs. The remaining 50 percent of project costs must be obtained from other sources. Approximately $350,000 may be disbursed per round but the actual disbursement depends on the quality of the projects. Table 1 on the next page displays the actual amount awarded in each round since the launch of the MOST Program. Table 1: Amount of MOST Funding Awarded by Round
Note: “+” indicates additional money allocated but not spent Table 2 shows the logic model for the MOST Program. The logic model was originally created as part of a Results-based Management and Accountability Framework (RMAF) in 2001, at the time of the last program renewal. Table 2: MOST Logic Model
The study was conducted to assess the program’s relevance, success, efficiency, and whether there are alternative, more cost-effective ways of delivering the program in order to provide input for future decision-making. An evaluation of the MOST Program was previously completed for Phase 1 (1999 to 2001). Therefore, the focus of this evaluation is on Phase 2, which refers to the period 2002 to June 2005, when the evaluation began. More specifically, the Evaluation Team assessed the results achieved by MOST-funded projects that were funded between Rounds 5 and 9. The projects in Round 10 and Round 11 had not yet begun when the evaluation started. The evaluation study addressed the following questions: A. Program Relevance
B. Program Success and Impacts
C. Efficiency
D. Cost-effectiveness
1.6 Methodology and Data Sources The Evaluation Team developed an evaluation strategy and selected various methods for collecting the data needed to address the evaluation questions. See Annex 3 for the evaluation framework. Methodology File/Document review The Evaluation Team reviewed relevant websites and various documents from the program files including application submissions, final project reports, final report assessments, and MOST Advisory Committee Records of Decisions. A list of key references can be found in Annex 4. Questionnaire In July 2005, questionnaires were administered by direct mail to successful and unsuccessful applicants of the MOST Program. (See Annex 5 and 6 for the questionnaires.) Follow-up packages were sent one month later to non-respondents. The primary purpose of the questionnaires was to obtain stakeholder views about the program. The response rate for the unsuccessful applicants’ questionnaire was 31% with a sample size of 22. The response rate for the successful applicants’ questionnaire was 53% with a sample size of 25. Interviews The Evaluation Team conducted face-to-face interviews with the MOST Program management and the chair of the MOST Advisory Committee in order to obtain their views about the MOST Program. In addition, the Evaluation Team also conducted telephone interviews with three funding recipients as part of the Case Study method (see below). The list of interview questions can be found in Annex 7. Case Studies Case studies were conducted to gain insight about the success of the MOST Program. Given that MOST-funded projects vary widely in scope, case study methodology provides detailed analyses of specific projects. The Evaluation Team used the following criteria to select three case studies:
The three case studies chosen are Bathurst Sustainable Development, Pollution Probe and Smart Growth BC. These case studies are presented in Annex 8. Limitations The evaluation relied primarily on performance information provided by funding recipients in their final project reports to evaluate the success of the program. While these documents provided some useful information at the project level, the reliability and validity of the data varied from project to project. As a result, it was difficult to compile results and draw conclusions at the program level. The case study approach was adopted in order to address this limitation.
The MOST Program advances TC’s third strategic outcome of an environmentally responsible transportation system that contributes to Canada’s sustainable development objectives. In TC’s 2005 – 2006 Report on Plans and Priorities, TC commits to continue developing and implementing programs to achieve a more sustainable transportation system. TC recognizes that sustainable transportation is a shared responsibility and commits to working with partners and stakeholders, such as the general public, non-governmental organizations (NGOs), and industry. The MOST Program is an example of a program that contributes to an environmentally responsible transportation system. The MOST Program is aligned with current government priorities for the environment as outlined in the Speech from the Throne. The design of the MOST Program promotes partnerships in sustainable transportation. This is consistent with the Government of Canada’s commitment to work with partners to build sustainable development systematically. The focus of the MOST Program on innovative projects is consistent with the Government of Canada’s recognition that “human ingenuity will turn increasingly to ways to produce and use energy more cleanly and efficiently; to eliminate toxins from our air, water, and soil, and to build more sustainable communities.” Furthermore, some of the projects funded by the MOST Program have contributed to the Government’s commitment to the Kyoto Accord on climate change. TC’s 2004 – 2006 SDS is the third successive three-year strategy developed by the department. It outlines seven challenges and 32 specific commitments in which the department can do better to integrate sustainable development into its activities. The seven challenges are:
The MOST Program is identified as one of the commitments to address the first challenge of encouraging Canadians to make more sustainable transportation choices. This is to be done by creating awareness and educating Canadians on the issues, benefits, and trade-offs, as well as practices and choices that individuals can adopt to reduce the adverse impacts of transportation on the environment. It involves partnerships with other federal departments, other levels of government, industry, NGOs, and other stakeholders. Changing behaviour is an essential part of the response to the environmental impact of transportation. All segments of society need to understand the impacts of their transportation behaviour in order to make choices that reduce the adverse effects of transportation on the environment. Although the MOST Program was identified as a specific commitment to address the first challenge in TC’s SDS, the outcomes of the program also contribute to some of the other SDS challenges. For example, the MOST Program’s outcome of “Development and testing of innovative tools, approaches and practices in support of sustainable transportation” contributes to the second SDS challenge of “Enhancing innovation and skills development”.
There is an on-going demand for the MOST Program from stakeholders. Demand for MOST funding as indicated by the number of requests for funding continues to remain strong. Table 3 displays the number of applications submitted in each round. The average number of applications in Phase 1 is 24 and the average number of applications in Phase 2 is 25. This suggests that the number of requests for funding has generally remained consistent. Table 3: Number of Applications Submitted to the MOST Program
The demand for the MOST Program is expected to continue. According to questionnaire responses, an overwhelming majority of successful applicants said that they would re-apply for MOST funding (92%). Similarly, a majority of unsuccessful applicants said that they would re apply (65%). Through their contact with stakeholders, MOST Program management reported that there is a demand for more stable and longer-term funding for on-going projects. The Evaluation Team made the same observation based on the questionnaire responses. At present, the MOST Program provides funding to short-term, innovative projects. Due to the terms and conditions of the program regarding project length, on going projects cannot be funded. According to MOST Program management, the kinds of projects being submitted for funding have changed over time. There is a shift away from awareness-building type projects to projects aimed at changing behaviour. MOST Program management described this shift as the “natural evolution of a program” where early years are dedicated to building awareness. MOST Program management believe there is now an increased level of understanding among the Canadian public on the topic of sustainable transportation. In this way, the MOST Program management feel that the need for the program has changed. There is less need for projects aimed at educating Canadians about sustainable transportation and more of a need for projects aimed at encouraging Canadians to adopt sustainable modes of transportation.
There are a number of other partners typically involved in MOST-funded projects besides TC, including other federal government departments, other levels of government (i.e. local, regional, and provincial), businesses, not-for-profit organizations, foundations and associations, research labs, and academic institutions. Even though there are a number of other partners involved, TC remains a very important player in funding sustainable transportation projects. MOST is the only Canadian federal contribution program that is dedicated to providing funding for innovative sustainable transportation projects. Other environmental programs at TC do not have the same direct objective. There are programs at other federal departments that may provide some funding to sustainable transportation projects, such as Environment Canada’s Climate Change Action Fund, One Tonne Challenge and EcoAction or Natural Resources Canada’s Fleetsmart Program. However, these programs address the broader issue of climate change or general environmental issues rather than focusing solely on sustainable transportation projects. Some provincial or municipal governments, industry associations, and private foundations may also provide funding to sustainable transportation projects. However, these programs also address broader objectives which are not necessarily focused on sustainable transportation. In general, contributions from other sources of funding are less than the contributions from the MOST Program. Compared to other sources of funding, the MOST Program provides funding up to 50% of total project costs. Without the federal contribution, smaller organizations would have to find additional sources of funding to make up for the contributions that could have been provided through the MOST Program. Table 4 suggests that the MOST Program was a primary funding source and did contribute to the achievement of funded projects’ stated objectives. This role of the MOST Program is considered to be very important by successful applicants. In open-ended responses of the questionnaire, the successful applicants speculated that if MOST funding was not available, their projects would not have occurred, the projects would have resulted in delays, or the projects would have been forced to scale back to a more limited focus. Table 4: Rating of the Role of the MOST Program by Successful Applicants
The speculations of the successful applicants are substantiated by the responses from the unsuccessful applicants, in which the majority said that without MOST funding, their projects did not occur. In some cases, unsuccessful applicants were able to find other sources of funding but the focus of the original project changed and they were forced to launch a smaller-scale version of the original project. Additionally, MOST funding has been used by some not-for-profit organizations to leverage funding from other partners because funding from the federal government is seen as a stable source of funding. This funding arrangement has been vital for smaller organizations, which already have limited resources. It also helps to promote partnerships between other organizations and governments. MOST stakeholders also believe TC should take a leadership role in sustainable transportation because transportation activities have a wide range of impacts on the environment, including resource use (materials and energy), undesirable residuals (emissions, spills and leaks), and land use, including impacts on wildlife. In particular, the transportation sector has a significant impact on climate change as it is the single largest contributor to greenhouse gas emissions. One stakeholder remarked as follows:
3.0 PROGRAM SUCCESS AND IMPACTS
During Phase 2, the MOST Program provided a contribution to 46 projects across the country. The majority of projects funded in Phase 2 originated from British Columbia, Ontario, and Quebec, accounting for 78% of all projects funded. The remaining 22% of projects were initiated in Alberta, New Brunswick, Nova Scotia, and Saskatchewan. Sustainable transportation is a broad topic and the MOST Program provides contributions for projects that address a wide range of sustainable transportation issues. Given the diversity of MOST-funded projects, three case studies were selected for this evaluation as examples (see Table 5). For more information on these case studies, see Annex 8. Table 5: Project Descriptions of Case Studies
More details on all others projects that have received contributions from the MOST Program can be found on the program’s website, http://www.tc.gc.ca/programs/environment/most/menu.htm
It is difficult to determine the extent to which the MOST Program, as a whole, realizes quantifiable results on TC’s sustainable development priorities. MOST-funded projects vary widely in scope and each project tends to measure and report on different performance indicators and targets. As well, the methods for collecting this performance information are inconsistent. Therefore, the quantitative performance data across all projects is not always comparable, making it difficult to compile all the data and draw conclusions about the kinds of quantifiable results produced by the MOST Program. MOST Program Managers have indicated that they prefer measures such as, “gas saved”, “kilometres travelled saved”, and “modal shifts before and after a project” to describe quantitative environmental or sustainable transportation results. Of the three case studies, Pollution Probe’s S M-A-R-T Movement Program was the only project to produce this kind of information. In the first phase of its project, Pollution Probe could only conduct follow-up surveys with two out of the five participating companies within the timeframe it had to report to the MOST Program. The results in Table 6 reflect early results on the environmental impact of this project. Table 6: S-M-A-R-T Follow-up Employee Transportation Survey Results (2003)
Company A experienced a decrease in single-occupant-vehicle (SOV) usage and a decrease in vehicle kilometres travelled. Consequently, this led to a decrease in air pollutants emitted and energy intensity. Similarly, Company B experienced a slight decrease in SOV usage. However, it also experienced a decrease in transit usage. Consequently, this led to an increase in vehicle kilometres travelled and average air pollutants emitted. But, overall energy intensity had decreased. The provision of these types of quantitative environmental results is not always feasible due to the focus or scope of some projects. For example, in the case of Smart Growth BC’s project, the charrette produced a report with recommendations for changes to make the Tillicum Burnside community more sustainable transportation-friendly. The charrette report itself could not directly lead to environmental or sustainable transportation results. Rather, the recommendations of the charrette would have to have been implemented in order to assess the environmental impact. Smart Growth BC could not be expected to deliver on providing quantitative environmental or sustainable transportation results. Furthermore, the environmental results may not be visible except in the long-term, where the attribution between the charrette report and environmental results would be somewhat weak. Similarly, Bathurst Sustainable Development (BSD) could not be expected to provide quantitative environmental results because the project was a study to examine the feasibility of implementing an urban transit system. The actual implementation of the bus system itself had not taken place when BSD submitted its final report to the MOST Program. Despite this limitation, BSD did provide some estimates about the environmental impact if a transit system were to be implemented (see Annex 8 for more details). An examination of the three case studies suggests that some projects can realize direct quantitative environmental or sustainable transportation results, but the objectives of others do not intend to lead to quantitative results.
The extent to which the MOST program provides Canadians with practical information and tools for better applying sustainable transportation thinking to their lives can be demonstrated, in part, by the number of projects that are still on-going. Table 7 displays the total number of MOST funded projects in each round and the number of projects that have continued after MOST funding had ended. More than half of the projects in all of the rounds are still on-going after MOST funded ended. It is interesting to note that all of the projects in Round 7 are still on-going. Table 7: Number of Projects That Have Continued After Funding Ended
Note: “+” indicates the number of additional projects that are possibly on-going However, this indicator only reveals part of the story since some MOST-funded projects are not intended to continue after MOST funding. All projects that receive contributions from the MOST Program must report on the following:
In general, all of the recipients reported that their projects received exposure in varying ways and that there was increased awareness among the target audience of the project itself as well as increased awareness of possible solutions available for addressing sustainable transportation challenges. The extent of adoption and implementation of innovative sustainable transportation activities, tools, and practices also varied depending on the project scope. Beyond these general observations, it is difficult to aggregate the results in a meaningful way to determine the extent to which the MOST Program, as a whole, provides Canadians in different communities with practical information and tools for better applying sustainable transportation thinking to their lives. It is easier to observe the extent to which activities, tools, and practices have been implemented at the individual project level. This was examined using our case study approach. A summary of how each project meets this program objective is provided in this section. For more details, see Annex 8. Case Study #1: Urban Transportation Project: Addressing Climate Change in the city of Bathurst, New Brunswick – Feasibility Study (Bathurst Sustainable Development) The study concluded there was sufficient ridership demand for a bus service, which has the potential of being financially sustainable within two to three years. BSD estimated that the bus system has the potential to significantly reduce greenhouse gas emissions. In a follow-up interview, the Evaluation Team learned that BSD is currently assisting the City of Bathurst with a pilot project to test a bus system in the city. Early results reveal a steady increase in ridership levels. Case Study #2: S-M-A-R-T Movement Program (Pollution Probe) In the first phase of its project, Pollution Probe recruited five pilot organizations to participate in the S-M-A-R-T Movement Program. In the second phase, they recruited another seven for a total of twelve workplaces. Results of follow-up surveys were conducted and submitted to the MOST Program with participating companies where the program had been operational long enough. These early results revealed that participants generally became more aware about the impact of transportation on the environment and the benefits of sustainable transportation. As shown in Table 6, there was some behavioural change in Company A, which resulted in having a positive impact on the environment, while there was less behavioural change in Company B. In working with twelve companies, Pollution Probe was not progressing as far along as they would have liked. In its experience, sustaining this type of voluntary project has been challenging for Pollution Probe. In order to be able to implement the employer program, Pollution Probe noted the need to get senior management on board by highlighting the economic benefits of a program. Through its efforts, Pollution Probe has tried to engage senior management of participating companies and has achieved limited success. Case Study #3: Tillicum Burnside Urban Village Community Roundtable (Smart Growth B.C.) The project produced a charrette report, which contained recommendations to improve the Tillicum Burnside community including:
In a follow-up interview with Smart Growth BC, the Evaluation Team learned that the District of Saanich adopted a Streetscape Action Plan in June 2005, which incorporated many of the charrette report’s recommendations. The District of Saanich has allocated some funding for the first round improvements on Burnside Road and construction is slated to begin in Spring 2006. The results of the case studies suggest that the MOST Program is experiencing some success in achieving outcomes consistent with this program objective. However, it is difficult to assess overall success because it is still too early to report on longer-term results for most of these projects.
There were three key recommendations in the 2001 evaluation of the MOST Program. The recommendations were offered to assist the program management in expanding program exposure, obtaining better and more timely information on client needs, and implementing more effective results measurement and reporting. The first recommendation was to enhance the feedback and consultative mechanisms of the MOST Program by doing the following:
Table 8 summarizes the MOST management action plan in response to this recommendation and the impacts of those actions on the program. Table 8: Summary and Impact of Management Action for Recommendation #1
The MOST Program management carried out most of the actions as planned in response to the first recommendation. Although the actions taken to gather stakeholder feedback are not having the desired impact, they are not having a negative impact on the program. The MOST Program management should consider alternate ways to reach target groups and solicit feedback. One suggestion would be to adapt the questionnaires used as part of this evaluation (see Annex 6 and 7). The second key recommendation in the 2001 evaluation of the MOST Program was to promote MOST success stories by:
Table 9 summarizes the MOST Program management’s response to this second recommendation and the impact of those actions. Table 9: Summary and Impact of Management Action for Recommendation #2
The first action described in Table 9 was carried out by management and appears to be having the desired impact. However, the second action is not achieving its full impact. This is partly due to the lack of regular website updates. Given the difficulties in regularly maintaining the website, MOST Program management may wish to consider alternate ways to promote success stories. At the same time, the MOST Program management may wish to consider alternative ways to expand program exposure in order to achieve a regional balance of project submissions. The third key recommendation in the last evaluation was to promote the use of the results-based approach to program management by:
Table 10 presents the summary and impact of the management action in response to this recommendation. Table 10: Summary and Impact of Management Action for Recommendation #3
Of the two actions in Table 10, the most problematic change has been the requirement for recipients to report on performance indicators. The “Performance Indicators – Project Planning Table” document has resulted in much confusion by applicants when they are completing their application forms. The program does offer assistance to help clarify the concepts for applicants but this involves considerable time and effort of the MOST Program Officer. In addition, the format of this document leads the applicant to develop performance indicators that reflect activities and outputs, rather than outcomes. The applicant may not be aware of this error due to lack of understanding of results-based management concepts. As a result, recipients, who complete these forms as part of their reporting requirements are not providing accurate performance information. Furthermore, the performance information that has been collected is not very useful to describe program level performance. 4.0 EFFICIENCY
Using the questionnaire method, both successful and unsuccessful MOST applicants were asked to provide feedback on various aspects of program delivery. Overall, the MOST Program received average to above average scores on a 1 to 5 agreement and importance scale for nine program delivery aspects. This suggests that these stakeholders are generally satisfied or very satisfied with the delivery of the MOST Program. Table 11 displays a summary of the ratings from the questionnaires for successful applicants. Table 11: Rating of Program Delivery by Successful Applicants
Note: n = 25 On average, successful applicants highly rated the usefulness of the MOST website (4.7), especially in order to obtain the proper information to complete application requirements (4.8). Additionally, there was a high rating given to the ease of completing mid-term reporting requirements, where applicable (4.6). Although timeliness of the proposal evaluation received the lowest rating, the satisfaction level is still slightly above average (3.8). These findings suggest that the MOST Program is performing to the satisfaction of successful applicants. Table 12 displays a summary of the ratings from the questionnaires for the unsuccessful applicants. Table 12: Rating of Program Delivery by Unsuccessful Applicants
Note: n = 22 On average, the unsuccessful applicants highly rated the timeliness of MOST Program Staff responses to inquiries (4.8) and the usefulness of the information provided by MOST Program Staff (4.7). The program delivery aspects that received the lowest ratings were the timeliness of the proposal evaluation process (3.0) and the clarity and transparency of the proposal evaluation process (2.9). These two aspects also received above average ratings of importance, which results in a large gap. This gap suggests an area for improvement in the MOST Program. Some comments about the proposal evaluation process by unsuccessful applicants are as follows:
Currently, the MOST Program clearly indicates on the website that the expected timeframe for decisions on proposal evaluation is approximately three months after the application submission deadline. According to MOST Program management, the proposal evaluation process has generally been completed within this timeframe throughout Phase 2, with the exception of Round 10, where there was an unusual delay. Although the MOST Program has demonstrated consistent timeliness in carrying out the proposal evaluation process, the comments above suggest that some applicants feel three months is not a reasonable timeframe. Comments by unsuccessful applicants about the transparency of the proposal evaluation process are as follows:
The MOST Program management already has a mechanism in place to address the issue of transparency. After receiving notification of decision by email, applicants can telephone the Program Management and request a debrief about the proposal evaluation process. Aside from the above aspects of program delivery, stakeholders also identified other areas of inefficiency through open-ended responses of the questionnaires. These are as follows: the process for disbursing funds, the application and reporting requirements, the timeframe between funding notification and ministerial announcement, and website management. a) The Process for Disbursing Funds Stakeholders expressed some concern that the process for disbursing funds is not done in a timely manner. This is considered to be problematic for not-for-profit organizations, which already have limited resources. Stakeholders would like to see improved turnaround time for signing contribution agreements, processing invoices, and disbursing funds. Below are some responses from successful applicants outlining their concerns.
MOST Program management is aware of this concern. At present, MOST Program Managers rely on a finance officer borrowed from another group to facilitate this process. Without additional resources, management cannot speed up the administrative process leading up to the disbursement of funds. To address the issue of financial hardship among funding recipients under the current arrangement, management should consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones. Also, MOST Program management noted that the Finance and Administration Directorate is responsible for actual release of funds. While the Program Management tries to facilitate the process as much as possible, timing of actual disbursement is beyond their control. The MOST Program management should enter into discussions with the Finance and Administrative group to re-examine this process and assess options to improve processing time. b) Application and Reporting Requirements Stakeholders noted that application and reporting requirements are onerous, particularly for not-for-profit organizations with limited resources. They are concerned that much time and resources are taken away from the project itself in order to complete applications or collect information for reporting purposes. As some stakeholders noted:
MOST Program management should consider ways to simplify the application process and streamline the reporting strategy. The level of reporting should be commensurate with the level of funding allocated and take into consideration the scope and complexity of the project. c) Delays between notification and announcement Stakeholders raised some concern about the timeframe between funding notification and ministerial announcement. As one stakeholder noted:
MOST Program management are aware of the lag time between funding notification and ministerial announcement. However, this process is also beyond their control as the timing of the ministerial announcement is coordinated between TC’s Communications Group and the Minister’s Office. Management should advise the Communications Group that the timeliness of ministerial announcement is important. d) Website Management The website serves as an important tool not just for completing applications but also to learn about what others are doing in the area of sustainable transportation. Several applicants have raised the following concerns:
Due to the technical nature of website management and lack of expertise of current MOST Program staff to maintain the website, MOST Program management borrows a web officer from another group to assist them in this area. Given the importance of the website as a communication tool to stakeholders, the MOST Program management may require additional resources to regularly update the website, in particular to post final project reports in a timely manner. 5.0 COST-EFFECTIVENESS OF THE PROGRAM
A full cost-effectiveness analysis could not be done because the Evaluation Team experienced difficulties in selecting a single measure of effectiveness. This is due to limitations with performance reporting by funding recipients as well as the fact that outcomes varied from project to project. As an alternative, the Evaluation Team compared the costs and projected outcomes to provide some indication of the cost-effectiveness of the MOST Program. Table 13 shows an estimate of the relevant costs in the MOST Program. Even when including resources covered by other budgets, MOST is a low cost program. Table 13: Annual Cost Estimates for the MOST Program
1 Costs covered under the MOST Program’s budget
Table 14 displays a comparison of the annual costs and projected outcomes using logic and qualitative information. As noted in Question 3, MOST Program is the only Canadian federal contribution program that is dedicated to providing significant contributions for sustainable transportation projects. Since there are no other similar programs in terms of activities and outcomes, the only comparison that could be made is the scenario of providing no funding. Table 14: Comparison of MOST Program Costs and Projected Outcomes
Table 14 suggests that a case can be tentatively made that the MOST Program does provide value for money. The department is achieving some positive outcomes that contribute to an environmentally responsible transportation system at a low cost. 6.0 CONCLUSIONS AND RECOMMENDATIONSThe Evaluation Team concludes that some positive results are being achieved for the investments made by TC and in this context, TC should continue to manage and administer the MOST Program. It is a relevant program for TC to be involved in. The program is successfully meeting its stated objectives at the project level and is generally being delivered by TC in an efficient manner. This program is a cost-effective way for TC to achieve sustainable transportation outcomes. The section below presents the key recommendations stemming from the evaluation. Demand for longer-term funding There is a consistent demand for longer-term funding for sustainable transportation projects by its stakeholders. Through the SDS consultations, responses to questionnaires, and discussions with MOST Program management, stakeholders have requested for this type of program expansion. At present, the terms and conditions of the program outline that funding is dedicated for innovative projects up to a maximum of two years. On-going projects cannot be funded. Therefore, many of the MOST-funded projects are short-term and are accountable to the program for a short timeframe. For several of these projects, not enough time would have passed in order for the project to demonstrate any impact on the environment.
Performance Reporting There are a few challenges with project performance reporting that made it difficult to evaluate the extent to which the MOST Program was successful in meeting its stated objectives at the program level as well as conduct a full cost-effectiveness analysis.
Although there are five project categories outlined in the program’s eligibility criteria (see Annex 1), the majority of MOST-funded projects fall under more than one project category. Since MOST-funded projects vary widely in scope and cannot be grouped into categories, it was difficult to aggregate performance data by project type in order to draw meaningful comparisons at a program level. If distinct project categories are constructed, this could facilitate the roll up of performance data to describe program level results.
Stakeholders have raised some concerns that the reporting strategy is onerous and time consuming. While some stakeholders recognize the value of monitoring performance in demonstrating the progress of their projects, they believe that the current reporting strategy requires too much of their time and resources.
The reliability and validity of the data provided to the MOST Program may be questioned as MOST funding recipients are responsible for collecting and reporting performance information for their projects. The reliability of the data is a concern as some MOST funding recipients, especially not-for-profit organizations, do not have the necessary financial and human resources to collect quality performance data. As well, there is no consistency with regards to the type of performance data that is collected. Each project tends to measure and report on different performance indicators and targets. Furthermore, the methods used to collect data tend to vary, which makes it difficult to compare across projects and aggregate results. The validity of the data is a concern because much of the performance data that tends to be reported reflects performance indicators for activities and outputs, not environmental or sustainable transportation results. This is understandable given that data for activities and outputs are easier to collect and are more readily available within the short timeframe of a project’s life. Lack of understanding of performance measurement and results-based management concepts by funding recipients may also be an issue. These limitations make it particularly difficult for the MOST Program to “realize quantifiable environmental and sustainable development results on TC’s sustainable development priorities” – even at the individual project level.
Program Delivery There are a few areas of program delivery in which the MOST Program management can improve its performance, including communication with stakeholders, program exposure, resource availability and timeliness of program delivery.
The 2001 evaluation recommended that the MOST Program management needed to obtain better and more timely information on client needs. Although the MOST Program attempted to do so by using SDS consultations, conducting telephone follow-up surveys and website surveys to gather feedback, these changes had almost no impact on the program.
Since the last evaluation, MOST Program management attempted to expand program exposure in order to achieve a better regional balance of project submissions and to promote MOST success stories using the website. Although the actions taken to reach underrepresented regions are not having a negative impact, a regional imbalance of application submissions persists. The website is an effective way to promote the MOST Program but management encounters difficulties in maintaining the website on a regular basis.
Stakeholders raised some concerns about the timeliness of three program delivery areas: proposal evaluation, disbursement of funds, and ministerial announcements of successful projects. The program’s website clearly states that the expected timeframe for decisions on proposal evaluations is approximately three months after the application submission deadline. The MOST Program should ensure that these timelines are adhered to or revise the standard if it is unrealistic. In cases when delays may occur, the MOST Program management should openly communicate to all applicants explaining the reasons for the delay. The timeliness in disbursement of funds is important to funding recipients, particularly for not for-profit organizations with limited resources. To improve this process, the MOST Program management may consider adopting a process that involves disbursing funds in increments throughout a project’s life by tying the release of funds to certain project milestones. The reasons for delays in disbursement of funds as well as the ministerial announcements of successful projects are also often beyond the control of the MOST Program management. TC’s Communications Group is responsible for coordinating the ministerial announcement of projects while TC’s Finance and Administration Directorate is responsible for disbursement of funds. MOST Program management should enter into discussions with these TC groups to re-examine the processes associated with these areas and assess options to improve processing time.
TC staff devoted to the MOST Program consists of one full-time program officer and one manager, who also manages other environmental programs. To assist with web management and financial matters, a web officer and a finance officer are borrowed from other branches in TC’s Programs Group. The level of staff time devoted to the MOST Program appears to be insufficient for efficient and timely program delivery. Given that the finance officer and web officer must focus on their primary duties first, the MOST Program management may not be able to improve the processing time leading up to the disbursement of funds or regularly maintain the program’s website.
7.0 ANNEXESAnnex 1: MOST Program Eligibility Criteria The following five criteria must be met for projects to be eligible for program funding:
Proposals must target the Canadian public including, but not limited to, the general public, the transportation sector, youth, municipalities, First Nations and Aboriginal Peoples, and educators. The proposal must obtain a minimum of 50 percent of resources (cash and/or in-kind* from sources other than the Government of Canada, so that the federal contribution does not exceed 50 percent (cash and/or in-kind). *Note: Proposals with a combination of in-kind and cash support will be viewed more favourably than those with only in-kind support. Proposals must contain quantifiable sustainable transportation targets and performance indicators to measure and report on the environmental and sustainable development impacts expected as a result of the project. Each proposed target must have an associated performance indicator. Qualitative targets and performance indicators can complement the quantifiable ones. Proposals must contain a detailed communication plan for the dissemination of any project-related materials or results that will serve to further the program’s objective of providing Canadians with practical information and tools for better applying sustainable transportation thinking to their daily lives. The communication plan must include the following details: Annex 2: MOST Program Evaluation Criteria If a project meets the mandatory eligibility criteria, the following evaluation criteria is used to assess the relative strength of each proposal.
Other criteria In its overall selection, the MOST Advisory Committee will give consideration to the following (in addition to the above-described criteria) to ensure an assortment of initiatives are funded which produce results supportive of program objectives:
Annex 4: List of Key References Bathurst Sustainable Development Website Smart Growth BC Website Pollution Probe Website Moving on Sustainable Transportation Website
TC’s Sustainable Development Strategy 2004 – 2006
TC’s Report on Plans and Priorities 2005 – 2006 Speech from the Throne, October 5, 2004 Consultation Report: Toward Sustainable Transportation – Transport Canada’s Third Sustainable Development Strategy MOST Evaluation Report (2001) MOST Management Action Plan (2001) Annex 5: Questionnaire for “Successful” MOST Applicants MOST Invites Your Feedback!
Thank you for completing this survey!All information will be kept confidential and protected by the Access to Information Act and the Privacy Act. Annex 6: Questionnaire for “Unsuccessful” MOST Applicants MOST Invites Your Feedback!1. How did you first learn about the MOST program?
2. Please circle the number to show to what extent you are in agreement with each statement and then, what level of importance this aspect of the MOST program has for you.
Thank you for completing this survey!All information will be kept confidential and protected by the Access to Information Act and the Privacy Act. Interview Questions for MOST Program Managers
Interview Questions for Chair of the MOST Advisory Committee
Interview Questions for Case Study Stakeholders
Three case studies were conducted for this evaluation. The primary objective of the case study approach was to gain insight into the longer-term results of selected MOST-funded projects. The case study approach was also be used to gather information on the following specific evaluation questions, which focuses on the behavioural change and outcome-focused objectives of the program:
Case Study #1: The Urban Transportation Project (Bathurst Sustainable Development) Background In Round 6, the MOST Program awarded $30,000 to Bathurst Sustainable Development (BSD) for a project called, “The Urban Transportation Project: Addressing Climate Change in the City of Bathurst, New Brunswick – Feasibility Study”. The Bathurst City Council had decided to implement an urban transit system “in order to assist citizens with improving their quality of lives and to provide them with a transportation option other than their personal automobiles as a means of reducing their greenhouse gas emissions.” The Council asked BSD to conduct a study to examine the feasibility of introducing a bus service in the City of Bathurst. The objective of the study was to identify challenges and potential opportunities for implementing an urban transit system in the City of Bathurst, which would include an assessment of citizen needs as well as research into the financial and operational aspects of starting a bus system. In addition to MOST, BSD had partnerships with the City of Bathurst, the Federation of Canadian Municipalities, SMT-Acadia Bus Lines, Dupon Trolley Industries, Codiac Transit, and Natural Resources Canada. While there are a number of other partners involved, MOST funding was described by BSD as “critical” to the project. “Having Transport Canada as a partner is critical to bringing credibility to the project and encourages the participation of others.” The feasibility study was completed on April 30, 2004. BSD conducted the following activities as part of the study.
In addition to submitting a final report to the MOST Program upon project completion, BSD compiled the findings of its study into a final study report, which is available on its website, http://www.bathurstsustainabledevelopment.com. Results Achieved
The BSD’s final report suggested that the implementation of the transit bus service would allow individual citizens who use the transit service daily, instead of driving their personal automobiles, to reach 50-70% of their One Tonne Challenge goal of reducing their personal greenhouse gas emissions by one tonne annually. BSD launched a pilot test of an urban transit system in May 2005. Although it is still too early to measure the environmental impact, performance information is being collected, such as ridership counts shown below. Urban Transit Test Project Rider Counts (2005)
*Free transit was provided In general, BSD has achieved positive early outcomes with this project. Since MOST funding ended, BSD was able to continue its efforts based on the feasibility study and is well on its way to providing Canadians in the City of Bathurst with the capacity to adopt more sustainable modes of transportation. Case Study #2: S-M-A-R-T Movement Program (Pollution Probe) Background The MOST Program awarded $30,000 in Round 5 and $52,900 in Round 8 to Pollution Probe to pilot a two-phased project called, “S-M-A-R-T Movement”. S-M-A-R-T Movement is a workplace-based trip reduction program that is directed at reducing employee single-occupant vehicle (SOV) trips. It is both an information resource and support service to help guide medium to large organizations that want to reduce employee car trips. S M A-R-T Movement aims to demonstrate to employers and employees that reducing SOV trips and vehicle kilometres travelled will save money and time, reduce stress and absenteeism, increase productivity, and address a number of broader issues such as traffic congestion, air quality, climate change, and other problems associated with urban sprawl. On a broader scale, S-M-A-R-T Movement aimed not only to achieve behavioural change amongst its participants, but also to strengthen support for the emerging approach of transportation demand management (TDM) in order to further its adoption throughout Canada. In the first phase, the trip reduction program was initially directed at a selection of five pilot organizations in the Greater Toronto Area for the first year and 20 organizations in the Greater Toronto Area by 2005, with the intention of delivering the program across the country in other large urban centres in Canada. In the second phase, MOST funds were directed to the on-going support of the program. According to stakeholder interviews, MOST Program was absolutely “critical” to the success of this project. Compared to other federal departments, TC is the only one with federal resources. In addition to MOST, other funding and service partners included the following:
Pollution Probe submitted a final project report to MOST in June 2005 and the S M A R T Movement Program is still on-going. In both phases, Pollution Probe engaged in the following activities to launch the project:
Results Achieved
In the first phase, Pollution Probe recruited 5 pilot organizations to participate in the S-M-A-R-T Movement Program. In the second phase, they recruited another 7 for a total of 12 workplaces. Pollution Probe continues to engage a variety of project partners in S-M-A-R-T Movement. It was found that working in partnerships where possible allows for more efficient use of resources and builds on each the strengths of others. Organizational partnerships have been maintained with Clean Air Champions, Teletrips, the Town of Markham, Carpooltool.com, City of Toronto Bicycle Promotions and the BUG Network. These partnerships involved co-promotion through materials and events to S-M-A-R-T Movement participating workplaces as well as strategic discussions and information sharing about each other’s projects where appropriate. Pollution Probe’s S-M-A-R-T Movement has also partnered with a number of organizations on new projects that are mutually beneficial. In particular, Pollution Probe has made great strides in the further development of partnerships with the Greater Toronto Area and Hamilton Smart Commute Initiative.
In the first phase, follow-up surveys were conducted with participating companies. According to survey results, 68% of respondents felt they were more or significantly more aware of the benefits of sustainable transportation, transportation issues linked to smog/air quality, and transportation impacts linked to climate change and more or significantly more concerned with the health impacts/benefits of sustainable transportation. As well, there was an increase in employee knowledge of the S-M-A-R-T Movement program. Forty-six percent of respondents in one company and 82% of respondents in another company reported knowing about the program. In the second phase, a follow-up survey with one company revealed the following results:
These results suggest that the S-M-A-R-T Movement program led to increased awareness and understanding of sustainable transportation issues by its participants as well as possible solutions available for addressing sustainable transportation challenges.
The following are the results of the two workplaces where the program had been operational long enough to warrant a follow-up survey conducted in Phase 1. S-M-A-R-T Follow-up Employee Transportation Survey Results (2003)
The results in the above table reveal that Company A experienced a decrease in single-occupant-vehicle (SOV) usage and a decrease in vehicle kilometres travelled. Consequently, this led to a decrease in air pollutants emitted and energy intensity. Similarly, Company B also experienced a slight decrease in SOV usage. However, it also experienced a decrease in transit usage. Consequently, this led to an increase in vehicle kilometres and average air pollutants emitted. But, overall energy intensity had decreased.
In working with 12 companies, Pollution Probe was not progressing as far along as they would have liked. In its experience, sustaining this type of voluntary project has been challenging for Pollution Probe. In order to be able to implement the employer program, Pollution Probe noted the need to get senior management on board by highlighting the economic benefits of a program. Through its efforts, Pollution Probe has tried to engage senior management of participating companies and has achieved limited success. Given the operational difficulties with delivering this program, Pollution Probe has transferred its implementation to other organizations, including SMART Commute, 20/20, and Clean Air Foundation. In general, Pollution Probe has achieved some positive immediate outcomes with this project. Although it encountered some challenges with on-going implementation, Pollution Probe continues to take steps towards the advancement of the TDM agenda. Pollution Probe felt the need to step back from implementation of this program in order to conduct some analysis. Next steps for Pollution Probe could include the completion of a report of case studies that documents its experiences with program execution and results in terms of modal shift, emission reductions, costs and savings. This shift in focus reveals that even though MOST funding ended for this project, Pollution Probe is continuing its efforts based on the S-M-A-R-T Movement Program to significantly expand knowledge regarding TDM in the Canadian context and to explore the need for TDM supportive policies on a broader regional and national basis. Case Study #3: Tillicum Burnside Urban Village Community Roundtable (Smart Growth B.C.) Background In Round 5, the MOST Program awarded $20,000 to Smart Growth B.C. for a project called, “Tillicum Burnside Urban Village Community Roundtable”. The project took place in the Gorge Tillicum neighbourhood in Saanich, B.C. The intersection of Tillicum Road and Burnside Road forms the centre of the neighbourhood. Both Tillicum and Burnside Roads consist of four wide lanes each from the TransCanada Highway to Craigflower Street. The rationale for the project grew out of concern that the neighbourhood had become an unsafe area. Both roads carry large automobile traffic volumes and the area lacks facilities for other users, such as pedestrians, cyclists or transit riders. When Smart Growth B.C. submitted its application to the MOST Program, it reported a vehicle count of approximately 23,000 per day on Tillicum Road and 22,500 vehicles per day on Burnside Road. It also reported that two deaths had occurred at this intersection and 500 motor vehicle accidents had taken place during a two-year period before its application. To address these issues, Smart Growth B.C. engaged in a community-led visioning and design roundtable called, “a charrette”, with the ultimate objective to revise the automobile-dominated neighbourhood into a pedestrian and transit-oriented urban village setting. In addition to MOST, Smart Growth BC had partnerships with the Gorge Tillicum Community Association, D’Ambrosio Architecture and Urban Design, the District of Saanich, and UBC James Taylor Chair. According to stakeholder interviews, MOST Program was “crucial” otherwise, the project would not have taken place. While they did receive some in-kind support, the architecture firm, which facilitated the charrette process, did require some payment. MOST funding was used to cover this cost. The project was completed in March 2004. Smart Growth BC conducted the following activities as part of its project:
In addition to submitting a final report to the MOST Program upon project completion, Smart Growth BC compiled the findings and recommendations of its projects into a final report, which is available on the Smart Growth BC website, http://www.smartgrowth.bc.ca Results Achieved Based on information in Smart Growth BC’s final report submitted to the MOST Program, the project achieve the following results:
The projec400s given a fair amount of local exposure. Public information sessions and the charrette event were promoted in local media and Smart Growth BC’s listserv. Over 1,295 people were reached directly and 32,900 people were indirectly reached through the media. The charrette roundtable consisted of 20 people presenting a variety of stakeholders, including the District of Saanich staff, residents, property owners, seniors, youth, and developers. Following the session, charrette participants reported having an increased awareness about sustainable transportation issues and supportive land uses. In addition, Smart Growth BC reported an increase in awareness of innovative solutions. The District of Saanich staff and elected officials were exposed to solutions agreed upon by local residents, property owners, and other stakeholders, such as reduced lane widths and improved facilities for pedestrians and cyclists.
This project facilitated relationship building in the Tillicum community through the charrette process and these relationships continue today. In its final report, Smart Growth BC noted, “By bringing together a variety of stakeholders (including residents, government, property owners, and developers), the process allows these disparate groups to work through competing interests to arrive at common solutions. These solutions addressed transportation issues in the study corridors, and also addressed land use changes that will support more sustainable forms of transportation.”
The charrette report contained graphic and visual recommendations, such as development concepts for key sites and proposed street sections. In addition, a variety of principles, goals and directives were generated to guide development in the Tillicum Burnside areas including:
In June 2005, the District of Saanich adopted a Streetscape Action Plan, which incorporated many of the charrette report’s recommendations. Currently, the District of Saanich allocated some funding for the first round improvements on Burnside Road. These improvements include: landscape medians, widening of sidewalks, bicycle lanes on both sides of the road and urban furniture. The District of Saanich is currently looking for funding to improve Tillicum Road and the intersection of both roads. Construction with transportation improvements is slated to begin in Spring 2006.
Smart Growth BC has promoted the success of the charrette process in the District of Saanich through its website and its work with other communities. The project is showcased as a model for other communities facing transportation issues. The charrette roundtable has been adopted by other communities and modified to meet their needs. For example, Smart Growth BC reported that a community on Cortez Island was engaging in a charrette in September 2005.
No quantifiable sustainable transportation results are available at this time as the recommendations have not been implemented yet. In its report, Smart Growth BC reported, “When the recommendations of the charrette report are implemented, particularly including an increase in residential and commercial densities, improvements to pedestrian, cycling, and transit facilities, and traffic claming measures, a decrease in greenhouse gas emissions can be expected. Residents and employees in the area will have a much greater range of viable and attractive transportation options for accomplishing their daily trips, including the ability to walk to many more of their daily needs and easy accessibility to transit and cycling”. Currently, Smart Growth BC’s role has changed with respect to this project. It now serves an advisory role to the District of Saanich when required. Smart Growth BC has no plans or expertise, nor were they expected by the District of Saanich to monitor environmental impact once recommendations were adopted. Smart Growth BC reported that “the charrette process was tremendously effective in generating a set of design recommendations, including revised street sections, improved pedestrian, cycling, and transit facilities, and supportive land use changes.” This project has demonstrated positive early outcomes. While it is still to early to report on longer-term results, the District of Saanich’s acceptance of an action plan based on the charrette recommendations suggests that this project is headed in the right direction towards creating capacity for Canadians to apply sustainable transportation thinking to their daily lives.
|
![]() |
![]() |
|||||||
|
Transport Canada |
Pacific Region |
Prairie & Northern Region |
Ontario Region |
Quebec Region |
Atlantic Region |
About us |
Our offices |
Organization and senior management |
Departmental publications |
Programs and services |
Acts |
Regulations |
[More...] |
Media room |
Advisories |
Contacts |
e-news |
News releases |
Photo gallery |
Reference centre |
Speeches |
Video gallery |
[More...] |
Emergencies |
Emergencies and crises |
Emergency preparedness |
Security |
Transport of dangerous goods |
[More...] |
Air |
Our offices |
Passengers |
Pilots |
Flight instructors |
Maintenance technicians |
Commercial airlines |
Security |
Transport of dangerous goods |
[More...] |
Marine |
Our offices |
Small commercial vessels |
Large commercial vessels |
Pleasure craft |
Marine security |
Marine infrastructure |
Transport of dangerous goods |
[More...] |
Rail |
Our offices |
Safety at railway crossings |
Rail infrastructure |
Rail security |
Transport of dangerous goods |
[More...] |