Summative Evaluation of the CFIA’s Enhanced Bovine Spongiform Encephalopathy (BSE) Initiative

Action Plan

March 17, 2009

---

• Executive Summary
• List of Tables and Figures
• Abbreviations
• 1 Introduction
• 2 Program Profile
  • 2.1 Description of the Initiative by Component
  • 2.2 Inputs and Resources
• 3 Evaluation Strategy
  • 3.1 Evaluation Issues, Scope and Objectives
  • 3.2 Evaluation Approach and Methodology
  • 3.3 Evaluation Limits and Challenges
• 4 Key Findings and Conclusions
  • 4.1 Success and Achievements
    • 4.1.1 Activities and Outputs
    • 4.1.2 Outcomes
  • 4.2 Governance
  • 4.3 Design and Delivery
    • 4.3.1 Communication and Outreach Activities
    • 4.3.2 Collaboration with Partner Organizations and Stakeholders
    • 4.3.3 Human Resources and BSE Expertise
  • 4.4 Relevance and Continued Need
  • 4.5 Performance Measurement and Reporting/Results-Based Management
  • 4.6 Cost-Effectiveness and Alternatives
• 5 Recommendations
• Appendix A – List of Evaluation Advisory Committee Members
• Appendix B – CFIA Enhanced BSE Initiative Logic Model
• Appendix C – Evaluation Issues and Questions
• Appendix D – International Markets Partially Re-opened¹, 2003–2008

---

Executive Summary

Profile

Canada has had 14 domestic cases of bovine spongiform encephalopathy (BSE) as of November 2008. In response to the first of these cases in May of 2003, the Government of Canada, through Health Canada and the Canadian Food Inspection Agency (CFIA), enhanced its existing BSE-related activities and rapidly developed a comprehensive suite of internationally recognized science-based measures to effectively minimize the likelihood of exposure, amplification and spread of BSE within the cattle population. Through a fair and efficient regulatory regime that also supports domestic and international markets, the CFIA aims to protect Canadian and international consumers from associated human health risks and ensure the sustainability of the Canadian cattle herd. Since 2003, the CFIA obtained additional financial resources for the Enhanced BSE Initiative, including the following components:

• Enhanced BSE Surveillance and Testing
• Enhanced Tracking and Tracing and Enforcement of Cattle Identification (ID)
• Facilitating the Expansion of Domestic Slaughter Capacity
• Removal of Specified Risk Material (SRM)
• Expanding Export Markets (including Export Certification and Strengthening Science & Support.)

Note that this evaluation does not cover the "Strengthening Animal Feed Restrictions" component (also known as the Enhanced Feed Ban). During the period under evaluation (2003-2004 to 2007-2008), the Enhanced BSE Initiative (with the exception of the Enhanced Feed Ban) received a total of $160,320,000.

Evaluation Strategy

This summative evaluation of the Enhanced BSE Initiative was conducted by Science-Metrix, under contract to the Evaluation Directorate of the CFIA’s Audit, Evaluation and Risk Oversight Branch (AERO) to assess the success and achievements, governance, design and delivery, relevance and continued need, performance measurement and reporting, and cost-effectiveness and alternatives of the Enhanced BSE Initiative and provide recommendations as necessary.

The data collection tools for this summative evaluation included primary data sources—telephone and in-person interviews with 30 key informants internal to the CFIA (6 from senior management and 24 from CFIA branches) and 29 key informants external to the CFIA, including informants from federal and provincial departments (n = 9) and a variety of industry representatives for both beef and dairy (i.e., producers, processors, renderers, exporters, etc.) from across Canada (n = 9)—as well as secondary data sources, including a review of documents and files (such as performance measurement data and reports specific to this initiative) from the CFIA and relevant external national and international organizations.

Difficulties were experienced in linking outputs and outcomes of the Enhanced BSE Initiative to specific inputs because of the way in which tied funding at the CFIA is tracked. In addition, attribution issues have arisen due to the fact that this horizontal initiative shares outcomes with other federal and provincial departments and involves multi-stakeholder participation.

Key Findings and Conclusions

Success and Achievements

Notable achievements included: rapidly surpassing the planned minimum number of samples tested for BSE; creating an effective network of provincial and federal laboratories; completing and reporting 11 traceback investigations (under the evaluation period) of confirmed BSE cases in Canada; developing and implementing a streamlined review and approval process and increasing line inspection capacity for slaughter plants; adjusting and implementing import policy to ensure coherence with that of countries to which Canada exports; and posting four veterinary technical experts abroad. All of the above have supported the re-opening and expansion of export markets, one of the major concerns of the initiative’s stakeholders. As for science-based activities and advisory outputs, more than half of internal informants reported scientific knowledge within the CFIA to be current and of high quality. The CFIA is also involved in large-scale collaborative projects and provides funding for research on BSE and other transmissible spongiform encephalopathies (TSEs). However, a number of planned outputs at the provincial or national levels have either not been achieved or data verifying performed outputs were unavailable at the time of the evaluation. These issues are addressed in the Recommendations, below.

Evidence on outcomes suggests that the initiative has successfully responded to the needs of principal stakeholders targeted by this initiative (i.e., the Canadian public and the Canadian beef and cattle industries). Informants praised the CFIA’s involvement in the accurate identification and management of BSE cases, noted its efforts to communicate transparently with the Canadian public in the wake of BSE crises (stating that these were largely responsible for increased consumer confidence), and cited the recognition from international authorities of the science-based management of BSE in Canada. Informants from some stakeholder groups, however, expressed dissatisfaction with the initiative’s trade-related outcomes, particularly as they relate to expanding domestic slaughter capacity and restoring or opening international markets. Various unexpected outcomes, such as the development of stronger working relationships with provincial and industry partners and other "spillover" benefits, were also observed.

Governance

The CFIA’s governance and delivery of the Enhanced BSE Initiative was well integrated within the existing management and operational structure. Almost all internal informants reported that the current management framework was adequate for delivering BSE-related activities. However, evidence from the internal interviews suggests that while the roles and responsibilities were fairly well understood within each branch/unit, they were less so at the organizational level. A dedicated BSE Task Force was in place until the end of 2004, followed by a BSE coordinating committee until 2006, but the initiative has since suffered from the absence of structured/coordinated efforts.

Design and Delivery

Overall, the design and delivery of the Enhanced BSE Initiative and the CFIA’s role in programs and activities led by partner government organizations were adequately integrated, primarily because the CFIA’s activities were directly linked to their enforcement and regulatory mandate on issues relating to food safety and animal health. However, the CFIA’s dual role as both a regulatory body and a supporter of the Canadian beef and cattle industries was not always clear, particularly for external stakeholders, resulting in these stakeholders’ unsuitable expectations.

Communication and collaboration: The CFIA’s communication efforts with the Canadian public were uniformly praised by internal and external informants. Internal informants were also very satisfied with the dissemination of BSE information to partner organizations and stakeholders and were overwhelmingly positive about federal and provincial involvement and cooperation with the CFIA. However, less than half of external informants from provincial departments or industry associations expressed satisfaction with the level of communication they received, and many of these informants cited issues with the CFIA: confusion about roles, responsibilities and directives; problems with coordination of national and regional activities; and gaps in the handling of specific components. Internal informants were also dissatisfied about communication and coordination within the CFIA, particularly between branches and between the National Capital Region and regional operations.

Human resources and training: Human Resources (HR) received funding from the BSE initiative to support corporate overhead and training activities through the Learning Division, and this funding enabled the CFIA to maintain its expertise and competency through hiring of highly qualified scientists and technicians. However, BSE expertise appears to be decreasing within the CFIA due to ongoing and upcoming changes in BSE staffing due to retirement, promotions/re-assignments, changes within the management structure of the organization, or for other reasons.

Relevance and Continued Need

The Enhanced BSE Initiative is aligned with government priorities and the CFIA’s mandate, and it generally met and continues to meet the needs of its two principal beneficiaries: the Canadian public and the Canadian beef and cattle industries. Key informants, both internal and external, were in nearly unanimous agreement regarding the continued need for tracking, tracing and enforcement; removal of SRM from the human food supply; and expanding export markets. Only the expansion of domestic slaughter capacity was seen as having lost its relevance.

Performance Measurement and Reporting/Results-Based Management

The review of performance measurement and reporting strongly suggests that these are not adequate. The consensus among internal informants is that the current lack of results-based management within the CFIA needs to be addressed. The CFIA does not have an integrated performance measurement and reporting system for BSE. Although internal informants reported that the information and data were collected at the operations and program levels and were sometimes used to help manage field- or micro-level activities at the branch level, performance information was not used in management decision-making. Instead, the key driver behind the collection of performance data was to fulfill external reporting requirements. The production of documents requiring these data was reported to be resource-intensive and inefficient because performance indicators were collected by different branches and used in isolation from the BSE initiative as a whole, and some indicators that reflect industry performance or compliance were not collected in a systematic or integrated way. The diversity and constant change of type and scope of BSE performance indicators that were needed-including data for the Departmental Performance Reports (DPR), OIE reporting, reporting against the Results-based Management and Accountability Framework (RMAF), and other internal reporting needs-posed an additional challenge.

Cost-Effectiveness and Alternatives

Significant issues related to financial tracking and attribution created challenges in determining whether the Enhanced BSE Initiative and its individual components were cost-effective. Nevertheless, internal and external informants nearly unanimously felt that Canadians received value for their money with the Enhanced BSE Initiative and that its importance in mitigating the economic and health impacts of BSE in Canada should not be underestimated. The CFIA is currently considering the following alternatives to increase the cost-effectiveness of certain initiative components: reducing the number of samples tested; evaluating the impacts of discontinuing the Sample Reimbursement Program (SRP), and ending support of the expansion of domestic slaughter capacity while maintaining the confidence of domestic and international consumers.

Recommendations

The findings of this summative evaluation suggest that the CFIA’s Enhanced BSE Initiative has made a major contribution to the management and mitigation of the BSE crisis in Canada. Moreover, the evidence suggests that all individual components of the initiative remain relevant, with the exception of "Facilitating the expansion of the domestic slaughter capacity". The following eight recommendations have been developed and increase the cost-effectiveness of the remaining components of the initiative.

Recommendation 1: Coordination and communication

To strengthen coordination and communication with regard to BSE-related activities, both within the CFIA and with external partner organizations and stakeholders, a dedicated, expert-led coordination structure should be established in which roles and responsibilities are clearly established and communicated.

Recommendation 2: Performance measurement and reporting

To improve current performance data collection and reporting practices, the CFIA should consider the following measures: dedicate targeted funding for BSE performance measurement and reporting and subsequent evaluations; develop and implement a results-based management plan that integrates both program management and senior management needs and external reporting needs. To this effect, the CFIA should dedicate at least one position to the coordination of BSE performance measurement activities across the organization and promote a performance management culture at all levels of the agency.

Recommendation 3: HR and BSE expertise

To develop and sustain a critical mass of BSE expertise within the CFIA, succession planning should center on a commitment to hiring highly qualified staff and allow for an extended transition period so that new staff can be mentored and trained by existing BSE specialists within the agency.

Recommendation 4: Financial tracking

To better manage and assess outputs of BSE tied funds, the CFIA should track funds at the level of specific activities funded under the BSE initiative. This would also allow for cost-effectiveness analysis, both at the level of sub-activities and program components.

Recommendation 5: BSE surveillance testing and the Sample Reimbursement Program

The CFIA’s future decisions as to its BSE surveillance testing targets and the SRP should continue to be supported by science-based guidelines addressing both the effectiveness of the risk management measures for BSE in animals as well as the effectiveness of the feed ban. The CFIA should also develop a communication strategy (including a consultation strategy) to better inform stakeholders with regard to ongoing discussions and developments on the future of the SRP. To this end, better coordination and communication is needed between those at the front lines of BSE testing.

Recommendation 6: Tracking, tracing and enforcement

The CFIA should continue to work with industry and other governments to ensure that the national cattle identification (ID) program and database is standardised and includes date of birth, movement capture, identification of deadstock, and better identification of premises. The CFIA should also enforce activities beyond tagging compliance (e.g., enforce the submission of data to the national database).

Recommendation 7: Expanding export markets

To help manage industry expectations, the CFIA should clarify its role and responsibilities for this component, particularly in relation to other government agencies involved (e.g., DFAIT, AAFC). The CFIA should also continue to integrate BSE trade-related activities into broader country- or commodity-based strategies and confirm the added value of its regionally focused technical experts.

Recommendation 8: Continuity and scope

As part of its continued efforts to develop an integrated approach to food safety, public health and risk management for animals and humans, the CFIA should expand the scope of specific activities within the Enhanced BSE Initiative beyond BSE, particularly to other issues related to TSEs and other emerging diseases. Finally, the CFIA must ensure that it continues to meet stakeholder expectations and maintain confidence regarding the legitimacy of its regulatory/enforcement role for BSE.

List of Tables and Figures

• Table 1
  • Financial resources received and spent ($’000s)1 and total number of FTEs, 2003 - 2004 to 2007 - 2008
• Table 2
  • Perception of internal key informants and external key informants on the success of the Enhanced BSE Initiative’s outcomes (as identified in the 2007 RMAF)
• Table 3
  • Perceptions of external key informants on the success of the Enhanced BSE Initiative’s communication and education activities
• Table 4
  • Perceptions of internal key informants on the continued need for the Enhanced BSE Initiative, overall and for each component
• Figure 1
  • Beef (kg) available in Canada, per person and per year (adjusted for losses), 1997-2007
• Figure 2
  • Frequency of BSE-related content in Canadian newspapers by year, 1997-2008*
• Figure 3
  • World exports of Canadian beef and cattle (in millions CAN $), 2002-2007

Abbreviations

• AAFC
  • Agriculture and Agri-Food Canada
• AERO
  • Audit, Evaluation and Risk Oversight
• ATQ
  • Agri-Traçabilité Québec
• BSE
  • Bovine Spongiform Encephalopathy
• CCIA
  • Canadian Cattle Identification Agency
• CCIP
  • Canadian Cattle Identification Program
• CFIA
  • Canadian Food Inspection Agency
• CJD
  • Creutzfeldt-Jakob Disease
• CWD
  • Chronic Wasting Disease
• ERC
  • Expenditure Review Committee
• DCM
  • Data Collection Matrix
• DFAIT
  • Department of Foreign Affairs and Internal Trade
• DPR
  • Departmental Performance Report
• FAIT
  • Finance, Administration and Information Technology
• FTE
  • Full-Time Equivalent
• HR
  • Human Resources
• ID
  • Identification (in reference to cattle ID program)
• IGAC
  • Industry-Government Advisory Committee
• MAPAQ
  • Ministère de l'Agriculture, des Pêcheries et de l'Alimentation du Québec
• NCFAD
  • National Centre for Foreign Animal Disease (Winnipeg)
• OIE
  • World Organization for Animal Health¹
• PTD
  • Professional and Technical Development
• PWGSC
  • Public Works and Government Services Canada
• RMAF
  • Results-based Management and Accountability Framework
• SRP
  • Sample Reimbursement Program
• SRM
  • Specified Risk Material
• TB
  • Treasury Board
• TSE
  • Transmissible Spongiform Encephalopathy

1. Introduction

Bovine spongiform encephalopathy (BSE) is a progressive, fatal neurodegenerative disease found in cattle. It is a member of the family of transmissible spongiform encephalopathies (TSEs), which are diseases associated with the presence of prions, such as scrapie in sheep, chronic wasting disease (CWD) in deer and elk, and Creutzfeldt-Jakob Disease (CJD) in humans. Science-based evidence indicates that BSE can be transmitted through consumption of feed containing specific tissues of infected animals. Consumption of these tissues by humans has been linked to a variant form of CJD, and as such, BSE is considered to be a food-borne disease.

The presence of BSE in a country’s domestic cattle herd therefore poses a risk to human and animal health, which is necessarily accompanied by economic impacts, particularly international trade implications. As a prion disease, BSE also differs from several other animal diseases, such as in its mode of transmission (through feed rather than through contact with an infected animal) and its long incubation period; the nature of this disease therefore has important implications in terms of its control and management (see also Section 4.6). Following emergence of BSE in Europe (particularly the United Kingdom), the Government of Canada implemented a series of BSE-related measures, including making BSE a reportable disease (1990), BSE surveillance testing (1992) and, a ban on certain bovine materials (1997) in feed for ruminants.²   In 2001, a cattle identification program was established by the industry-led Canadian Cattle Identification Agency (CCIA) and Agri-Traçabilité Québec (ATQ), in a key move toward enabling the tracking and tracing of animals.

Following the first Canadian case of BSE, confirmed in May 2003, the Government of Canada, acting on the recommendations of an internal team of experts, rapidly moved to strengthen these existing measures. In particular, the Canadian Food Inspection Agency (CFIA) designed and implemented the Enhanced BSE Initiative to effectively minimize the likelihood of exposure, amplification and spread of BSE within the cattle population. This initiative comprises a comprehensive suite of internationally recognized science-based measures by which the CFIA aims to protect Canadian and international consumers from associated human health risks, and to ensure the sustainability of the Canadian cattle herd through a fair and efficient regulatory regime that contributes to market access.

The present summative evaluation of the CFIA’s Enhanced BSE Initiative covers the period from 2003-2004 to 2007-2008. It was conducted by Science-Metrix, under contract to the Evaluation Directorate of the CFIA’s Audit, Evaluation and Risk Oversight Branch (AERO), and overseen by an advisory committee led by AERO.

An overview of the Enhanced BSE Initiative is provided in the Program Profile (Section 2). The report goes on to describe the evaluation strategy (Section 3—including the evaluation issues, scope and methodology), and presents key findings and conclusions on six evaluation issues (Section 4): success and achievements, governance, design and delivery, relevance and continued need, performance measurement and reporting, and cost-effectiveness and alternatives. In light of these findings, appropriate recommendations are provided (Section 5 ).

Program Profile

The protection of public health, food safety, and animal health has been and continues to be a fundamental priority for the Government of Canada. Since the first domestic case of BSE was confirmed in May 2003 in Alberta, Canadian government organizations and industry have enacted a comprehensive strategy to safeguard public and animal health, maintain American integration (following from the high level of movement between the Canadian and the American cattle herds, including animals products and feed), support international trade and industry viability, and advance international advocacy. More specifically, the CFIA has developed and implemented a comprehensive suite of internationally recognized science-based measures to effectively minimize the likelihood of exposure, amplification and spread of BSE within the cattle population and to protect Canadian and international consumers from the associated human health risks. These measures are integrated within multiple branches and programs of the CFIA, and involve a large number of external partners and stakeholders (e.g., other federal departments, all provincial governments, and various industry stakeholders), resulting in a wide-ranging but complex initiative.

Over the five-year scope of this evaluation (2003-2004 to 2007-2008) the CFIA obtained additional financial resources to enhance its activities related to BSE. The design and delivery of the CFIA’s Enhanced BSE Initiative activities were based on the recommendations of an international team of experts, as well as the science and experience stemming from BSE management in previously affected nations.

2.1 Description of the Initiative by Component

The resulting Enhanced BSE Initiative comprises five main components, each of which has received dedicated TB funding. Note that an additional component of the CFIA’s Enhanced BSE measures, Removal of Specified Risk Material (SRM) from the Animal Feed Supply (also known as the Enhanced Feed Ban) is not included in this summative evaluation. The five components under evaluation and their objectives are:

1. Enhanced BSE Surveillance and Testing

This set of actions aims to evaluate the potential presence of BSE-infected animals in Canada. The CFIA conducts extensive testing on populations of adult cattle at high risk, thus following the recommendations of the World Organization for Animal Health (OIE). These efforts enable the CFIA to get a better understanding of the epizootic disease and to better evaluate the mitigation measures in place, in order to adapt or strengthen them if necessary. The Sample Reimbursement Program (SRP) was implemented in 2005, providing financial compensation for eligible surveillance samples collected by cattle producers and veterinarians to increase the availability of targeted samples.

2. Enhanced Tracking and Tracing and Enforcement of Cattle Identification (ID)

The tracking and tracing of cattle is crucial to controlling progressive diseases such as BSE. When an infected animal is found, an investigation is conducted to trace and contain its birth and feed cohorts (i.e., potentially infected animals). To this end, the CFIA inspects and enforces the Canadian Cattle Identification Program (CCIP) (which was created by the CCIA and ATQ), in which all cattle are uniquely identified with an approved ear tag upon leaving their farm of origin.

3. Facilitating the Expansion of Domestic Slaughter Capacity

Every year, over a million Canadian cattle are slaughtered outside of Canada, mainly in the US. Following the closure of export markets in 2003, which generated a large surplus inventory of cattle that could not be slaughtered domestically, the CFIA has worked to facilitate the expansion of domestic slaughter capacity by increasing their line inspection capacity and streamlining the review/approval process for opening new plants.

4. Removal of Specified Risk Material (SRM)

Since 2003, all SRM (including distal ileum of cattle of all ages and the skull, brain, trigeminal ganglia, eyes, tonsils, spinal cord, and dorsal root ganglia of cattle aged 30 months or older) must be removed from all cattle slaughtered in Canada to prevent them from entering the human food chain. This requirement went into effect in federally registered meat establishments on July 24, 2003, through CFIA directives and was extended to all other beef produced in or imported to Canada on August 23, 2003.

5. Expanding Export Markets

This component aims to expand export markets through technical oversight, the establishment of geographically or regionally based technical experts and adjustment of the BSE import policy.³ It also seeks to maximize market opportunities by strengthening science and support for securing market access, solidifying institutional support for meat inspection reform and quality assurance-based certification systems, and enhanced certification of exports. This last set of actions aims to help Canada maintain its trade relations with foreign partners, both as an importer and an exporter of beef products. In particular, it ensures that the new inspection and export certification requirements set by the US authorities in July 2007 are properly fulfilled so that this major export market can be preserved. Three main activities have thus been funded under this sub-program objective: a) Export Certification, b) Expanding Export Markets, and c) Strengthening Science and Support.

2.2 Inputs and Resources

The total funding received for the Enhanced BSE Initiative over the five-year period under evaluation is $241,728,000. As the Animal Feed Restrictions (Enhanced Feed Ban) component is not covered by this evaluation, the Enhanced BSE Initiative as referred to in this report received a total of $160,320,000. Table 1 presents a break-down of the financial resources granted to the CFIA by the TB for the period under evaluation for the Enhanced BSE Initiative, compared to those received and spent (pay and non-pay) to conduct eligible activities. The table also includes the Employee Benefit Plans (EBP) and Public Works and Government Services Canada Accommodation Costs (PWGSC), as well as the total number of full-time equivalents (FTEs) for BSE. The Finance, Administration and Information Technology (FAIT) Branch has confirmed that certain tied funding could not be tracked in greater detail, which has implications for performance measurement and management, as discussed in Section 4.5.

Table 1  - Financial resources received and spent ($’000s)1 and total number of FTEs, 2003-2004 to 2007-2008

Click on image for larger view

Notes:

1. The CFIA plans to pursue the renewal/extension of sunsetting BSE funding.
2. Item # 3 contains data from 4(b) and 5(b).  Separate Internal Order Codes were not established to track this information. Also, the SRP ($29,943.6) is included in the amount spent. A total of $17,736 (pay and non-pay) was received for the dedicated activities of this component, i.e., Increased Line Inspection Capacity and Established Review/Plan Approvals.
3. Animal Feed Restrictions (i.e., the Enhanced Feed Ban) are not covered by this summative evaluation.
4. Expanding Export Markets: Funds could not be tracked as the activities were not designated under separate Internal Codes. This relates to the difficulty in tracking the funds received under a single TB submission; see note 2.
5. Total number of FTEs over the period (sum of annual number of FTEs). This table also includes Employee Benefit Plans (EBP) and Public Works and Government Services Canada Accommodation Costs (PWGSC).

3 Evaluation Strategy

The present evaluation follows the plan set out in the Evaluation Framework prepared by the evaluation team during the design phase, which was overseen by the evaluation advisory committee at the CFIA. This committee, led by the AERO Branch, includes CFIA representatives from Public Affairs, Programs & Policy, Operations, and FAIT Branches (see Appendix A).

The evaluation issues and questions were determined by the AERO Branch in consultation with the evaluation advisory committee. Subsequent stages of this study, including data collection and reporting of the findings, were also overseen by the evaluation advisory committee. For instance, the committee provided feedback on interview questionnaires during the study design, facilitated data collection in the case of CFIA documents, and provided guidance on the preliminary drafts of the evaluation report.

The original logic model included in the draft consolidated Results-based Management and Accountability Framework (RMAF) has been revised and adjusted in consultation with AERO and the evaluation advisory committee. The revised logic model is presented in Appendix B. The evaluation team developed a data collection matrix (DCM), which links the evaluation issues as identified in the logic model with the related evaluation questions, performance indicators, and/or research methods and data sources.

The interview guides were developed according to the indicators identified in the DCM for each evaluation question. The DCM also allows for the clear identification of the data that will be collected using each instrument and for both secondary and primary data collection methods. Additional details on evaluation instruments are included in the Methodology Supplement.

3.1 Evaluation Issues, Scope and Objectives

The purpose of this evaluation is to assess the 1) success and achievements; 2) governance; 3) design and delivery; 4) relevance and continued need; 5) performance measurement and reporting, and; 6) cost effectiveness and alternatives of the Enhanced BSE Initiative (excluding the Enhanced Feed Ban component) and to provide recommendations as necessary. Evaluation issues and specific questions relating to each are presented in Appendix C; evaluation issues are cross-referenced with Expenditure Review Committee (ERC) Issues in the Methodology Supplement.

The evaluation covers five components of the CFIA’s Enhanced BSE Initiative for which funding was received from the TB; these are presented with their specific objectives in Section 2. The period evaluated covers fiscal years 2003-2004 to 2007-2008. Funding of most of the Enhanced BSE Initiative’s components, with the exception of the Enhanced Feed Ban, is sunsetting in 2008-2009. This evaluation is performed according to the TB standards and requirements and is conditional for funding renewal.

3.2 Evaluation Approach and Methodology

The methodology for this evaluation incorporated multiple lines of evidence, both qualitative and quantitative, to obtain the most comprehensive, valid and reliable findings. The methodology included key informant interviews, document and file reviews, including websites and external documents relating to BSE (e.g., scientific publications and media reports).

The methods used to address specific evaluation issues are presented in Appendix C and the revised logic model of the CFIA Enhanced BSE Initiative is presented in Appendix B. The DCM, which cross-references evaluation issues, specific evaluation questions and the logic model, is presented in the Methodology Supplement. The data collection methods for this summative evaluation are based on the following data sources:

Primary Data: Key Informant Interviews (Telephone and In-Person) (N= 59)

Internal Key Informants

• Internal key informants (n = 30 interviewees; approximate average interview length = 70 minutes) from the CFIA; these were distributed as follows: 6 with CFIA senior management and 24 with CFIA Branch representatives. Two interviews were conducted with two informants present.
• The evaluation advisory committee recommended priority potential respondents. The final list was approved by the evaluation advisory committee.
• A member of the AERO group acted as an observer in each of the interviews conducted with CFIA informants, either by phone or in person. ⁴.
• Interview guides for internal informants are presented in Methodology Supplement

External Key Informants

• External key informants (n = 29 interviewees; approximate average interview length = 40 minutes), including informants from federal and provincial departments and a variety of industry representatives for both beef and dairy (i.e., producers, processors, renderers, exporters, etc.) from across Canada; these were distributed as follows: 11 from government departments (2 national, 9 provincial) and 18 from industry associations or other external stakeholders (9 from national-level organizations, 9 from provincial-level organizations). Two interviews with government representatives at the provincial level were conducted with two informants present. Two interviews were conducted with two informants present.
• Potential external key informants were first identified using the list of attendees at the Beef Industry Value Chain Roundtable from 2005 to 2008.⁵ The evaluation advisory committee recommended priority potential respondents. Regional CFIA representatives were subsequently consulted to validate this list or to suggest additional potential interviewees. The final list was approved by the evaluation advisory committee.
• Interview guides for external informants are presented in the Methodology Supplement.

Secondary Data: Document and File Review, including review of performance measurement systems and analysis of data collection and information and other research

• Documents and files from the CFIA, the federal government, and international organizations (e.g., OIE), including websites.
• Documents and files from the CFIA that relate specifically to performance measurement, which may include information provided by internal key informants during interviews.
• Review of scientific publications and other external documents relating to BSE.
• Media reports, including news releases, relating to BSE.

3.3 Evaluation Limits and Challenges

Absence of mid-term evaluation and evaluability assessment

This summative evaluation constitutes the first assessment of the CFIA’s Enhanced BSE Initiative. The evaluation strategy presented in the original RMAF included a mid-term evaluation (formative evaluation), but this was not conducted. According to the initiative's Performance Measurement Strategy, review of performance measures on outcomes had been planned for the mid-term evaluation. Its absence constituted a challenge for this evaluation. Moreover, this evaluation would have benefited from an evaluability assessment (an assessment of the availability and quality of data) of the sub-program activities prior to the summative evaluation. While this type of assessment is not required by the TB, it is state-of-the-art in evaluation practice and is particularly helpful in evaluating large and complex initiatives.

Exclusion of international stakeholders from the evaluation

International stakeholders of the Enhanced BSE Initiative were not consulted as part of this summative evaluation. Findings of this evaluation as to the international outcomes (e.g., recognition and benchmarking) of BSE-related activities led by the CFIA are therefore limited to data found in files and documents and statements made by Canadian informants interviewed.

Program management structure, multi-stakeholder participation, and horizontal initiatives

The activities of the Enhanced BSE Initiative are embedded in other CFIA program activities that are distributed across the organizational structure. As such, the managers and key CFIA staff responsible for Enhanced BSE Initiative activities do not belong to a cohesive structure, nor are they located in a single area or building. In addition, during the period under evaluation, there was significant turnover in senior management, including retirement and promotion/re-assignment of staff involved in the BSE file. Furthermore, the stakeholders and partner organizations involved in the delivery of the Enhanced BSE Initiative are numerous, varied, and scattered across Canada. These include several industries, as well as federal and provincial government authorities. These horizontal aspects pose a challenge for the assessment of incrementality and attribution of the initiative’s successes and achievements.

Evaluation scope: Exclusion of the feed ban from the evaluation

Activities related to Strengthening Animal Feed Restrictions (the Enhanced Feed Ban) were not included in this summative evaluation. Developed during the period under evaluation and implemented in July 2007, this activity will be evaluated separately at a later date. However, this activity is essential to eradicating the incidence of BSE in Canada and appears to have been among the most challenging, both for the CFIA and its stakeholders. Inclusion of the design and early implementation of the Enhanced Feed Ban would therefore have increased the significance of this summative evaluation of the Enhanced BSE Initiative.

4 Key Findings and Conclusions

4.1 Success and Achievements

The present section presents the successes and achievements of the CFIA’s Enhanced BSE Initiative, more specifically those related to the outputs and outcomes of its activities that are identified in the logic model for the initiative (see Appendix B).

4.1.1 Activities and Outputs

In evaluating the activities and outputs of the Enhanced BSE Initiative, the inputs and resources (see Section 2.2) were considered and discussed in relation to the extent to which planned activities have been implemented and have produced their expected outputs. As mentioned previously, difficulties in detailed tracking BSE funds and FTEs were encountered, which stem in part from the high degree of integration of BSE within CFIA activities, as well as the way in which BSE activities are coded within the financial tracking system (see also Section 4.5)⁶. Nevertheless, the evidence suggests that, as a whole, the CFIA implemented the activities pursuant to the TB submissions and has measurable outputs for most of these activities. Based on available data, here are the key achievements of the Enhanced BSE Initiative, listed for each of the five components:

1. Enhanced BSE Surveillance and Testing

• The CFIA rapidly surpassed the planned minimum number of samples tested for BSE, having tested 23,550 samples in 2004 (compared to a stated minimum objective of 8,000) and 55,000-58,000 samples per year in the three following years (compared to a stated minimum objective of 30,000)⁷. This large number of samples contributed to obtaining controlled risk status from the OIE. This effort was supported in part by the SRP, which acted to increase the availability of targeted samples. Close to 99% of these samples met minimum lab quality test standards and were compliant with the 48-hour timeliness standard.⁸
• An effective network of laboratories (provincial and federal) was created, including the Winnipeg-based National Centre for Foreign Animal Disease (NCFAD- now located in Lethbridge), which has been designated as a BSE reference lab by the OIE.

2. Enhanced Tracking, Tracing & Enforcement

• In addition to enforcing a roughly 98% compliance rate for cattle ID in Canada (visiting 7,000-9,500 cattle/bison sites per year since 2005, up from 1,800 in 2004⁹), the CFIA provided policy and regulatory input to the CCIP to improve the tracking and tracing system.
• Processes have been put in place to manage internal and external aspects of BSE traceback investigations; comprehensive report summaries are published on the CFIA website, on average, 2.5 months after confirmation of an infected animal, although this turnaround time has increased slightly (to 4 months) in the last two years,¹⁰ for reasons that could not be clearly identified; however, tracking and tracing issues were highlighted by a number of informants.
• The CFIA completed and reported 13 traceback investigations of confirmed BSE cases in Canada (plus 1 Alberta-born case in Washington; 1 investigation is pending as of November 2008).

3. Facilitating the Expansion of Domestic Slaughter Capacity

• To reduce dependency of the cattle and beef industries on exports, the CFIA developed and implemented a streamlined review/approval process for domestic processing plants, trained regional staff in this process, and hired 29 staff in 2005-2006.¹¹ This enabled the registration of two new processing establishments and line capacity enhancements at nine establishments; however, the actual utilization of domestic slaughter facilities has since decreased, likely due to the resumption of live cattle exports to the US.

4. Removal of SRM from the Human Food Supply:

• Internal informants agreed that regulations for removal of SRM were developed and administered "very quickly" and that these regulations were communicated effectively to producers and processors. This is a key part of the initiative, as it helps to ensure the safety of the Canadian food supply and maintain public and export market confidence.
• An overall 97% compliance rate was found for the three tasks related to removal of SRM from the human food supply in federal establishments, as verified by CFIA inspections.¹² Increased support and monitoring of compliance inspections in non-federal establishments, for which compliance verification is not the sole responsibility of the CFIA, was also reported for the 2005-2006 fiscal year,¹³ but enforcement actions performed for identified non-compliant and non-federal establishments may be inadequate in some areas; a process to report on the compliance in provincial establishments should be in place by the end of 2008-2009.¹⁴

5. Expanding Export Markets:

• The CFIA adjusted and implemented its import policy to ensure coherence with that of countries to which it exports, including conducting the required country evaluations/reviews for equivalency.¹⁵
• As of November 2006, the CFIA posted four veterinary technical experts in Mexico City, Brussels, Tokyo and Beijing as a pilot project to provide scientific/technical knowledge in the context of policy and trade negotiations with targeted export markets. As of February 2008, the four positions have been re-classified with the Department of Foreign Affairs and Internal Trade (DFAIT) from term to indeterminate positions. Internal informants report that significant work was done by these experts to support re-opened markets, such as Japan.
• Internal informants reported that significant efforts went into export certification, especially for Japan and the US. There have been no BSE-related rejections of Canadian beef exports since 2005-2006.¹⁶

Other Outputs: Science-Based and Operational Outputs

• Relevant science-based activities and outputs include Animal Health Risk Analysis projects; reviews and reports (including over a dozen BSE-specific risk assessments); a three-part study into the natural history of BSE in North America; and 14 research projects related to BSE, mainly on the development of new diagnostic tests, that were led by 11 different researchers and, in some instances, were undertaken in collaboration with researchers from other government departments or academia (e.g., Canadian Prion Disease Network, PrioNet Canada, and the Alberta Prion Research Initiative). Many of these research outputs have been shared through a variety of means and outreach activities, both internal (technical reports, training and dissemination activities) and external (CFIA website, peer-reviewed papers, presentations, etc.).
• Internal informants have also highlighted the value in having BSE experts at the CFIA who are internationally recognized. Among these experts, one is the government of Canada delegate to the OIE, and another is the Head of Pathology at the CFIA’s NCFAD lab—an OIE Reference Laboratory for BSE since 2006—and has been designated by the OIE as the lab’s BSE expert. Canada is thus in a position to contribute to OIE standards, science, and recommendations. These links with the OIE and other connections (such as participation in government-academic research networks) also ensure that the CFIA has rapid and comprehensive access to the latest science on BSE.
• More than half of the internal informants reported that scientific knowledge within the CFIA was current and of high quality, and none said that this knowledge was inadequate. The overwhelming consensus was that this knowledge was readily available to those who needed it and that it was adapted to the needs of managers and thus could be effectively put to use in making science-based regulatory decisions.
• For each component of the initiative, training needs were identified and approximately 20 BSE-related courses (with appropriate materials) were designed, developed, and delivered by the Learning Division and the Professional and Technical Development (PTD) section; all courses were subsequently evaluated by participants. See Section 4.3.3 for more details.
• Further HR outputs, such as recruitment and succession plans, are discussed in Section 4.3.3.
• The Public Affairs Branch commissioned a series of opinion polls and surveys that generated research findings pertaining to consumer confidence in beef safety and that tracked Canadians’ confidence in the CFIA’s handling of BSE.¹⁷

Partial Outputs and Missing Data

There are, however, a number of planned outputs at the provincial or national level that have either not been achieved or for which the data to verify that outputs were effectively produced were unavailable at the time of the evaluation:

• A major need exists for continued implementation and improvement of the tracking and tracing program, as evidenced by certain difficulties that had to be overcome in the course of traceback investigations.¹⁸ To ensure rapid and comprehensive traceability in the Canadian herd (including animal movement), the aspects that need the greatest improvements are: more comprehensive and standardized data capture and reporting of cattle movements and tag retirement through a central national database (e.g., by abattoirs); infrastructure development; assigning premises identification to all feedlots, backgrounders, packing plants, exporters, renderers, deadstock sites, etc.; and improved communication and awareness of identification/tracking requirements, guidelines and procedures among the non-compliant minority. Progress on this front within the next five years is expected based on recent and well-documented industry-government efforts.¹⁹
• As mentioned above, there is room for continued improvement of verification and enforcement actions related to compliance with SRM removal regulations, especially in non-federally registered establishments (where inspection is not the sole responsibility of the CFIA), as well as a greater focus on enforcement now that the education/communication work has been done.
• Increased laboratory testing to verify SRM removal has not been implemented, according to internal informants, although no clear reason for this has been provided.
• Data are unavailable to confirm whether the samples obtained by the CFIA for surveillance purposes were from animals displaying clinical symptoms consistent with BSE or were from those belonging to the so-called 4-D groups (dead, dying, diseased and downers). The general perception is that the value of the samples (i.e., their representativeness) could be improved.
• Data were not available on the number of export certificates issued for animals, animal products, or feed.
• No evidence was found of the following activities outlined in the TB submissions within the "Strengthening Science & Support" sub-component (within the Expanding Export Markets component): "Meat Inspection Reform" and "Quality Assurance Based Certification System." There is also no clear understanding among internal informants as to what activities these programs represent, how they relate to the expansion of export markets, or whether they had been reprofiled.
• The available data on some of the outputs of science-based projects were incomplete, suggesting a need to improve the monitoring and coordinating of research within the CFIA and performed in collaboration with external partners and R&D stakeholders.

4.1.2 Outcomes

Overall, the CFIA Enhanced BSE Initiative has been successful in responding to the needs of the principal stakeholders and has done so in a manner that is intensive, focused and transparent. This statement is based on the perception of internal and external key informants as to the outcomes identified in the draft 2007 consolidated RMAF, ²⁰ shown in Table 2, and other evidence presented below.

External informants had a very similar perception of the relative success of all outcomes as internal informants, although they tended to give a lower absolute rating to outcomes relating to slaughter capacity and expanding export markets. Note that the ongoing successes of the Enhanced BSE Initiative can also be attributed in part to measures that were put in place by the CFIA before 2003, including routine surveillance, measures taken to trace and eliminate animals imported from the UK, and the feed ban implemented in 1997. Indeed, as explained by key internal informants, these measures minimized the size of the BSE episode in Canada and provided a solid structure on which to develop the enhanced activities (see also Governance, Section 4.2).

Accurate Identification and Prompt Containment of BSE Incidents

The CFIA’s involvement in accurately identifying BSE cases and containing BSE incidents was by and large highly praised by both internal and external informants despite the obstacles (e.g., record limitations, an imperfect cattle ID system and the size and scope of investigations), as investigations were described as fast, thorough, intense and excellent. A measure of this success is that none of the 14 BSE incidents that have occurred to date involved contamination of the animal or human food chain. However, according to the investigation reports of the first 13 cases, some cohorts were not fully traced, and the investigations were hampered by significant challenges,²¹ so there is still room for improvement.

It should be noted that most (19 out of 27) internal informants declined to rate the outcome "Decline in the incidence of BSE",²² stating that the CFIA is on track, but that it is too early to determine the success of the outcome. This is because this outcome is linked to the Enhanced Feed Ban, which was implemented in July 2007; the surveillance component will thus enable the CFIA to assess the effectiveness of SRM removal from animal feed in the eventual eradication of the disease in Canada in the long-term (e.g., in 7-10 years). Until then, the CFIA expects to continue to find a small number of cases of BSE in the domestic herd. As such, this outcome was not included in the table below.

Table 2 - Perception of internal key informants and external key informants on the success of the Enhanced BSE Initiative’s outcomes (as identified in the 2007 RMAF)

Click on image for larger view

Facilitating the Expansion of the Domestic Slaughter Capacity

Both internal and external interviewees had mixed perceptions with respect to this outcome, although generally the CFIA was considered to have adequately fulfilled its specific role of increasing line capacity and streamlining the review and approvals process in this program led by Agriculture and Agri-Food Canada (AAFC). External informants were more vocal than internal informants about the fact that the existing and expanded domestic slaughter capacity is currently not being fully utilized, but most agreed with internal informants that this was attributable to unfavourable market forces rather than the measures taken by the CFIA.

Increased Canadian Consumer Confidence

Increased Canadian consumer confidence is one of the main successes of the initiative. Indeed, as highlighted by nearly all informants (internal and external), confidence in the Canadian food supply has remained high throughout the evaluation period. If beef consumption is taken as a proxy measure of consumer confidence, then it is significant that, according to the indicators collected by Statistics Canada, beef consumption actually increased slightly in 2003, following the first Canadian BSE case (Figure 1), whereas beef consumption in most countries (e.g., France, Germany and Japan) dropped significantly following the discovery of BSE in the national herd.²³

Click on image for larger view

The latest consumer confidence survey, conducted for the CFIA in May 2005, showed that 92% of Canadians believed eating Canadian beef was safe.²⁴ Moreover, the proportion of people who believe Canada’s food safety system is among the best in the world had leapt from 47% in January of 2004 to 82% in May 2005. Trust and confidence in the CFIA’s handling of food safety and BSE (treated as two separate questions) increased from 61-64% in 2004 to 85-88% over the same period, while 83% reported in 2005 that they felt the government had done a good job of keeping them informed of BSE, up from 70% in 2004. A more recent survey (2007) on the safety of imported food suggests that confidence in Canada’s food supply has dropped slightly (with 60% of Canadians being more than moderately confident) but that BSE is now low on the list of issues that concern Canadians with regard to food safety.

This success was in large part seen as the result of the CFIA’s high-quality, transparent and timely communications directed to the Canadian public (see also Section 4.3.1). In fact, the effectiveness of the CFIA’s communication, education and information dissemination efforts in responding to the needs of the public may even have been the cause of the fact that these efforts are no longer in as high demand (compared to the demand following the initial discovery of BSE in Canada in 2003). As shown in Figure 2, the frequency of BSE-related content in Canadian newspapers has dropped dramatically since 2003. Moreover, as reported by the CFIA’s Public Affairs Branch, the discovery of new cases of BSE no longer produces a high volume of calls and emails from the public and the media to the CFIA, nor does it make front-page news. This suggests that the CFIA’s efforts have succeeded in increasing the level of confidence of stakeholders in Canada’s food safety system and reduced their need for additional information (see also Section 4.3.1).

Click on image for larger view

Increased International Consumer Confidence/Restored Access to International Markets/Opened New Markets

Success in restoring or opening international beef and cattle markets was not rated as highly as that for other program components, despite the rapid recovery of certain markets (including the partial re-opening of the US border to beef in 2003). This may be explained by the fact that world exports of Canadian cattle and beef remain below 2002 levels, as shown in Figure 3. While export market recovery is still not complete, a considerable number of markets have been partially or fully re-opened to Canadian cattle, beef, animal products, and bovine embryos/semen (see Appendix D). The CFIA’s contribution to the re-opening of these markets includes hosting foreign review missions, having its technical veterinarians/experts abroad to inform trade negotiations that were led jointly the CFIA, AAFC and DFAIT, and maintaining transparency regarding its science-based BSE-related activities. Alberta was also reported to be a stand-out for market recovery activities, hosting a high number of foreign delegations because of a large majority of Canadian cattle is slaughtered in and exported from western Canada.²⁵ The re-opening of these markets is also a proxy indicator of increased international consumer confidence, which was seen by key informants as a relatively successful outcome of the CFIA’s Enhanced BSE Initiative (internal and external informants, Table 2).

As perceived by internal informants, over half of external informants (55%) from both government and industry stated that they were not satisfied with one or more of the following: a) the achievements thus far with regard to trade-related outcomes; 2) the speed at which these were achieved, 3) the amount of resources dedicated to these outcomes, and; 4) the way in which export market recovery efforts have been managed more generally. Internal and external informants were well aware of the political barriers to restoring market access and highly praised the CFIA for establishing and demonstrating its BSE control measures to the international community (including the OIE and national regulatory bodies). However, the role played by the CFIA’s technical experts abroad was never noted by external informants, and the latter expressed confusion and a generally critical opinion regarding the CFIA’s coordinated efforts with AAFC and DFAIT; this key point is discussed at greater length in Section 4.3.2.

Click on image for larger view

Overall Recognition of Canadian BSE Management by National/International Authorities

Another key successful outcome of the Enhanced BSE Initiative is the strong level of recognition from international authorities of the science-based management of BSE in Canada. This is exemplified by the controlled risk status granted by the OIE in May 2007 as this status is based on a set of indicators,²⁶ which mainly rest on assessing risk factors (including the level of control of SRM in feed) and on targeted surveillance data. However, maintaining this status is conditional upon a yearly review of the epidemiological situation for BSE in Canada and on documentation showing the continued observance of OIE standards (see also Section 4.5). Note that the CFIA’s openness and transparency is also seen by both internal and external informants as a key contributing factor to maintaining a high level of recognition from international authorities.

Inter-Regional Differences

Differences in outcomes between regions must be discussed in the context of the cattle industries in different areas, listed here in the order of increasing size of their total cattle herd: Atlantic, Quebec, Ontario and Western Canada. Alberta has the largest herd, with close to 40% of Canada’s cattle (as of January 2008).²⁷Thus, outcomes of the Enhanced BSE Initiative are likely to be more important in the Western provinces, particularly as all of the BSE cases also occurred in this area. This difference also has implications for many components of the initiative; the need for increased slaughter capacity, for example, is much smaller in the Atlantic region, which has a herd that is a mere fraction (5%) the size of Alberta’s and which exports very little beef or cattle. Nevertheless, when taking scale into account, surprisingly few differences in the perceived success of outcomes were noted between regions or when comparing the regions to the national level.

Inter-regional differences mainly concerned cattle ID and tracking, and SRM removal. In particular, Quebec’s traceability activities were generally seen by all regions as the best in Canada, due to the strengths of Agri-Traçabilité Québec (ATQ), although Alberta’s efforts were also praised by internal informants. Verifying SRM removal was also seen to be influenced by regional differences because the provinces are responsible for inspections in provincial plants (as opposed to federally registered plants that are inspected by the CFIA), and the resources available for these inspections vary greatly per region. The Atlantic region has fewer resources and infrastructure than do Quebec, Ontario, Alberta and British Columbia and therefore experienced certain unique difficulties in terms of adjusting to the Enhanced BSE Initiative; for example, they have had less previous experience and fewer inspectors to verify that the enhanced BSE measures are applied across their territories.

Unexpected Outcomes

Various unexpected outcomes (both positive and negative) have also resulted at both the provincial and national levels. Key lessons have been learned regarding the development of new guidelines, for example in adjusting guidelines based on the result of targeted science-based studies or projects. Similarly, the need and importance of quality systems has emerged from the implementation of new practices and guidelines, particularly for testing and cattle ID components that are being conducted by provincial or external organizations. Finally, surprise was frequently expressed by both internal and external informants regarding the time it takes to implement new measures and see the results (i.e., can be slower than expected, such as for re-opening or expanding export markets) and the need to manage expectations.

Overall, many informants agreed that BSE has enhanced a number of aspects within the CFIA. This is to be expected as BSE activities were designed to be integrated within the day-to-day work of the agency to improve the delivery of both BSE and non-BSE activities (see also Section 4.3). For example, BSE-related FTE recruitment has provided the CFIA with "a bigger mass of expertise and competency," and because most resources do not work exclusively on BSE, this has increased the CFIA’s capacity to conduct other activities as well (e.g., increased inspection capacity; identification of partners, networks, and potential collaborators; and foresight and science intelligence groups for zoonotic diseases and emerging diseases, including other TSEs and avian influenza). Thus, a spillover benefit of the BSE funding can be seen in terms of greater impulsion for other programs and a subsequently greater capacity to manage other health safety risks, particularly other foreign animal diseases: "We established a way to do business through BSE, and it’s carried over into other things."

This increase in capacity is also linked with the stronger working relationships that have been developed with provincial and industry partners; five out of the ten internal informants who identified unexpected outcomes reported that BSE spearheaded better cooperation and communication between the CFIA and external stakeholders within the federal government, the provinces, and industry. This aspect was most often highlighted in the case of the national laboratory network, which includes several provincial installations that perform surveillance testing. Internal informants also often spoke of "maintaining stakeholders in the file" and "engaging our partners" outside of the CFIA, again pointing out the importance of transparency and of "building that relationship and that trust and that understanding of what drives them and their understanding of what drives you." Thus, BSE is seen as having not only increased but enhanced the CFIA’s overall communication capacity by changing the way the CFIA shares information with its stakeholders. One possible exception raised by external informants is the way in which the CFIA collaborates with some other government departments on shared responsibilities, particularly with regard to expanding export markets (see discussion in Section 4.3.2).

Long-Term Outcomes

Taking all of these successful outcomes into account, it becomes apparent that BSE has provided a channel to a more integrated approach to risk management in the human food supply for other diseases; for example, tracking and tracing can address not only BSE but also other diseases. This is an important achievement in light of the CFIA’s three strategic outcomes (see Logic Model, Appendix B). In particular, ensuring the sustainability of the animal resource base is an outcome that extends beyond BSE, involves external factors outside of CFIA control or jurisdiction (i.e., domestic slaughter capacity), and includes aspects (i.e., traceability) that are still in development. Specifically regarding BSE, the CFIA’s level of success for this outcome is generally seen as good and is expected to improve in the coming years as a result of the Enhanced Feed Ban (see also Section 4.6 for a discussion of how the nature of BSE affects risk management of this disease, and the implications of this with regard to long-term outcomes.)

4.2  Governance

The CFIA is one of the only regulators in the world (together with the Ministère de l'Agriculture, des Pêcheries et de l'Alimentation du Québec [MAPAQ]) that oversees the entire food chain, from the farm to the plate, under a single authority. The CFIA’s governance and delivery of the Enhanced BSE Initiative is integrated within its existing management and operational structure. The backbone of this structure is a series of branches led by Vice Presidents or Executive Directors; these branches include Policy and Programs, Operations, Science, and Public Affairs, among others. As the Enhanced BSE Initiative draws on all of these branches, one can speak of a matrix management approach, where governance of relevant activities is not only vertical but horizontal (i.e., across branches). This approach has demonstrated its strengths; however, in a recent evaluation of related CFIA programs, accountability and governance (i.e., clearly defined roles and responsibilities) were recognized to be an issue,²⁸ which is corroborated by the insights of internal informants (see below).

In the case of the Enhanced BSE Initiative, the programming was well-tailored to build on existing management structures and strengths, which reduced the learning curve in implementing the activities of the initiative. For example, all of the additional policy, programs, science and operations field work involved was well-integrated and grafted onto previously existing programming activities. Moreover, almost all internal informants reported that the current management framework was adequate for delivering BSE-related activities; more specifically, they did not see a need for a separate management structure for BSE.

However, this matrix management approach necessitates effective controls and communication tools to guarantee the ownership of each part of the BSE initiative and to ensure that roles and responsibilities are well understood across the organization and over time. More specifically, without proper governance, the vertical reporting relationships may hinder the ability of the CFIA to manage laterally. At the onset of the BSE crisis in 2003, governance was provided by a dedicated BSE Task Force (also referred to as the "working group"), which had the capacity to act in an authoritative manner and which had a clearly defined chain of command. However, the Task Force was subsequently dismantled, and management of the BSE initiative was moved back into the CFIA’s regular management structure at the end of 2004. A BSE coordinating committee was put in place from 2004 to 2006, but there has since been no systematic coordinated management of the Enhanced BSE Initiative at the CFIA. Essentially, when the Task Force was dismantled, internal communication deficiencies and uncertainty about the chain of command with regard to BSE began to increase.

This conclusion is supported by evidence from the internal interviews, which suggests that the roles and responsibilities are fairly clear and well understood within each branch or unit (i.e., vertically), but not at the lateral or organizational level. For example, many internal interviewees appeared confused as to who led or managed various BSE files (such as with regard to the complementary roles of the CFIA and AAFC) outside of their branch. Others cited problems with communication and governance between the National Capital Region and some of the regional operations and laboratories. For instance, some internal informants noted instances wherein themselves or other individuals working on BSE in regional area offices or in different branches were disconnected or underutilized within the organization, meaning that they were not involved as intended within decisions and program implementations-this was said to be an issue particularly in the case of recently hired FTEs, whose managers may not have known they should have been contacted.

An important realignment took place in January of 2008 that will likely address some of the issues caused by the horizontal governance structure. Policy activities have been merged with the Programs Branch to bring policy capacity within the programs. In the past, policy functions and governance were fragmented among different organizational structures; now they are consolidated under one branch with a common management structure. It is expected that this merger will facilitate the coordination and communication around BSE between policy and programs. It will also likely help the CFIA to be more effective at identifying important issues and future needs for BSE management. A remaining challenge of this matrix management approach therefore likely resides in the coordination between the Policy and Programs Branch and the Operations Branch. The systematisation of exchanges between program network staff, the National Operations Directorate, and regional operations is an ongoing issue (see also Section 4.3.1).

Thus, although the present structure is generally working as intended, the evidence points to a continued need for a dedicated BSE coordinating structure, that would improve the effectiveness of the initiative. This is different from a separate management structure in that the coordinating structure acts as a bridge that connects BSE-related actors in different CFIA branches and regional offices to enable the sharing of necessary information and expertise between those making BSE-related management decisions; BSE-related data collection could also be coordinated through this structure (see also Section 4.5). This is justified by the fact that internal informants noted that the CFIA’s core BSE expertise resides in a small number of individuals spread across the CFIA’s units. A dedicated BSE structure could provide a framework within which these individuals could coordinate, combine, and oversee the transfer of this expertise, including tacit knowledge as well as Canadian and international network contacts (such as those in multiple partner organizations and stakeholder groups involved in the initiative). This structure would thus help ensure that all persons associated with the BSE file are kept up to date on BSE measures within the CFIA and help facilitate succession and maintenance of a critical mass of BSE expertise within the CFIA (see Section 4.3.3).

The need for a dedicated coordination structure for BSE should be balanced by the fact that the CFIA is responsible for the surveillance and control of diseases besides BSE, as well as for several other aspects of food safety. Through the current management approach, BSE has already become increasingly integrated within non-BSE programming, such as surveillance, trade negotiations, and tracking and traceability. For example, the veterinarians posted abroad as part of the BSE initiative are increasingly solicited for information on other commodities and for outreach/communication on other areas of the CFIA’s programming. As such, several internal informants strongly believed that BSE should continue to be integrated within other routine CFIA activities. Thus, a dedicated BSE coordination structure should also act to identify areas in which BSE activities could be integrated into broader animal health-, country- or commodity-based strategies, such as for the surveillance, tracking and tracing, and trade examples noted above.

4.3 Design and Delivery

Design and delivery of the Enhanced BSE Initiative was complex as it involved not only internal coordination between multiple branches and programs, but also a high level of communication and coordination with partner organizations (including other federal departments and provincial governments), as well as with various external stakeholders and the general public. Some of these groups had conflicting interests and priorities, not to mention different levels of understanding of the nature of BSE and the implications for risk management of this disease (see also discussion in Section 4.6). Internal and external factors that were found to have influenced the performance of the program, both positively and negatively (see below), included communication and coordination issues, and so the following section discusses to what extent communication and collaboration with partner organizations and stakeholders were adequately integrated in the design and delivery of the initiative. Also, implications of the design and delivery of the initiative with regard to Human Resources management and BSE expertise within the CFIA are discussed.

Factors Influencing the Performance of the Program

Key internal and external informants pinpointed several major factors that, in their perception, influenced the delivery and performance of the Enhanced BSE Initiative. The main positive factors internal to the CFIA were very clearly perceived to be the funds allocated to the initiative and the high level of expertise, management, and commitment exhibited by CFIA staff (see also Governance, Section 4.2 and Section 4.3.3). The perception was that overly stretched financial and human resources,²⁹ as well as certain management issues (particularly with regard to performance data management, see Section 4.5) and coordination issues (particularly with certain partners and stakeholders, see Section 4.3.2) were the main negative internal factors.

Positive factors external to the CFIA but within Canada are the strong communication activities with stakeholders and the public (leading to a high level of consumer confidence, see Sections 4.1.2 and 4.3.1), coupled with strong participation of partners and stakeholders (see Section 4.3.2). At the international level, recognition of Canada’s BSE control measures was seen as the main positive factor, mainly through the controlled risk status granted by the OIE, based on international science-based standards. Conversely, the fact that some markets were often slow or difficult to open, either because of protectionist measures or for other political reasons, was most often brought forward as a negative factor at the international level.

4.3.1 Communication and Outreach Activities

Communication with External Stakeholders

Internal informants were very positive with respect to the timely and effective dissemination of information on BSE to external stakeholders, including the Canadian public and industry. The key public communication tool is the CFIA’s website on BSE, which is information-rich and up to date. For each Canadian BSE case, the CFIA also publishes a full epidemiological report, which is distributed to a broad audience via email subscription.

In addition to the website, a 1-800 line is available to the public 8 am to 8 pm daily.  Dedicated communication efforts with industry cover the entire value-chain and include developing outreach material (posters, leaflets, and publications in the trade literature) to inform stakeholders of various aspects of the Enhanced BSE Initiative.

The large majority of external informants rated the CFIA’s communication efforts at a high or very high level in terms of quality, effectiveness, and timeliness, primarily with regard to communication activities with the public and individual stakeholders (Table 3). Almost all external informants reported using the CFIA as a primary source of information on BSE; close to half specifically mentioned the use of their own BSE website (entitled "BSE in North America"). However, several interviewees expressed a worry that overall communication efforts with the public and Canadian and external stakeholders has started to slip since the worst of the BSE crisis has abated and that stakeholders who are still concerned about BSE may not be receiving the same quality of information as before (see also the discussion in Section 4.1.2).

Table 3 - Perceptions of external key informants on the success of the Enhanced BSE Initiative’s communication and education activities

Click on image for larger view

When speaking of the CFIA’s communication with their own organization (provincial department or industry association), however, less than half of the external informants were satisfied. In fact, "better communication" was one of the most frequent answers to questions asking how the CFIA could better respond to their organization’s needs. A number of clear examples were reported in which external informants had difficulty obtaining the information they needed or learned of CFIA decisions that directly concerned them after the fact or from other sources (e.g., from the media or their clients). This was especially true for respondents in other government departments (both federal and provincial); Quebec is an exception, as a high level of satisfaction was reported by all informants from this province.³⁰Otherwise, there were no clear trends in terms of which regions or industries reported communication problems.

Communication with International Stakeholders

Overall, the effectiveness of dissemination of BSE information to international stakeholders was also ranked as high. In collaboration with DFAIT, the CFIA sent messages virtually on a daily basis to all Canadian embassies and regularly visited and spoke with relevant authorities in other countries. Veterinarians and BSE experts posted abroad also contributed to answering and clarifying the status of investigations of new BSE cases detected in Canada. According to a number of interviewees (both internal and external), the international community has complimented Canada on its openness and the transparency of information disseminated on BSE, and Canada compares favourably on this point to other BSE-infected countries. The perception within the CFIA is that Canada’s open and transparent management of BSE has facilitated the engagement of certain countries in trade discussions. While most external informants also recognized this, over one-third (11 out of 27) remained critical of the way in which activities to expand export markets are being managed (see also Section 4.3.2).

Internal communication

Paradoxically, observations on how information was disseminated within the CFIA were less positive, and internal communication shortcomings were perceived to have worsened since the profile of BSE started to decline within the organization and since the BSE Task Force was disassembled in 2004. A frequently noted example cited among internal informants (7 out of 16 answers) mentions the effectiveness and timeliness of communication on the latest domestic BSE cases; these informants, including some who worked to expand export markets, said they were first informed of the existence of new BSE cases (especially the most recent cases) through the media or from industry partners. This represents a serious challenge for those who have a critical role communicating this type of information to international trading partners. Internal communications between Policy and Programs in the National Capital Region and the Operations and Programs staff outside of this region have also been identified as weak links with respect to BSE. In sum, among the internal factors that should be addressed to improve the effectiveness of the Enhanced BSE Initiative, internal communication and coordination rank first and foremost.

4.3.2 Collaboration with Partner Organizations and Stakeholders

Collaboration with Federal Partner Organizations

The CFIA’s role in programs and activities led by federal partner organizations was adequately integrated in the design and delivery of the initiative’s components, primarily because the CFIA’s contributions were directly linked to their clearly established mandate of surveillance, inspection, and regulation of issues relating to food safety. Key collaborative activities were seen with: 1) AAFC on tracking and tracing, as well as in facilitating domestic slaughter capacity and expanding export markets; 2) Health Canada for removal of SRM from the human food supply and; 3) DFAIT on expanding export markets. Thus, in working with all these federal partners, the CFIA’s role was to provide regulatory, enforcement, or technical expertise as needed.

Internal informants generally perceived their collaboration with federal partner organizations to be very good. Unfortunately, it was not possible to survey more than two informants from these federal partner organizations, but the available information suggests that the CFIA collaboration with these partners may occasionally suffer from certain communication and coordination problems.

Collaboration with Provincial Partner Organizations

The CFIA’s provincial partner government organizations include departments dealing with agriculture and rural development, as well as provincial testing laboratories for enhanced surveillance and testing activities. For all components of the initiative, provincial involvement and cooperation with the CFIA was almost always described as "a close relationship" or "good" by internal informants, particularly for Alberta and Quebec. For example, all respondents from Quebec (government and industry) were satisfied with their relationship with the CFIA. Moreover, those informants from across the country who were less involved with international trade were generally more positive than those who were very involved.

However, some provincial government informants reported confusion about roles and directives with regard to the Enhanced BSE Initiative, problems with communication and coordination of national and regional activities, and gaps in the way that certain components were handled. Informants identified some of the causes of these problems, such as a lack of resources at the CFIA (including technical expertise spread too thinly over the territory), frequent staff turnover in some regions, and issues with performance data management. Moreover, most informants recommended that activities from at least one component should be transferred either to the provinces or to the AAFC. Suggestions as to what should be transferred were varied, but the most frequent recommendations were that export market activities should be handed over to the AAFC (4 out of 9) or that BSE surveillance could be overseen by the provinces (2 out of 9).

Collaboration with Industry and Non-Governmental Organizations

The CFIA works directly with several industry and non-governmental organizations; for example, it is a member of the Industry-Government Advisory Committee (IGAC) on traceability for livestock and poultry, which has thus far made "significant progress towards agreement on national standards and performance targets for livestock traceability," including cattle. Again, views on the CFIA’s collaboration efforts were mixed, with several industry and non-governmental organizations agreeing that their relationship with the CFIA was good, but others remained critical. For example, 7 out of 18 noted significant overlaps with CFIA activities and found fault with the "coordination, communication, and collaboration between the various agencies that are working in this important area."

Note that although informants were instructed not to discuss aspects relating to the Enhanced Feed Ban, this program may have influenced some general comments made by external informants who were most familiar with it. The Enhanced Feed Ban also provoked some of the most negative comments from external informants when they were asked about the CFIA’s communication and collaboration efforts related to BSE.

Expanding Export Markets

The majority of the informants (internal and external) thought that the CFIA’s role in expanding export markets was either poorly understood or poorly coordinated with those of DFAIT, AAFC, and the provinces. These organizations are seen to be "working in two worlds" and are "siloed," the result being that efforts to expand export markets would "freeze" or were slowed by "the usual run-around between three departments." When looking at evidence from all external informants (government and industry), 11 out of 27 identified one or more gaps, overlaps, or potentially transferable activities involving the export market component. Conversely, the CFIA representatives interviewed repeatedly highlighted the unrealistic expectations of industry with respect to the CFIA’s role in expanding export markets. However, there was a strong view among external partners and stakeholders that changes in the management of this component should be made, such as establishing a trade directorate (which could include better coordination with and greater involvement of industry) or developing a more clearly defined and centralized management structure to coordinate these trade-related activities. External informants say that if export market activities could be coordinated under one roof or in one jurisdiction, the situation would improve through having a "one stop shop" with which to deal with on export-related issues.

Even within the CFIA, there were some marked differences in the perceptions of internal informants about the roles and responsibilities of the CFIA regarding the expansion of export markets. While some internal informants argued that the CFIA should have done more to support the re-opening of export markets, others believed that, as per the CFIA’s mandate, its role was rightfully limited to supporting the trade policy initiatives led by other organizations (e.g., DFAIT and AAFC) through demonstrating the credibility of its BSE surveillance control measures and providing clear communication and expertise on scientific and technical aspects of BSE. This confusion reflects that expressed by external informants regarding the CFIA’s dual role as both a regulatory body and a supporter of the Canadian beef and cattle industries.

Overall collaboration

In conclusion, there were no significant instances of gaps or major overlaps between the CFIA’s activities and those of other government organizations or external stakeholders at the level of the entire Enhanced BSE Initiative. In fact, both internal and external informants commonly described the CFIA’s relationships with partner organizations as "complementary." As for transferring non-trade activities, such as surveillance and testing, to the private sector or to the provinces, the mindset within the CFIA at the time of the 2003 crisis was that providing these measures was crucial to maintaining consumer confidence, although this appears to be changing. The consensus of informants in all sectors (the CFIA, government and industry), however, is that even if some activities could be transferred out (see above), the CFIA has a crucial regulatory, inspection, and enforcement mandate, that this mandate is relevant to Canada as a whole, and that the role played by the CFIA is key in maintaining the credibility and reputation of Canada’s food safety system. This widely-held view lends strong support to the conclusion that there is a legitimate need for government to be involved in BSE programming and, more specifically, that the role currently held by the CFIA should continue to be held by the federal government and delivered in partnership with different regions.

4.3.3 Human Resources and BSE Expertise

The Human Resources Branch received specific funding from the BSE initiative to support corporate overhead (i.e., for hiring FTE personnel) and training activities through the Learning Division, more specifically through the Professional and Technical Development (PTD) section.

BSE Training

The PTD had BSE-related training in place prior to 2003 and has since developed further training materials that specifically address BSE, often as part of core BSE courses (Surveillance of TSEs, Animal Health SRM, etc.) or as modules of courses of a broader scope, such as the yearly foreign animal disease course. These materials cover, for example, BSE surveillance—brain tissue collection; verification of SRM removal; procedures for licensing and registering meat facilities; feed ban regulations/BSE removal; and export certification of animal products. The Learning Division coordinates new training needs with the Operations and Programs & Policy Branches, with input from the Science Branch and CFIA headquarters, using BSE funds for BSE-related training initiatives. All of the resulting courses are evaluated by participants for monitoring and to inform course updates; even if these courses did not deal exclusively with BSE, these evaluations show good levels of satisfaction and include a number of constructive comments.

Nevertheless, several informants expressed concern regarding the maintenance of the CFIA’s BSE-related scientific expertise at the current level. Within the CFIA, research results and other science-based knowledge on BSE are generally communicated through technical reports, presentations, and teaching and training. It was therefore suggested by internal informants that efforts be made to facilitate the mobility of researchers, laboratory pathologists, and BSE specialists to improve teaching and training (i.e., face-to-face knowledge transfer and sharing of internal expertise) as well as to bring back more "front-end" science (e.g., from technical and scientific conferences in Canada and internationally). The issue of BSE expertise should be addressed as part of the CFIA’s succession and renewal plan, discussed below.

Human Resources for BSE

As mentioned in Section 4.1.1, FTEs supported by BSE funding are integrated within the day-to-day operations at the CFIA (including, in many cases, with non-BSE related activities), it is only possible to indicate the total number of FTEs employed with BSE funds, as shown in Table 1. No precise review is available to track how FTEs were utilized within branches or to estimate the number of FTEs responsible for performing activities within Enhanced BSE Initiative components. Nevertheless, BSE funding enabled the CFIA to increase its mass of expertise and competencies by hiring and training highly qualified scientists and technicians. Not all of these were initially BSE experts; they were hired for their ability to adapt and focus their work along with changing organizational priorities, and thus now represent increased capacity for the CFIA as a whole, as well as a growth in BSE expertise.

Significant and recent changes impacting the delivery of the BSE file—mainly retirement, but also promotions/re-assignments, changes within the management structure of the organization, or other changes—were reported by at least half of the internal informants, as well as by external informants who are in direct contact with CFIA representatives. The most affected areas appear to be the Science Branch and the tracking and tracing component. As many near retirement eligibility, especially at the senior management level, further changes are expected, and attraction and retention of professionals constitute important challenges in securing a critical mass of BSE expertise; this reality also fits into the larger HR challenges faced by the CFIA, which are addressed in the agency’s Renewal Plan 2008-2013.

Over time, BSE experts gain tacit knowledge of Canadian and international stakeholders, and this knowledge is instrumental for the delivery and the success of BSE management in Canada. As such, renewing professional relationships and networks in Canada and abroad is a critical succession issue for the CFIA. Thus, succession planning should continue to include a strong commitment to "attracting talent",³¹ with an extended transition period so that new staff can be mentored, trained and integrated into BSE networks by existing BSE specialists across the organization. This will facilitate the transfer of essential knowledge of partners and stakeholders and awareness of ongoing BSE initiatives led by external organizations, both Canadian and international.

4.4 Relevance and Continued Need

The evidence examined for this evaluation reveals that the Enhanced BSE Initiative has been closely aligned with many of the Government of Canada’s priorities from which the CFIA draws its mandate, including: bolstering economic prosperity, strengthening the safety of the food supply, and contributing to the health of Canadians.³² As a science-based regulator, the CFIA is responsible for administering and enforcing several Acts,³³such as the Health of Animals Act, the Meat Inspection Act, the Feeds Act, and the Fertilizers Act, some of which have been amended to strengthen Canada’s safeguards against BSE. Thus, the mandate for the Enhanced BSE Initiative is directly derived from this federal legislation.

The Enhanced BSE Initiative has also contributed to the CFIA fulfilling its strategic objectives, as discussed in Section 4.1.2 (under "Long-term Outcomes"):

• minimized and managed public health risks associated with the food supply and transmission of animal disease to humans;
• contributed to a safe and sustainable plant and animal resource base;
• contributed to consumer protection and market access based on the application of science and standards.

As will be discussed in greater detail below, the Enhanced BSE Initiative has met the needs of its two principal beneficiaries:

• The health of the Canadian public was protected through sustained/increased food safety, and confidence in Canadian food safety was maintained by the measures taken by the CFIA to respond to BSE.
• The Canadian beef and cattle industries (including producers, processors, renderers, distributors, and exporters) benefits from a strong, albeit incomplete, market recovery, as well as from multifaceted efforts to help them adopt, adapt to, and understand the benefits of the new food safety regulations. The CFIA has also played a key role in meeting Canada's international reporting obligations on BSE, particularly with regard to the OIE, which have contributed to market recovery. However, the Canadian beef and cattle industries are not completely satisfied with the present results of the CFIA's BSE programs, in part due to additional costs and loss of competitiveness incurred by increased regulations (some of which are stricter than US regulations) and by the continued dependency on export markets, the latter being largely outside the CFIA's control.

This initiative has served and continues to serve the Canadian public interest, as well as industry needs. Indeed, the large majority of informants from both government and industry agreed with internal informants that the initiative overall had benefited their stakeholders and had responded to many of their needs. Furthermore, the relevance and continued need of the initiative overall and of its individual components were nearly all judged to be high by internal and external informants. In fact, as shown in Table 4, only the expansion of domestic slaughter capacity is seen as having lost its relevance. Note that overall and in the discussion below, the perception of external informants was almost identical to that of internal informants, with nearly 95% of external informants seeing a continued need for the Enhanced BSE

In particular, agreement was nearly unanimous regarding the very high need for tracking, tracing, and enforcement; removal of SRM from the human food supply; and expanding export markets. This first component has not yet met its stated objectives, but is seen by many as a fundamental support element for BSE and several other animal diseases. The CFIA should thus continue to work with both government and industry partners to accelerate the implementation of the traceability system (including date of birth registration) and to enforce the cattle ID program. This will also support faster traceback investigations of future BSE cases. Removal of SRM underpins all other BSE control efforts, as it ensures a high level of protection for human health and is "critical to the credibility of our meat system"; as such, no informants suggested reducing or eliminating this component.

Most interviewees (internal and external) also agreed on the continued need for expanding export markets for at least five more years, "given that a huge volume of the industry is exported" and that full market recovery has not yet been achieved. Efforts should also be made to diversify the beef and cattle export markets to reduce Canada’s current dependency on the US. Finally, export certification needs to continue at a similar or enhanced level due to the more stringent requirements of some countries, such as Japan.

For surveillance activities, the relevance for the activity remains high, but the design and/or delivery should be modified to reflect the most recent science- and evidence-based knowledge. In particular, more detailed data is needed on surveillance samples to confirm whether these were collected from animals displaying clinical symptoms consistent with BSE or from animals belonging to the so-called 4-D groups (dead, dying, diseased and downers). Surveillance activities must continue to both meet OIE requirements and assess the effectiveness of the feed ban, and the available evidence suggests a need to support surveillance beyond cattle. There are, however, widely dissenting viewpoints about the continued need for the number of samples required and, consequently, for the BSE Reimbursement Program, particularly with regard to the quality/representativeness of the sampling (see also Section 4.6 ).

Although the activities to facilitate the expansion of the domestic slaughter capacity were relevant at the height of the BSE crisis in 2003-2004, the need for domestic slaughter capacity has fallen following the re-opening of the US border, and the number of domestic slaughter establishments and/or their capacity has subsequently receded. According to interviewees, this is mainly due to the absence of profitability of slaughtering activities performed in Canada. While reducing dependency on the US for slaughter capacity generally remains relevant, the continued need for CFIA involvement in enhancing the domestic slaughter capacity is no longer justified, according to more than half of the internal informants and a number of external informants who answered this question.

Table 4 - Perceptions of internal key informants on the continued need for the Enhanced BSE Initiative, overall and for each component

Click on image for larger view

4.5 Performance Measurement and Reporting/Results-Based Management

Performance measurement and reporting/results-based management activities relating to the Enhanced BSE Initiative respond to several separate needs, including reporting for the TB (i.e., Departmental Performance Report [DPR], RMAF), reporting for the OIE, and internal performance management. For all of these needs, sizeable achievements have been observed, but important shortfalls currently hinder the efficiency and effectiveness of these activities. As performance measurement, management and evaluation also depend on appropriate financial tracking, issues relating to the tracking of BSE-related funds are discussed at the beginning of this section. In brief, evidence obtained during this evaluation and presented in this section suggests that:

1. the performance measurement strategy and monitoring plan has been partially implemented and delivered, but not without important challenges which are discussed below.
2. the implemented performance measurement activities have not been formalized and coordinated in a way that provides comprehensive and timely performance measures for effective result-based management and organizational reporting requirements.
3. the performance information which is collected is not effectively used in the planning and in the management of the Enhanced BSE Initiative.

However, the evaluation of this initiative suggests that, following recent increase in awareness within the CFIA with regard to the purpose and potential usefulness of performance measurement and management, the agency now has an opportunity to consolidate and expand its performance measurement capacity. This study should also act as a source of evidence and reflection for future development of this capacity.

Performance measurement and reporting, as well as results-based management, stand to benefit from good financial tracking. For instance, detailed analysis, such as cost-effectiveness, is not possible without appropriate financial data. As such, this section begins with a discussion of financial tracking of BSE activities within the CFIA, followed by evaluation findings on performance measurement and reporting for internal and external needs.

Financial Tracking

Because BSE activities are generally integrated within day-to-day CFIA activities and FTEs are rarely dedicated exclusively to BSE-related tasks, it is not possible to identify in detail the amounts used for individual sub-activities. An additional difficulty arose because funds and FTEs for several components of this initiative were not tracked with dedicated internal order codes and so the use of certain funds could not be accounted for in detail by the FAIT Branch (see note 2, Table 1).³⁴ For example, several cases were seen in which separate components or separate activities were all tracked under the same code because the funds for these came from the same TB submission. As separate codes were not systematically created for separate activities, there is no way to track or determine if or how funds were spend on certain proposed activities for which there is no evidence of the activity having taken place activities (see the paragraph below on "Strengthening Science & Support"). Moreover, several internal informants also described instances of activities supported by BSE funds across multiple branches or with external stakeholders, of branches using funds from multiple allocations, and of BSE activities being purposefully and justifiably integrated within the pre-existing or enhanced structures at the CFIA.

Within this context, financial tracking of BSE tied funding was frequently inadequate. In fact, senior internal informants confirmed the observation that tracking tied funding appears to represent a challenge for all CFIA tied funding.  The program’s structure and the fact that the CFIA is not effectively tracking tied funding thus makes it impossible in most cases to attribute specific outputs and outcomes to specific funds or specific FTEs,³⁵ or to assess the cost-effectiveness of specific programs (see also Section 4.6).

This problem is particularly evident for the "Strengthening Science & Support" sub-component, for which only 3% of the received funds have been reported as spent (accounting for a total of $534,300), according to Table 1. While there is clear evidence of science-based activities and advisory outputs (see Section 4.1.1), it is not at all clear how these outputs are related to activities proposed under this sub-component, particularly those called "Meat Inspection Reform" and "Quality Assurance Based Certification System" in the TB submission. For example, much work done by the laboratory operations of the Science Branch appears to fall under the enhanced surveillance and testing component (e.g., confirmations, proficiency panels for provincial labs, standard operating procedures, validate new tests, training provincial technicians); it is not clear whether the funding comes from the TB funds for this component or from those of "Strengthening Science and Support."

In contrast, nearly three quarters of managers at the branch level said they could demonstrate whether BSE funds were used for their intended purpose within their branch or area. Moreover, most senior managers interviewed said they did not know whether BSE funds had been allocated appropriately outside of their branch or area.  None raised specific concerns about whether BSE funds overall had been allocated and used as intended.

Internal Performance Measurement and Results-Based Management (including TB evaluation requirements)

The AERO Branch has overseen the production of two RMAFs for the initiative (followed by a draft consolidated RMAF prepared in 2007), which present both a performance measurement plan and an evaluation plan. In these plans, performance indicators were identified and defined and data were collected during the period under evaluation.³⁶

Almost all of these data are available and can be tallied and compiled in such a way as to verify and inform on the outputs of the activities falling under the initiative. Indeed, evidence indicated that some of these performance indicators, such as the number of samples tested, market recovery milestones, and the progression of foreign reviews, were reviewed on a regular basis during the BSE coordinating committee meetings between April 2005 and May 2006. The number of sample tests and the reports on the investigations of individual BSE cases were also made available on the CFIA’s BSE web pages. In several cases, this data may also be used for external reporting needs, as discussed in the following section.

Internal informants reported that the information and data are collected at the operation and program levels and are sometimes used to help manage field- or micro-level activities at the branch level. However, several senior and branch managers mentioned that they were not using performance information in their decision-making and management duties, and that these data were not driving or guiding the program at a higher level. In fact, there is no integrated performance measurement and reporting system on BSE. The senior managers, including those within the AERO Branch, indicated that the status of the performance measurement system and current use of performance data on BSE were unclear or simply unknown. Some of the interviewees, even those within the Senior Management Committee, were not even aware of the availability of performance indicators as defined in the initial performance measurement plan of the RMAF.

Finally, data collection and compilation, and performance reporting for the RMAF and for this summative evaluation were done primarily within the AERO Branch, whereas the TB’s Evaluation Policy indicates that there should be significant involvement of department/branch managers throughout this process.³⁷ As the point of the exercise is the use of performance indicators for management purposes, the CFIA should continue working towards conforming to the TB’s Evaluation Policy so that its performance measurement system meets the needs of managers at all levels of the agency. Nevertheless, it is the evaluators’ opinion that, on the basis of an environmental scan they recently performed,³⁸ the CFIA has responded to the recent requirements regarding performance measurement, reporting, and results-based management at a level that is consistent with that observed in some other science-based government organizations. However, a number of organizations are more advanced in terms of their performance management culture and the implantation of performance measures using an integrated and centralized approach/tools that are coordinated with strategic and financial planning.

External Reporting - Departmental Performance Report (DPR) and OIE Updates

The key driver behind the collection of performance data is therefore to fulfill external reporting requirements for two key needs: 1) the annual CFIA DPR submitted to the TB and, more recently, 2) the CFIA’s annual update to OIE for BSE country categorization. However, internal informants reported that the production of these two documents was resource-intensive and inefficient. More specifically, information can be requested as needed, but up-to-date or real-time performance measures are not always readily available to those preparing these reports.

Specific instances have been identified by internal respondents where the information that is currently collected is not complete/adequate or does not reflect changing needs. There are many possible reasons for this, involving both internal and external factors. First, performance indicator data are collected by different branches (the Policy and Programs Branch and the Operations Branch) and may be initially compiled and used at the branch level for distinct purposes for external reporting on BSE. Second, a number of indicators that reflect performance or compliance within the industry are not collected in a systematic or integrated way in both federally and provincially registered establishments (particularly for SRM removal) or between provinces. Third, some of the indicators are collected by different units (or partner organizations) year-to-year, which makes it difficult for users to know who is responsible at any given time. Forth, having so many separate data needs (i.e., internal performance reporting, DPR and OIE reports, and occasionally other reports) that examine different time-scales and in varying levels of detail create confusion and gaps when individuals are asked to provide data for each of these needs. Finally, there is evidence to suggest that the level of effort dedicated to performance measurement in general may have decreased in the last two years, leading to incomplete data collection or compilation for specific external reporting needs.³⁹

A significant amount of data capturing and harmonization is therefore required before the analysis and reporting of data can be completed properly. Difficulties also arise when the needs of the external organization (i.e., the OIE) change or increase over time. In short, there is no formalized and integrated performance measurement structure for the collection and reporting of data for the OIE update, and, to a slightly lesser extent, for the DPR.

In the case of OIE reporting, there is a clear need to formalize data collection. The OIE introduced a new framework to describe a country’s BSE status in 2005, after which the CFIA was required to submit a report for the evaluation of its BSE status. When the OIE’s Scientific Commission BSE Ad Hoc group recommended that Canada be designated a controlled BSE risk country in 2007, it also provided several comments for Canada to consider in maintaining its controlled risk status, including a requirement for detailed annual reports documenting the level of compliance achieved on the surveillance program and on feed ban compliance. Moreover, the OIE has asked for more detailed information than that captured to date by CFIA staff (e.g., for the DPR), so the submission cannot simply rely on the same data as used for the DPR. Even though all of these types of emerging needs have since been identified, there is no feedback mechanism currently in place within the CFIA to incorporate these changes in the data collection strategy. As a result, the collection of information for this annual update to the OIE involves the work of a tremendous number of people across the organization, each holding a part of the required information, some of which is not in electronic format.

Efforts are currently underway at the CFIA to better address data collection and reporting needs for the OIE submission. As a case in point, an internal CFIA document outlines a series of recommendations to ensure that the CFIA’s surveillance program (and the Enhanced Feed Ban) meets the standards prescribed by the OIE, including the data required for the annual update as well as the number and type of samples required for surveillances purposes; this issue is discussed at greater length in Section 4.6.⁴⁰

External Reporting - Other Needs

As discussed in Section 4.3, external stakeholders, such as representatives from trading countries or from other government departments, also request BSE-related data for which the CFIA is responsible. In these cases, important achievements (i.e., the performance-related information provided to external stakeholders by CFIA technical experts posted abroad, including during trade negotiations⁴¹) are balanced with specific instances in which the CFIA was not able to provide the necessary data or performance measurements.⁴² For example, informants from government departments working with the CFIA on BSE issues have also reported difficulties in obtaining the data they need from the CFIA. Note that some of the difficulties encountered by the CFIA in this regard relate to the changing requirements of countries that import Canadian beef and cattle, as well as those of partner organizations.

Overall Performance Measurement and Reporting/Results-Based Management

All of the above indicate a lack of performance measurement and performance management culture at the CFIA, both at the senior and program management levels, wherein performance measurement data is collected to satisfy essential reporting requirements but said data is not used to support monitoring and decision-making (i.e. for management purposes) within the agency. Furthermore, internal and external reporting needs are currently addressed in an unstructured manner and would greatly benefit from being formalized and integrated, so as to promote efficient and effective data collection. There is some evidence that such a process has begun within the CFIA; see Recommendation 2 for a detailed list of proposed measures to help guide these efforts. Overall, the CFIA is now in a good position to develop and strengthen its performance measurement and results-based management activities based on the lessons learned from this evaluation.

4.6 Cost-Effectiveness and Alternatives

Cost-Effectiveness of the Enhanced BSE Initiative

Several factors in this evaluation have made assessing the cost-effectiveness of the Enhanced BSE Initiative a challenge, especially due to attribution issues stemming from the complexity of the initiative (i.e., the matrix management structure, multi-stakeholder participation, and horizontal initiatives) and the inadequate financial tracking of activities within components and sub-components of the initiative; these are discussed at greater length in Sections 3.3 and 4.1.1, respectively. Moreover, it was not possible to compare it with similar programs or initiatives in other BSE-infected countries, such as those developed in the UK, France, and Germany; this work was outside the scope of the evaluation. The following assessment is therefore based primarily on the compilation and analysis of statements collected from both internal and external informants, as well as the factual analysis of activities, outputs, and outcomes based on documentation review.

Based on the evidence obtained from both internal and external informants, the nearly unanimous view (47 out of 49 answers) is that Canadians received value for their money and/or benefited from the Enhanced BSE Initiative. This wide-reaching and complex initiative significantly helped to mitigate the impact of BSE in Canada by meeting the needs of its stakeholders and by achieving most of its main outcomes (i.e., animal and human health risk management, consumer confidence, and market recovery). Many informants reached this conclusion by speculating on what might have happened in the absence of the CFIA’s Enhanced BSE Initiative (or by comparing what took place in Canada with what happened in other BSE-affected countries), such as sustained market closures, loss of consumer confidence, the risk posed to the health of the animal resource base, the loss of sustainability of the cattle- and beef-related professions, and the more harmful impacts (economic, mental, etc.) on Canadian communities.

There is no doubt as to the importance of the initiative’s outcomes in terms of the number of individuals that could have been affected, the billion-dollar scale of the beef and cattle industries, and the timescale over which the impacts of a large-scale BSE outbreak could have been felt. As such, it could be argued that the true value of avoiding a full-blown BSE epizootic is impossible to calculate. Thus, considering the success of the Enhanced BSE Initiative (in concert with other BSE measures) at mitigating the impact of BSE in Canada, it is possible to conclude that overall, Canadians received value for their money.

Given the context of the crisis in 2003-2004 and the fact that the CFIA implemented various components of the BSE initiative in a climate of great urgency—i.e., in a reactive mode (as opposed to in a proactive mode) and over such a short period—more than half of the internal informants argued that the CFIA could not have designed and delivered the Enhanced BSE Initiative in a more cost-effective manner. In other words, the CFIA leveraged existing structures to minimize duplication and to ensure the cost-effectiveness of the various components of the initiative based on the knowledge available at the time. However, the on-going cost-effectiveness and possible alternatives of some specific components of the Enhanced BSE Initiative have recently come under discussion.

Cost-Effectiveness and Alternatives by Component

Alternatives are currently being considered by the CFIA to increase the cost-effectiveness of some components of the BSE initiative, including: 1) BSE surveillance, 2) the SRP, and 3) facilitating the expansion of domestic slaughter capacity. Moreover, the fact that the cattle ID program is not currently fully implemented has implications in terms of the efficiency and effectiveness of traceback activities, such as in the agency’s ability to rapidly and comprehensively trace cohort animals. Continued improvements of the cattle ID program will therefore contribute to increasing the cost-effectiveness and decreasing the turnaround time of traceback activities. The cost-effectiveness of these and other components of the Enhanced BSE Initiative are discussed below.

Enhanced BSE Surveillance Testing

According to several key internal informants, and as reported in some recent media reports,⁴³ reducing the number of samples tested for BSE and/or discontinuing the SRPs are presently being considered as part of a strategy that would focus on fewer samples with higher value, which would therefore provide the same OIE score.⁴⁴ Although the absolute level of testing done by the CFIA (almost 60,000 samples in 2007-2008) is lower than that of some other countries, this sampling level is generally accepted internationally, given the depth of science behind the testing of fewer but higher value animals (i.e., 4-D animals,⁴⁵ or those within relevant age cohorts which may or may not be exhibiting clinical symptoms). Note that basic information for each animal is necessary to assess the value of individual samples (i.e., age, clinical expression, collection point, or origin/distribution in the Canadian herd). However, this data is not currently systematically recorded or reported; this has implications in terms of the way data is collected at the time of sampling or as part of the cattle ID program because without this essential information, it is not possible for the CFIA to set a lower sampling and testing target that is scientifically sound and that represents the same relative point value to meet OIE requirements. Note that Alberta has already begun modifying its provincial sampling program according to the OIE point system, in close consultation with CFIA operations in this region.⁴⁶

Similarly, the SRP is also under review, as it was intended to increase the number (but not necessarily the quality) of samples for BSE testing. The SRP is recognized as a key driver for surpassing the objectives of the enhanced surveillance testing component and as being instrumental in raising the awareness of the cattle industry and facilitating the involvement of industry in the surveillance program. Its abolition would represent a savings of approximately $6 million per year for the CFIA and could possibly represent additional savings for provinces if they also decide to discontinue their SRPs.

However, a number of concerns have been expressed by key informants regarding these alternative approaches. In the views of many informants, any reduction of the number of samples, including changes in the SRP, constitutes a risk, because the CFIA does not have information on how this reduction would be perceived by the Canadian public, industry, and international stakeholders. As such, moving forward with changes to the surveillance program without serious consultative work threatens the credibility and validity of the BSE surveillance program, as perceived by stakeholders. A strong two-way communication strategy would have to be implemented to mitigate these risks and to educate Canadians and international stakeholders on the scientific and technical rationale behind any changes to the level of sampling and to the modus operandi of the BSE surveillance program. Possible impacts on the willingness of producers in different regions to continue providing samples without the reimbursement program also need to be carefully measured to ensure an adequate number of samples will still be obtained.

Facilitating the Expansion of Domestic Slaughter Capacity

Activities to facilitate the expansion of domestic slaughter capacity received approximately $18 million under the BSE initiative. As explained previously, the outputs generated by this activity were adequate, but the expected outcomes did not materialize because of factors external to the CFIA; thus, this activity is no longer judged as relevant. However, as Canada continues to be overly dependent on the US for cattle processing, the expertise and capacity developed for the review, approval, and registration processes for new plants may be required in the future in the event of unforeseen border closures. Thus, the cost-effectiveness of this component would be increased if the CFIA maintained a minimum level of capacity for this program component over the years. This could be done through ongoing training, for example.

Expanding Export Markets

It is very challenging to assess the cost-effectiveness of the activities under expanding export markets, especially for the technical experts posted abroad (in Japan, Mexico, Belgium, and China). In fact, the technical experts are involved in several other issues outside of BSE, and the outputs and outcomes of their activities have not yet been fully captured by any performance information or reports. Most informants argue that the presence of technical experts in key foreign markets, while expensive, is highly valuable for and beyond BSE; some internal informants believe them to be so successful as to justify additional postings in other regions (e.g., Russia or South America), while others are less convinced of the value that would be added by additional postings. However, it is not possible at this stage and with the available information to provide evidence of the effectiveness of this activity for the budget allocated under the Enhanced BSE Initiative. An alternative suggested by some was to develop a more integrated approach in which BSE resources and activities help support non-BSE issues, such as through a commodity-based or community-based strategy. The CFIA experts stationed abroad are already helping to support the management of other food safety risks, but this could be more formally recognized and utilized. Similarly, the cost-effectiveness of other BSE activities, such as surveillance and tracking and tracing, could be increased by broadening the scope of these activities beyond BSE and cattle.

Alternatives to the Enhanced BSE Initiative

When the Enhanced BSE Initiative is considered as a whole, broadening the scope of the initiative’s activities beyond BSE would be consistent with the way in which the CFIA has traditionally developed its routine activities through strategic integration of complementary activities; for instance, plant inspections, training programs, and livestock identification programs help the CFIA protect animal and human health from disease beyond BSE. Thus, evidence from this evaluation shows that the BSE-targeted funding has already brought benefits to several non-BSE activities at the CFIA in addition (and at no detriment) to BSE-targeted objectives.

For example, it appears that its experience in dealing with BSE has already informed and supported the CFIA’s planning for avian influenza. BSE funding also made possible the expansion of the existing resources in the Operations and Policy and Programs Branches and supported the creation of new initiatives—such as, but not limited to, traceability—that contribute to the achievement of the CFIA’s overall strategic outcomes. In a similar way, expansion of the benefits of BSE funding to develop expertise on TSEs could, in the words of one internal informant, "ensure that we’re not simply capable of responding [to emerging TSEs], but we’re capable of doing some more proactive work as well." Allowing the BSE initiative to benefit the management of TSEs and other animal diseases is also consistent with the CFIA’s mandate and management structure.

Finally, when considering other potentially more economical alternatives to the current BSE initiative, it is important to understand what the nature of BSE implies in terms of risk management. Because of the nature of the disease (prions) and its long incubation period, the control of BSE necessitates particular scientific tools, long-term investment, and large-scale risk management strategies. Conventional tools and approaches used to manage other animal diseases (viral or bacterial) are not suitable or are not directly applicable. As the BSE-infected cows that have been identified through surveillance were infected years before, it is not possible to test the feed and find out the precise provenance of the infection. The only way to deal with BSE is to use a systemic approach in order to control and ultimately eradicate the disease—but there is a cost associated with this approach. The unconventional nature of BSE thus makes it expensive to control, as the impact of measures put in place today can only be seen after five or six years. Thus, feedback loops take even more time to influence the design and delivery of BSE management measures.

In Canada, the level of BSE awareness from government and industry in the 1990s made it difficult to justify serious proactive measures. In fact, the environment in which the CFIA operated impeded such measures, and the low level of perceived risk at that time hindered the implementation of the feed ban in 1997. However, five years after the 2003 BSE crisis, the value and benefits of the measures that were put into place are widely recognized. Thus, in light of the Canadian experience in managing BSE, potential proactive measures would gain from being envisioned and seriously analysed in a context beyond this disease, as was noted would also be the case for avian influenza.

5. Recommendations

Multiple lines of evidence indicate that the Enhanced BSE Initiative has been a success, as much in terms of its achievements and outcomes as for the enabling effects it has had within the CFIA. The evidence presented in this report suggests that most components of the BSE initiative are as relevant as ever and there is no evidence to suggest that they should be discontinued at this stage. Because of BSE’s long incubation period, the effectiveness of the BSE measures that have been put in place (including the recently implemented Enhanced Feed Ban) will not be measurable for another five to eight years. This initiative has greatly contributed to reducing the intensity of the BSE crisis, but the CFIA should not lose sight of the fact that several BSE-related scientific, economic, and political factors may change at any time. As such, the CFIA must maintain its capacity to respond to these changes in a timely manner. The CFIA should also stay the course of openness and transparency and continue to make decisions using science-based evidence. Moreover, the CFIA should continue to apply the lessons learned from BSE to develop proactive strategies for the risk assessment of other diseases (e.g., avian influenza).

The findings from this evaluation point to a series of alternatives and future directions that would help to leverage the money spent so far and increase the effectiveness of long-term activities that have been developed as part of the Enhanced BSE Initiative. BSE surveillance and traceability will necessitate ongoing, enhanced support and additional tied funding. Enhanced capacity needs to be maintained for several components of the initiative (see below) and should eventually be established as an integral part of the CFIA’s long-term mandate; this includes the removal of SRM from the human food supply, science support, HR/training, public affairs/communication, and some aspects of the expansion of export markets. The only component of the initiative for which the continued need is no longer justified is the facilitation of the expansion of domestic slaughter capacity; however, the expertise that has already been developed to help build this capacity should be maintained to respond quickly to changing or future needs.

Recommendation 1: Coordination and communication

To strengthen coordination and communication with regard to BSE activities, both within the CFIA and with external partner organizations and stakeholders, a dedicated, expert-led coordination structure should be established in which roles and responsibilities are clearly established and communicated.

Internal communication deficiencies within the CFIA currently hinder the proper coordination (i.e., design and delivery and governance) of BSE activities. The chain of command (including roles and responsibilities) is not always clearly understood with regard to BSE activities, and communication does not always flow as it should among senior management and the Policy and Programs, Operations, and Science Branches, as well as between the National Capital Region and the area branches. Internal coordination and communication should therefore continue to be improved, especially in the long term, as BSE continues to become less high-profile. Moreover, coordination and communication with external partner organizations (including other government departments) and stakeholders should also be strengthened, particularly with respect to the expansion of export markets (see recommendation 7).

• A dedicated, expert-led coordination structure should be established to review and improve coordination and communication on BSE-related activities, which should include regular input from a dedicated BSE performance measurement officer in order to support data collection and reporting activities for all BSE needs (e.g., internal performance management and external reporting to the TB and OIE; see recommendation 2).
• The CFIA should make contact information more clearly available so that external partner organizations can rapidly get the information that responds to their needs; this should include designating a key contact person for each component of the enhanced BSE programming.

Recommendation 2: Performance measurement and reporting

To improve current performance data collection and reporting practices, the CFIA should consider the following measures: dedicate targeted funding for BSE performance measurement and reporting, and subsequent evaluations; develop and implement a results-based management plan that integrates both program management/senior management needs and external reporting needs. To this effect, the CFIA should dedicate at least one position to the coordination of BSE performance measurement activities across the organization and promote a performance management culture at all levels of the agency.

BSE-related performance data that are currently collected should be better coordinated and tailored to ensure use for the following purposes: 1) decision-making, management needs, and critical assessment of achievement; 2) reporting (internal and external); 3) integration within a comprehensive system; and 4) to make information readily accessible in a timely fashion. As such, the recommendations listed above are offered to improve the effectiveness and efficiency of BSE performance measurement and reporting and, to a broader extent, increase the CFIA’s commitment to result-based management. More specifically:

• The results-based management plan for BSE activities should be developed based on a review of management needs of both programs and operations managers/staff (including BSE experts) and senior managers. This plan should ensure that BSE performance measurement and reporting is integrated within the CFIA’s overall management structure and existing performance measurement system. The system used for BSE performance measurement and reporting should also be accessible and respond to the multiple data collection and reporting needs, including compliance with external reporting requirements (TB, OIE, and others).
• The CFIA should also examine other performance indicators that are closely related to BSE (e.g., for TSEs) to avoid duplication and leverage the efforts developed for non-BSE initiatives (e.g., traceability and inspection), as well as harmonize data collection with that of third parties, which collect data as part of other Canadian initiatives (i.e., ATQ, provincial establishments, and labs).
• The one or more position(s) dedicated to the coordination of BSE performance measurement activities across the organization would help to consolidate identified and emerging performance information needs (internal and external) and ensure that the data is properly collected on a regular basis within an integrated performance measurement system (see also recommendation 1).
• The CFIA should involve key persons/units accountable and responsible for corporate performance measurement, such as the Corporate Secretariat, who, in consultation with the AERO Branch, will engage the organization in this process by promoting a performance management culture at all levels of the agency.

Recommendation 3: HR and BSE expertise

To develop and sustain a critical mass of BSE expertise within the CFIA, succession planning should center on a commitment to hiring highly qualified staff and allow for an extended transition period so that new staff can be mentored and trained by existing BSE specialists within the agency.

There is currently much concern regarding succession at the CFIA, as BSE expertise is due to decrease through imminent and ongoing turnover involving key managers, specialists and scientists.

• Increasing the CFIA’s commitment to hiring highly qualified persons for BSE activities and allowing for an extended transition period in hiring new staff will facilitate the maintenance of BSE expertise within the CFIA, as well as promote internal transfer of essential knowledge (i.e., of key partners and stakeholders), so that the CFIA retains its awareness of and involvement in ongoing BSE initiatives led by external organizations (both Canadian and international).

Recommendation 4: Financial Tracking

To better manage and assess outputs of BSE tied funds, the CFIA should track funds at the level of specific activities funded under the BSE initiative. This would also allow for cost-effectiveness analysis, both at the level of sub-activities and program components.

The BSE funds were not appropriately and consistently tracked within the CFIA’s financial system. The amounts received and spent are not tracked at the level of specific activities, or even at the level of components as described in the TB submission (i.e., Enhanced BSE Surveillance & Testing, Facilitating the Expansion of Domestic Slaughter Capacity, etc.), making the cost-effectiveness analysis impossible. For instance, no evidence was found of the activities proposed in the TB submissions within the "Strengthening Science & Support" sub-component (i.e., "Meat Inspection Reform" and "Quality Assurance Based Certification System"), and so it was not possible to identify if or how the funds requested for these activities were used.

• Options to track funds at the level of specific activities funded under the CFIA BSE initiative should be investigated and a revised system should be implemented to better manage and assess outputs.

Recommendation 5: BSE surveillance testing and the SRP

The CFIA’s future decisions as to its BSE surveillance testing targets and the SRP should continue to be supported by science-based guidelines addressing both the effectiveness of the risk management measures for BSE in animals as well as the effectiveness of the feed ban. The CFIA should also develop a communication strategy (including a consultation strategy) to better inform stakeholders with regard to ongoing discussions and developments on the future of the SRP. To this end, better coordination and communication is needed between those at the front lines of BSE testing.

The surveillance and testing program of the Enhanced BSE Initiative must be maintained to, at minimum, meet OIE guidelines. In addition, the results of science-based research from the CFIA and from other research organizations, as well as a better understanding of the epizootic stemming from current surveillance efforts (Lethbridge is the BSE Reference Lab and is also the OIE Recognized Lab for BSE in Canada) should continue to inform Canadian surveillance guidelines, and support decisions as to the most effective and most cost-effective way to achieve surveillance targets. However, mixed messages are coming from different sources within the CFIA as to the number of samples required, and consequently, the need for the SRP.

• The announcement of future decisions as to BSE surveillance testing and the SRP should be communicated in the clear and transparent manner that the CFIA has generally displayed for BSE in the past. As such, the CFIA’s communication strategy would benefit from a common front within the CFIA, which implies the need for better communication among those at the front lines of BSE testing, i.e., those conducting surveillance testing, those performing research on BSE incidences, and those developing regulations and policies.

Recommendation 6: Tracking, Tracing & Enforcement

The CFIA should continue to work with industry and other governments to ensure that the national cattle identification (ID) program and database is standardised and includes date of birth, movement capture, identification of deadstock, and better identification of premises. The CFIA should also enforce activities beyond tagging compliance (e.g., enforce the submission of data to the national database).

The CFIA should continue to fulfill its role in regulating and enforcing tagging within the CCIP and would gain in both efficiency and effectiveness from sustained collaboration with industry and other federal/provincial governments to move this program forward. The following would improve tracking and tracing outcomes:

• Ensuring the national database is comprehensive and standardised, such as for the date of birth, movement capture (including farm-to-farm and farm-to-auction), identification of deadstock, and better identification of premises (including auction barns, abattoirs, renderers and importer/exporters).
• Regulating activities beyond tagging compliance and enforcing the submission of data to the national database. Poor reporting, particularly by abattoirs (i.e., for the retirement of tags), has been documented and should be addressed.
• Additional education and awareness materials and operations should be developed to support and facilitate the CFIA’s continued efforts in the improvement of CCIP and traceability in Canada.

Recommendation 7: Expanding export markets

To help manage industry expectations, the CFIA should clarify its role and responsibilities for this component, particularly in relation to other government agencies involved (e.g., DFAIT, AAFC). The CFIA should also continue to integrate BSE trade-related activities into broader country- or commodity-based strategies and confirm the added value of its regionally focused technical experts.

The need for the CFIA to clarify its role and responsibilities with regard to expanding export markets, both with regard to its mandate and compared to that of its federal government partners, largely stems from the confusion surrounding its dual role as an enforcement/regulatory agency that is simultaneously working in support of industry stakeholders. In addition:

• BSE should be integrated into broader country-based or commodity-based strategies during international trade negotiations. As the benefits of a commodity-based strategy are wide-ranging, these and other activities to expand export markets should not be covered through targeted BSE funding only.
• The need to place geographic or regionally focused technical experts in additional export markets has been raised. Further investigation is recommended to ensure the added value of existing and potential additional experts. Once the value of these activities (i.e., their progress and contribution to outcomes) has been measured, this value should be articulated and promoted in terms of the benefits for BSE-affected trade, as well as for other commodities. This will help to increase awareness of the role of regionally focused technical experts by external stakeholders.

Recommendation 8: Continuity and Scope

As part of its continued efforts to develop an integrated approach to food safety, public health and risk management for animals and humans, the CFIA should expand the scope of specific activities within the Enhanced BSE Initiative beyond BSE, particularly to other issues related to TSEs and other emerging diseases. Finally, the CFIA must ensure that it continues to meet stakeholder expectations and maintain confidence regarding the legitimacy of its regulatory/enforcement role for BSE.

The CFIA should not become complacent with regard to the BSE threat and should continue to develop an integrated approach to risk management for animals and humans, including BSE and other animal diseases.

• The successes of the BSE initiative should be leveraged to benefit the CFIA’s efforts for managing the risk of other food-borne diseases. As noted, the CFIA is particularly well positioned to study and manage the risk of other TSEs for which it has already begun to develop its expertise (e.g., chronic wasting disease and scrapie) efficiently and effectively.
• Expectations created during the successful management and communication activities implemented for BSE are high, as the CFIA has successfully established the legitimacy of its regulatory and enforcement role for BSE in Canada. The CFIA must ensure that it continues to meet these expectations in order to maintain the current high levels of confidence and recognition by the three main stakeholder groups: the Canadian public, the Canadian industry, and international trading partners.

Appendix A - List of Evaluation Advisory Committee Members

Unit*	Name, Position
Corporate Evaluation, (AERO Branch)	Joanne Roulston, Director Peggy Smyth, Senior Analyst, Evaluation Julie Côté-Stordy, Evaluation Analyst
Corporate Financial System (Accounting Operations, FAIT Branch)	Eldon Taylor, Manager, Corporate Financial System
Foreign Animal Diseases/Animal Welfare (Animal Health Directorate, Programs/ Terrestrial Animal Health Division, Policy & Programs Branch)	Dr. Gary Little, Veterinary Program Officer
Western Operations (Operations Branch)	Philip Amundson, Executive Director
Animal Feed Division (Animal Health Directorate, Policy & Programs Branch)	Sergio Tolusso, Program Coordinator
Issues Communications (Public Affairs Branch)	James Stott, Manager (A/Director)

* Source: Government Electronic Directory Services (GEDS) (Accessed November, 2008)

Appendix B - CFIA Enhanced BSE Initiative Logic Model

Click on image for larger view

Appendix C - Evaluation Issues and Questions

Evaluation Themes & Issues	Evaluation Methods
	Internal Document / File Review	External Document / File Review	Internal Interviews	External Interviews
1.0 Inputs/Resources (Process)
1.1 Resources and Governance
Are the financial resources and human resources (FTEs) adequate for the activities of the enhanced version of the program?
How efficiently and effectively were Enhanced BSE funds used, both overall and by component/objective?
What impact did the Enhanced BSE funds provided have on the ability of the program to achieve its overall objectives and specific objective?
Are there appropriate management and decision-making structures in place to meet the objectives? Is this structure working as intended?
Are the roles and responsibilities, including authorities, involved in the program clear and well understood by internal and external stakeholders?
1.2 Relevance
Are the objectives clear and relevant for the activities under the enhanced version of the program?
To what extent does the enhanced program respond to the needs, priorities, and mandates of each of the major stakeholders?
Does the enhanced program continue to be consistent with government-wide priorities and the CFIA mandate?
Is there a continued need for the enhanced program, both overall and by program component?
1.3 Design and Delivery
What are the impacts of the current program structure on program delivery and performance?
What are the major internal factors influencing the performance of this program? How have these factors been addressed in the design and delivery of the program?
What are the major external factors influencing the performance of this program? How have these factors been addressed in the design and delivery of the program?
To what extent is the collaboration with partner organizations and stakeholders adequately integrated in the design and delivery of the program? Are there overlaps or gaps?
Are the communication outreach/training activities reaching their intended audiences and having the desired impact?
1.4 Performance Measurement and Reporting/Results-based Management
Is the current performance measurement strategy and monitoring plan appropriate? Does it provide a complete picture of performance?
Has the performance measurement strategy and monitoring plan been implemented and delivered?
Are the appropriate data collection (reporting/timeliness) systems in place, and do they meet program management and organizational reporting requirements?
How has CFIA and program management used performance information (e.g., in planning and decision-making)?
1.5 Cost-Effectiveness/Alternatives
Is the current approach of the Enhanced BSE Program cost-effective?
Are there alternative approaches to the current design and delivery of the program that would allow it to meet its objectives and mandate in a more effective and efficient manner? What would the cost implications be?
2.0-7.0    Success and Achievements
2.0-3.0    Activities and Outputs
To what extent have planned activities for each program component actually been implemented?
To what extent have implemented activities for each program component produced the expected outputs?
4.0-7.0    Outcomes (Immediate, Intermediate, Long Term Outcomes)
To what extent have implemented activities for each program component produced the expected outcomes (immediate, intermediate, long term, and CFIA corporate strategic outcomes)?
Are there any unintended (or unexpected) outcomes (positive and negative), and how have they been addressed?
What lessons have been learned? What changes in program design/delivery would be required, if any? What other considerations need to be taken into account?
To what extent are the measurable outputs and outcomes attributable to the Enhanced BSE Program activities?
Given the outputs and outcomes of the Enhanced BSE Program, how should the CFIA and stakeholders position and envision the orientation of the program for the future?

Appendix D - International Markets Partially Re-opened¹, 2003-2008

Click on image for larger view

Notes:

^1. International markets partially re-opened following the BSE trade ban of May 2003: Partially = boneless beef, animals under 30 months of age, etc.

^2. Includes one or more of: fetal bovine serum, hides/skins/furs, tallow, animal feed, antlers, porcine products, rendered fats/fat derivatives, non ruminant MBM, spray dried blood, bovine gelatin & collagen, dairy products not for human consumption, bovine blood and blood products, raw materials, ruminant meal and bone meal

Source: Data provided by CFIA representatives

---

¹ The acronym stems from the prior name of the organization - Office international des epizooties.

² See the CFIA website: http://www.inspection.gc.ca/english/anima/heasan/
disemala/bseesb/bseesbfse.shtml (Accessed October 2008)

³ The sample reimbursement program outputs and outcomes fall under Enhanced BSE Tracking and Surveillance.

⁴ Two interviews were conducted with two informants present.

⁵ The Beef Value Chain Roundtable was launched in January 2003 by AAFC. It emerged as a key vehicle for ensuring a coordinated response to BSE and was a key focal point for various working groups to engage with representatives of the entire beef value chain. Now more focussed on long-term issues, membership on the beef roundtable includes ranchers, feedlot operators, processors, retail and food service representatives, feed suppliers, and others. http://www.ats.agr.gc.ca/value_chain_roundtables/
beef_e.htm (Accessed June 2008)

⁶ It should be noted that fine-grained tracking of resources for reporting is often an issue within this type of organization and/or with this type of complex program, as it would require a great deal of additional time and effort on the part of inspection staff. As such, unreasonable expectations as to detailed financial tracking should be kept in check.

⁷ Objectives stated in TB Submission; surveillance sample numbers taken from the CFIA website: http://www.inspection.gc.ca/english/anima/heasan/
disemala/bseesb/surv/surve.shtml#num (Accessed September 2008)

⁸ Quality and timeliness data are taken from AERO annual performance reports, and are incomplete after 2005-2006.

⁹ All data are from Compliance Report 2007 – National Livestock ID Enforcement; Excel file provided by the CFIA.

¹⁰ Turnaround time is calculated as the time between confirmation of a positive case and publication of the report summary online; date of publication and the report summaries were found on the CFIA website: http://www.inspection.gc.ca/english/anima/heasan/
disemala/bseesb/comenqe.shtml (Accessed November 2008)

¹¹ AERO, CFIA Enhanced BSE Activities, 2005-2006 Semi-Annual Performance Report; data unavailable after 2005-2006.

¹² AERO, CFIA Enhanced BSE Activities, 2005-2006 Semi-Annual Performance Report; and SRM Removal in Federal Establishments for the Period July 12, 2007 to March 31, 2008; Excel file provided by the CFIA.

¹³  AERO, CFIA Enhanced BSE Activities, 2005-2006 Semi-Annual Performance Report.

¹⁴ AERO, Updated Info from Annual Reports, Word file provided by the CFIA covering 2007-2008.

¹⁵ AERO, Updated Info from Annual Reports, Word file provided by the CFIA covering 2007-2008.

¹⁶ AERO, Updated Info from Annual Reports, Word file provided by the CFIA covering 2007-08.

¹⁷ The following survey results were provided to the evaluators. Public opinion on BSE was the central topic of four waves of polls conducted between May 2003 and June 2005. Surveys marked with a « include questions on BSE but focus on a larger topic.
- Ekos, Tracking Public Opinion on Mad Cow Disease, covers Wave I: May 24-28, 2003, Wave II: June 15-19, 2003, Wave III: January 9-14, 2004 (Ekos)
- «Ekos, Public Views on Food Claims and Food Labelling, March 2005
- Redfern, Canadian Attitudes Toward BSE, June 2005
- «Maritz, Confidence in Imported Food Safety in Canada, August 2007

¹⁸ CFIA Internal document, Recommendations regarding CCIA resulting from CFIA’s BSE Investigations, updated March 2008.

¹⁹ For example, see: IGAC on Traceability for Livestock and Poultry, Roadmap, Draft dated April 4, 2008.

²⁰ AERO, Consolidated Results-Based Management and Accountability Framework for All CFIA Enhanced BSE Programming, Draft v. 6, December 2007; these are also listed in DCM under items 4.5 and 5.1-5.5 (see Appendix B).

²¹ Approximately one third of internal informants who commented on this outcome referred to obstacles or challenges encountered during traceback investigations, and the following CFIA internal document provided further details on difficulties relating to the CCIP: Recommendations regarding CCIA resulting from CFIA’s BSE Investigations, updated March 2008. CFIA traceback investigation reports can be found at: http://www.inspection.gc.ca/english/anima/heasan/
disemala/bseesb/comenqe.shtml (Accessed November 2008)

²² See outcome listed under item 5.4.1 in the DCM (see Appendix B) and questions 13 (Senior Management) and 14 (Branches) in the interview guides for key internal informants (see Methodology Supplement).

²³ Note that Statistics Canada uses beef available in Canada as an indicator of beef consumption, as these figures have been adjusted for loss, as stated on their website: "This release is based on food available for human consumption from the Canadian food supply, adjusted for losses. The data have been adjusted for retail, household, cooking and plate loss." See: http://www.inspection.gc.ca/english/anima/heasan/
disemala/bseesb/comenqe.shtml (Accessed November 2008). It should also be noted that the 2003 increase in beef consumed occurred during a time when red meat consumption was dropping in Canada (a trend that began in 1999) due to life-style or general health reasons rather than following concerns as to BSE or meat safely (see consumer survey conducted by Redfern, Canadian Attitudes Toward BSE, June 2005).

²⁴ For survey references, see footnote 17.

²⁵ Statistics Canada, Cattle Statistics 2008 (23-012-X).

²⁶ For more details, see the OIE website: http://www.oie.int/eng/normes/mcode/code2006/
en_chapitre_2.3.13.htm (Accessed September 2008)

²⁷ Statistics Canada, Cattle Statistics 2008 (23-012-X).

²⁸ TDV Global, 2007. Evaluation of the Feed Program Final Report, Prepared for AERO Branch, CFIA.

²⁹ Note that this is the opinion of certain internal and external informants and is not a conclusion of this evaluation; indeed, financial data could not be tracked in enough detail to either confirm or refute this perception.

³⁰ Four external informants were from Quebec; three from the provincial government and one from industry.

³¹ CFIA, Renewal Plan 2008-2013.

³² CFIA website: http://www.inspection.gc.ca/english/agen/agene.shtml (Accessed October 2008)

³³ For a complete list, see the CFIA website: http://www.inspection.gc.ca/english/reg/rege.shtml (Accessed October 2008)

³⁴ Note that all branches receiving BSE funding are responsible for tracking their BSE funds, including proper use of internal order codes. The FAIT branch creates and assigns these codes (this can be done in consultation with the branches) and so branch managers are not necessarily aware which TB submissions their funds come from.

³⁵ For example, the BSE training activities: at least $1,279,600 was spent designing and delivering various courses and materials that respond to national needs for all five components of the Enhanced BSE Initiative (including the feed ban); however, the managers for these activities do not necessarily know which TB submission their division’s funds came from (i.e., they report their use of funds with internal order codes, and may not know to how these codes relate to a specific TB submission). In addition, they horizontally integrated BSE training activities with non-BSE training by identifying common target audiences and by determining the most efficient delivery mechanisms. In parallel to this training at the national level, additional BSE funds are used to develop e-learning resources, area-specific training, and training of industry (end-user training).

³⁶ AERO produced comprehensive annual performance report on BSE for the 2004-2005 fiscal year, followed by a semi-annual and an annual report for 2005-2006. However, this process was not repeated in the subsequent years because, according to an informant, it was not utilized. In the context of this summative evaluation, AERO asked CFIA representatives to update performance information contained in the performance report for the last two years and, in this way, most of the performance information was obtained. However, the data provided for this evaluation were frequently incomplete, often with the most recent numbers missing.

³⁷ See specifically the "Policy Requirements" section of the Evaluation Policy, available on the TB website: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?
id=12309&section=text (Accessed on September, 2008)

³⁸ Bertrand F. and Caruso J. (2008) Review of a Social Sciences and Humanities Research Council Performance Measurement Initiative, Prepared by Science-Metrix for the Corporate Performance, Evaluation and Audit (CPEA),
Social Sciences and Humanities Research Council of Canada (SSHRC).

³⁹ This evidence includes the absence of Annual Performance Reports, which were prepared on a annual or semi-annual basis by the AERO branch up until 2005-2006, the fact that public confidence on the safety of Canadian food and consumer behaviour with regard to beef and BSE has not been measured since 2005. Finally, certain data requested for this evaluation, for instance with regard to science-based outputs since 2005, could not be made not available within the timeframe of the project.

⁴⁰ CFIA Internal Document, Recommendations arising from Canada’s BSE submission to the OIE for the evaluation of country status, May 2007.

⁴¹ This statement is based on statements made by internal informants. However, no documents or files were available on the outputs or performance-related indicators of veterinarians posted abroad, or an existing mechanism that they may use to report performance information.

⁴² For example, lack of date of birth information for cows tagged as part of the cattle identification program has become an obstacle to accessing international markets that have only been re-opened to animals under 20 or 30 months of age (although this is a deficiency of the CCIP rather than a fault in the CFIA’s inspection role, knowing the exact age of an animal can also be important from a surveillance and traceback point of view).

⁴³ For example: David Ljunggren, "Canada says it could cut funding for mad cow tests", Reuters World General News, 16 July 2008
John Cotter, "Feds making beef industry more responsible for food safety", The Canadian Press, 17 July 2008

⁴⁴ The OIE surveillance point system is described on their website: http://www.oie.int/eng/normes/mcode/code2006/
en_chapitre_3.8.4.htm#chapitre_3.8.4 (Accessed September 2008)

⁴⁵ 4-D animals: dead, dying, diseased and downers

⁴⁶ As described on the Alberta Agriculture and Rural Development website: http://www1.agric.gov.ab.ca/$department/
deptdocs.nsf/all/cpv9336 (Accessed September 2008)