Evaluation of the Food Recall and Emergency Response System

Management Response

Audit, Evaluation and Risk Oversight
Canadian Food Inspection Agency

April 2009

---

• Executive Summary
• 1. Introduction and Background
  • 1.1 Program Description
  • 1.2 Previous Review
  • 1.3 Evaluation Objectives
  • 1.4 Structure of the Report
• 2. Evaluation Methodology
  • 2.1 Evaluation Framework
  • 2.2 Data Collection
  • 2.3 Data Quality and Limitations of the Evaluation
• 3. Success
  • 3.1 Activities and Outputs
• 4. Design and Delivery
  • 4.1 Governance Structures
  • 4.2 Roles and Responsibilities
  • 4.3 Processes and Procedures
  • 4.4 Resourcing
  • 4.5 Performance Monitoring
• 5. Previous Recommendations
• 6. Conclusions and Recommendations
  • 6.1 Conclusions
  • 6.2 Recommendations
• Annex A - Advisory Committee
• Annex B - Evaluation Matrix
• Annex C - List of Documents and Websites Reviewed
• Annex D - Interview Guides
• Annex E - Case Study Methodology
• Annex F - Previous Recommendations

---

Executive Summary

One of the Canadian Food Inspection Agency’s (CFIA’s) top priorities is to protect Canadians from preventable health risks such as those associated with unsafe food.  The ability to prevent or contain food safety emergencies and incidents in a timely and appropriate manner is a critical component of CFIA’s mandate to protect Canadians from preventable health risks. CFIA’s Food Recall and Emergency Response System is triggered by various sources including consumer and industry complaints, inspections, audits, laboratory results or referral from other organizations.  Following an investigation, food recalls may be conducted that are aimed at removing food products from retail and distribution when a health risk and/or violation is confirmed. The Food Recall and Emergency Response System is based on the following five stages: food safety investigation, decision making, recall implementation, recall effectiveness and follow-up.

The purpose of this evaluation was to conduct an internal review of the food recall process in order to support continuous improvement. The objective of this evaluation was to assess the success of the food recall and emergency response program and the effectiveness of the program’s design and delivery.  The degree of implementation of the recommendations from a previous review of this system was also assessed. The evaluation focused on the three year time period since the last review, 2005/2006 to 2007/2008.

Evaluation Background

The evaluation was conducted as a process-based evaluation, focusing on the roles and responsibilities, procedures, resourcing and monitoring of the process.  Data collection for the evaluation was conducted during the spring and summer of 2008.  The assessment was conducted using multiple lines of evidence, including:

• document review;
• review of Web publications;
• staff and management interviews;
• Health Canada (HC) interviews;
• Area visits; and
• case studies.

Conclusions

Overall, the Food Recall and Emergency Response System is operating effectively with very dedicated staff. It has clear accountability and decision making, and is supported by good information management systems.  Decision making is risk-based. High-risk health incidents are consistently well-managed, with no indications of issues detrimental to food safety.

The relationship with Health Canada is strong and effective in terms of risk assessment.  Food emergency response roles and responsibilities related to other government departments and the provinces/territories are outlined in protocols and Memorandums of Understanding (MOUs).

Considerable progress has been made in implementing the recommendations of the previous review. Process improvements are noted in the development of common health risk definitions, guidelines for public warnings, updating of the Food Emergency Response Manual (FERM), and the completion of the Food Safety Investigation Manual. The upgrade of the information management system and the implementation of the quality assurance procedures are also notable achievements.  Improvements in bilingual capacity are evident within OFSR, the technical assessor (TA) group and in documentation.

Some additional process improvements are recommended with respect to sampling and ensuring follow-up activities are tracked and implemented.  Resource pressures are noted in some areas.

Success

Overall, the activities for the Food Recall and Emergency Response System are being implemented and producing the expected outputs and outcomes.

Food safety investigation, including laboratory testing, and recall activities are given priority over other planned surveillance and inspection activities.

Common health risk definitions have been implemented for risk assessments.  Most health risk assessments are completed in a timely fashion. In the past, the lack of agreed-upon time guidelines for risk assessments resulted in different expectations internally, and some health risk 2 and 3 assessments were not completed in a timely manner. Time standards are under discussion within CFIA and with Health Canada.

Recall activities are well-implemented. Public notifications and contact with other countries occur in a timely fashion. While primary recall incidents have been quite stable over time, large-scale investigations and events with many secondary recalls are erratic and can place significant pressure on the system.  Historically, these events have been managed well because management reallocates staff to priority activities.  This situation needs to be recognized and further consideration given to planning for extraordinary requirements.

The efficiency of recall effectiveness checks has increased due to the implementation of statistically based sampling guidelines, and the average number of checks per recall is stable.  The time standards contained in FERM for effectiveness checks are considered occasionally difficult to meet in complex recalls.  Enhanced verification is effectively used in high-risk situations.

Follow-up activities are not uniformly tracked as some commodity inspection staff do not use IMS and the field was not mandatory in system in the past.  Some follow-up activities are unevenly conducted, particularly in the manufactured and imported food sector.

Design and Delivery

Management structures within CFIA are appropriate and effective. The recall structure in the West is dispersed geographically. The relationship with Health Canada was found to be strong and effective. Roles and responsibilities with other departments and other levels of government are guided by MOUs and protocols. Awareness and execution of these could be improved with some provinces. Some issues exist with information sharing between jurisdictions.

Roles and responsibilities are generally clearly documented and understood.  Agreement has been reached on several key areas that required clarification including follow-up, product disposition and risk management recommendations during the health risk assessment step while scoping and sampling remain under discussion.  Decision making is generally effective and risk-based.

Overall, processes and procedures are well documented, understood and implemented with recent revisions to the FERM noted to clarify some specific areas.  Finalization and full implementation of the FERM is pending and fully supported including sampling and priority setting of an investigation.

While resource levels have fluctuated over the three-year period under review, shortages have been noted in food safety inspection staff in some areas and at times with the technical assessors in the Policy and Programs Branch in headquarters in Ottawa.  The unplanned nature of food safety investigations and recalls exerts pressure across branches, and particularly on inspection staff and laboratory operations, during large-scale or High Visibility Incidents.

The main system related to food safety investigations and recalls, IMS, has recently been upgraded and provides quality data related to the tracking of investigations, recalls and effectiveness checks. Some commodity inspection staff do not use IMS and therefore, the data may not be complete in terms of investigations and follow-up activities.  Post-mortems or lessons learned discussions related to non-routine incidents occur occasionally and better use of these forums would support continuous improvement.

Recommendations from the Previous Review

A comprehensive management action plan was developed to address the recommendations of the previous review.  Implementation is complete on the bulk of the recommendations. The small number that are nearing completion remain relevant and should be finalized.

Recommendations

CFIA should finalize and monitor the implementation of the recent process improvements and continue to seek opportunities to enhance the Food Recall and Emergency Response System while retaining the strong accountability, decision making and responsiveness of the system. Staffing issues should be addressed. The key recommendations should be formally tracked and progress reported to senior management to ensure their full implementation.  The key recommendations (incorporating the remaining recommendations from the previous review):

1. Complete the revisions and implementation of the FERM
   • Finalize outstanding areas: sampling, scoping and priority setting
   • Distribute manual
   • Monitor and refine, if needed, the revised roles and responsibilities
2. Continue to enhance policies/procedures as indicated
   • Finalize Standard Operating Procedures for risk assessment including time guidelines incorporating Health Canada’s role (Previous recommendations #9 and #11)
   • Finalize common priority setting process to define and communicate level of urgency including non-routine decision making, High Visibility Incidents (HVIs)
   • Communicate the public warnings policy to key partners (Health Canada, Public Health Agency of Canada) (Previous recommendation #13)
   • Further deploy IMS to commodity based inspectors
   • Continue to implement measures to improve the quality of information entered into IMS (Previous recommendation #19)
   • Complete the alignment of program manuals with FERM (previous recommendation #2)
   • Resolve issues related to content and timeliness of information sharing with provincial authorities investigations, recall effectiveness checks and recall implementations
3. Review and maintain awareness of existing  MOUs and protocols with key partners at the federal and provincial levels
   • Resolve issues related to information sharing across jurisdictions during food safety investigations and recalls related to outbreaks
   • Review and enhance procedures for information sharing with provincial authorities during investigations, recalls and recall effectiveness checks
   • Maintain awareness through activities such as annual reviews, periodic simulated exercises, updating key contact lists
4. Address staffing issues
   • Address food safety program resource shortages in Ontario, West
   • Ensure appropriate training is provided in a timely fashion
   • Review recall structure in the Western Area
5. Continue to improve conduct and tracking of follow-up activities
6. Pursue opportunities to improve teamwork and communication internally
   • E.g. daily joint situation reports, joint lessons learned from non-routine cases, secondments, co-location of units, information sessions

---

List of Acronyms and Abbreviations

AERO

Audit, Evaluation and Risk Oversight

AOC

Area Operations Coordinator

ARC

Area Recall Coordinator

A/VP

Associate Vice President

CAP

Corrective Action Plan

CV

Compliance Verification

ED

Executive Director

EIS

Enforcement and Investigation Services

FDA

Food and Drug Administration (US)

FERM

Food Emergency Response Manual

FIORP

Foodborne Illness Outbreak Response Protocol

FRER

Food Recall and Emergency Response (also known as OFSR)

FSD

Food Safety Directorate

HC

Health Canada

HRA

Health Risk Assessment

HVI

High Visibility Incident

IM

Inspection Manager

IMS

Issues Management System

MAPAQ

Le ministère de l’Agriculture, des Pêcheries et de l’Alimentation du Québec

MOU

Memorandum of Understanding

OCM

Operations Centre Manager

OFSR

Office of Food Safety and Recalls (previously and subsequently known as FRER)

QMP

Quality Management Program (e.g. Fish program)

QMS

Quality Management System

RD

Regional Director

RECs

Recall Effectiveness Checks

ROA

Report of Analysis

SOPs

Standard Operating Procedures

TA

Technical Assessor (in Food Safety Directorate)

VP

Vice President

---

1. Introduction and Background

1.1 Program Description

Background

One of CFIA’s top priorities is to protect Canadians from preventable health risks such as those associated with unsafe food.  The ability to prevent or contain food safety emergencies and incidents in a timely and appropriate manner is a critical component of CFIA’s mandate to protect Canadians from preventable health risks.

Food manufacturers use many controls to protect the safety of the products they produce. On occasion, for many different reasons, a food product may be manufactured and sold which may pose a human health risk. When an unsafe food product has left the control of the manufacturer or importer, it must be removed from the market. This process of removing the food product is called a "recall." The food industry carries out most recalls voluntarily once a health risk has been identified by the manufacturer or others. The recalling firm is responsible for the implementation of the recall and verification of the recall effectiveness. CFIA verifies the recall effectiveness as well, however, if a company is not available or willing to conduct the recall voluntarily, the Minister of Agriculture and Agri-Food can, under the Canadian Food Inspection Agency Act, order a company to recall a product if the Minister believes that it poses a health risk to the public.  This action is referred to as a mandatory recall.

When CFIA learns of a potential food safety issue, an investigation is launched to determine whether the product poses a health risk.  These investigations can be triggered by various sources including consumer and industry complaints, inspections, audits, laboratory results or referrals from other organizations.  As a result of the investigations, food recalls may be conducted and are aimed at removing products from the retail stores and distribution networks where a health risk and/or a violation is confirmed.

Within CFIA, the Office of Food Safety and Recall (OFSR, previously known as Food Recall and Emergency Response, FRER) is the single point of contact for both domestic and international food-related issues and food-related emergencies.  OFSR is responsible and accountable for the coordination and consistency of decision making on food safety issues and, if necessary, recalls.

Objectives

Food safety protection is the key objective of the food recall and emergency response program.  OFSR operates under the following four principals:

• timeliness - incidents involving actual or alleged illness, injury or high-visibility issues should be investigated promptly;
• appropriateness - the depth of an investigation should be appropriate to the hazard and the likelihood of its occurrence. The investigation should cover all relevant levels at which the product has been manufactured, imported, distributed, sold or consumed;
• consistency - the investigative procedures applied both within a specific investigation and to all investigations should be congruent; and
• thoroughness - investigations should identify all affected or potentially affected product.

Recall Process

There are various triggers that initiate a food safety investigation which could lead to a food recall. These triggers can include: reports by public health officials, consumer complaints, company-initiated concerns, international reports as well as regular CFIA inspections and testing/sampling programs. This evaluation used the following five stages of the food recall process, as outlined in the Food Emergency Response Manual (FERM): food safety investigation, decision making, recall implementation, recall effectiveness and follow-up.

Food Safety Investigation

The goal of a food safety investigation is to determine whether a food safety hazard or violation exists and to determine the nature and extent of the problem.  Information obtained throughout the investigation provides the basis for the assessment of risk and the development of appropriate risk management strategies to control affected products.

Decision making

CFIA and Health Canada (HC) work together to reach a decision about the risk posed by a product. Standard health risk definitions are in place for the risk assessment:

• Health Risk 1 – The health risk identified represents a situation where there is a reasonable probability that the consumption/exposure to a food will lead to adverse health consequences which are serious or life-threatening, or that the probability of a foodborne outbreak situation is considered high.
• Health Risk 2 – The health risk identified represents a situation where there is a reasonable probability that the consumption/exposure to a food will lead to temporary or non-life threatening health consequences, or that the probability of serious adverse consequences is considered remote.
• Health Risk 3 – This represents a situation where there is a reasonable probability that the consumption/exposure to a food is not likely to result in any adverse health consequences.

The level of risk contributes to determining the class of food recall. There are three classes of food recalls:

• Class I - represents a situation in which there is a reasonable probability that the consumption/exposure to a food will lead to adverse health consequences which are serious or life-threatening, or that the probability of a foodborne outbreak situation is considered high.
• Class II - represents a situation in which there is a reasonable probability that the consumption/exposure to a food will lead to temporary or non-life threatening health consequences or that the probability of serious adverse consequences is considered remote.
• Class III - represents a situation in where there is a reasonable probability that the consumption/exposure to a food is not likely to result in any adverse health consequences.

Recall Implementation

Where it is determined that the appropriate risk management strategy is to recall the affected product, the recalling firm is advised and CFIA staff work with the recalling firm to implement a recall. A public warning may be issued depending on the circumstances. In the case of a mandatory recall, CFIA will assume responsibility for the recall.

Recall Effectiveness

The recalling firm is responsible for ensuring the effectiveness of the recall.  In addition, CFIA officials conduct effectiveness checks to verify that the recalling firm has recalled the product effectively.  On some occasions, officials from provincial and territorial governments work with CFIA to conduct effectiveness checks. If the recalling firm is unwilling to remove the product for sale and appropriately dispose of it, CFIA may seize and detain the product. Further administrative actions, including prosecution, could also take place.

Follow-up

CFIA staff follow-up with the recalling firm during and following the recall to ensure that issues are handled consistently and appropriately. Follow-up activities may include determining the disposition of returned product and product in storage, evaluation of the corrective action implemented for the current affected product, and review and evaluation of the corrective action implemented to reduce the likelihood that there will be a reoccurrence of the issue. OFSR tracks recall trends for review by programs.

Key Players in the Food Safety System

The Canadian food safety system is a complex system which can involve the expertise of many groups and organizations. The key players are outlined in the table below. This evaluation is limited to the food recall and emergency response functions of the Canadian Food Inspection Agency.

Group	Players
Industry	Growers Processors Distributors Importers/exporters Food service outlets Retailers
Canadian Public	Consumers (store and prepare food safely)
Provincial/Territorial, Municipal, Local	Medical community Local health units Provincial/territorial health departments
Federal Government	Canadian Food Inspection Agency Health Canada Public Health Agency of Canada
International	Other national food safety agencies Other national public health agencies

1.2 Previous Review

In April of 2005, a review was done by Audit, Evaluation and Risk Oversight (AERO) of the Food Emergency Response Program which concluded, for the most part, that the program and system operated well.  However, the review highlighted three areas requiring attention: 1) the need to clarify and communicate the responsibility for recall decisions, 2) the need to improve the risk/technical assessment process and 3) the need to improve follow-up activities.  A management action plan was developed in April 2005 and progress reports on the implementation of the recommendations were completed twice after the review was released, in May 2005 and May 2006.

1.3 Evaluation Objectives

This evaluation follows up on the recommendations of the previous review.  The objectives are to assess the success of the food recall and emergency response program and the effectiveness of the program’s design and delivery.  The evaluation focuses on the three-year period since the previous review (i.e. from 2005/2006 to 2007/2008).

1.4 Structure of the Report

This report follows the approach developed for the previous review.  It examines the same five broad steps of the recall process (i.e. food safety investigation, decision making, recall implementation, recall effectiveness and follow-up).  The report contains the following sections:

• Section 1 outlines the Food Recall and Emergency Response System and provides context including information concerning the previous review;
• Section 2 summarizes the evaluation methodology, including the evaluation framework, data collection methods and limitations of the evaluation;
• Section 3 presents the findings and conclusions related to the evaluation issue of success of the system;
• Section 4 outlines the findings and conclusions related to the design and delivery of the Food Recall and Emergency Response System; and
• Section 5 summarizes the conclusions and the recommendations resulting from the evaluation.

---

2. Evaluation Methodology

2.1 Evaluation Framework

The evaluation focuses on the success of the Food Recall and Emergency Response System and the effectiveness of the program’s design and delivery.  The following list of evaluation issues was developed based on the Treasury Board Evaluation Policy, the previous review, and other issues pertinent to the Agency.  These are summarized below and the detailed performance indicators and data sources are outlined in Annex B.

Success

• Are recall activities producing the expected outcomes?  These activities include investigations (including samples), recalls, effectiveness checks and follow-up activities.

Design and Delivery

• Are there appropriate governance structures in place to deliver the program activities and how effective are they?
• Are the roles and responsibilities of the food recall process clearly documented and understood? (e.g. OFSR, Area Recall Coordinators (ARCs), Area staff, Health Canada, etc.).  How effective is the decision-making process throughout the food recall stages?  Is it effectively using a risk-based approach?
• Are the processes and procedures clearly documented, understood and implemented in each of the five stages of the food recall process?
• Is the level of resourcing adequate, given the scope and mandate?
• How effective are the performance monitoring systems for food recall activities? And the mechanisms in which they report and/or provide feedback?

Recommendations from the Previous Review

• To what degree have the recommendations from the previous evaluation been implemented?  Are outstanding recommendations still relevant?

2.2 Data Collection

Data collection consisted of the following elements:

• Document review including policy documents, program documentation, Web sites, previous evaluations and audits, Memorandums of Understanding (MOUs) and agreements, and performance reports including annual reports, performance reports and statistical reports
• Interviews with CFIA staff (see below for a list of the groups of interviewees, and Annex D for the interview guides);
• Interviews with key Health Canada informants;
• Site visits to two areas: Ontario and Quebec; and
• Case studies (see Annex E).

Interviews by Group - Number

• Internal – Programs
  • Food Safety Directorate - 8
  • Program Network - 4
• Internal – Operations
  • OFSR - 5
  • Quebec Area - 6
  • Ontario Area - 5
  • Western Area - 1
• Internal - Science Branch
  • Laboratory Operations - 2
• External
  • Health Canada - 2
• Total - 33

2.3 Data Quality and Limitations of the Evaluation

Overall, data quality from the main recall related system, Issues Management System (IMS), is considered to be good. The IMS System has been in operation for about eight years and has recently been updated.  Statistics related to incidents, investigations and recalls are well-recorded. Some concerns have been raised over data completeness because not all programs use IMS evenly.

Telephone interviews were used to solicit input from Atlantic and Western Areas.  A limited number of external informants were interviewed.

During the evaluation study, OFSR was relocated from Programs Branch to Operations Branch. In addition, the Food Safety Directorate was restructured to form the Food Safety and Consumer Protection Division, with the Food Safety Program becoming the Manufactured and Imported Food Program.  The impact of this restructuring was not fully evident.

---

3. Success

This portion of the evaluation contains an assessment of the success of the Food Recall and Emergency Response System in conducting its activities and producing the expected outputs. It examines the activities that are performed against the identified procedures and the trends in outputs where available. Findings are derived from the document reviews, site visits, staff and management interviews and case studies.

3.1 Activities and Outputs

Are recall activities producing the expected outputs?

The following tables list the key outputs of the Food Recall and Emergency Response System by stage of the process. Additional statistics are also included in each section.

Food Safety Investigation

A food safety investigation is conducted where a human health and safety concern and/or a violation of CFIA legislation is suspected. Operations staff in the Areas take the lead in the investigation with support from OFSR and others as needed. Science Branch provides the laboratory testing in support of investigations. Investigations may include interviews, review of records, visits to plants, sample collection, laboratory testing, etc.

Activities/Outputs	Are they being implemented?	Source
Investigations, IMS records, Samples collected, Samples analyzed	Investigations take precedence over other activities	Interviews
	Number of investigations relatively stable, average of 2,833 per year with a range from 2,585 to 3,077	IMS statistics
	Relatively stable proportions related to imported and domestic product	IMS statistics
	Reports that IMS not used uniformly across all commodity programs for issues	Interviews
	Consumer complaints are the largest source of triggers for investigations every year	IMS statistics
	Sample testing by labs (Science Branch) timely. Large sampling volumes can impact planned lab activity	Interviews

Diagram 1: Number of Investigations Annually

Note: Large number of unclassified investigations because the field was not mandatory prior to 2008

Diagram 2: Trends in Triggers for Investigations

Risk Assessment, Decision Making

When a violation has been confirmed, a risk assessment is completed based on the available information.  Health Canada is responsible for completing the health risk assessment. In cases where there is a Health Canada guideline in place, CFIA technical assessors (TAs) may make the risk assessment, referred to as a technical assessment. Based on the health risk assessment or the technical assessment, OFSR makes the decision regarding the risk management strategy. OFSR consults, as required, with Programs, Technical Specialists, Operations, and Health Canada to inform their risk management decision.

Activities/Outputs	Are they being implemented?	Source
Health risk assessment, Technical assessment, Recall authorization	Common health risk definitions have been developed	Document review, interviews
	Assessments that are expected to result in a Health Risk 1 are given priority and conducted in a timely fashion	Document review, interviews, case studies
	Lack of time guidelines in past led to differences in expectations across branches for potential Health Risk 2 and Health Risk 3 categories. Inspectors report that they need to know timelines in order to inform the company that is involved. The company may be holding product and keeping staff on alert pending assessment. Time standards are under discussion within CFIA and with Health Canada	Document review, interviews, case studies
	Recall authorization consistently meets defined time standard	Document review, case studies

Recall Implementation

When a recall is indicated, the involved firm is advised and CFIA works with the firm to implement the recall.  Food recalls may be voluntary or mandatory; however, the majority are voluntary.

• Voluntary recall means a recall that is initiated and carried out by the recalling firm without ministerial order.
• Mandatory recall means a recall as per Section 19 of the Canadian Food Inspection Agency Act.19.(1) “where the Minister believes on reasonable grounds that a product regulated under an Act or provisions that the Agency enforces or administers by virtue of Section 11 poses a risk to public, animal or plant health, the Minister may, by notice served on any person selling, marketing or distributing the product, order that the product be recalled or sent to a place designated by the Minister.”

The recalling firm is responsible for carrying out the recall.  A public warning may be issued and CFIA informs other countries if indicated. Media inquiries are responded to as required.  Area Operations and OFSR are the key players in this step. Public Affairs is involved when a public warning is issued.

Activities/Outputs	Are they being implemented?	Source
Recall notifications, Public advisories and warnings, Media lines, Notification of other countries	Recall notifications from OFSR to the Area Operations staff may be verbal initially with written documentation following	Interviews, case studies
	Weekly teleconferences between ARCs and OFSR are effective	Interviews
	Number of primary recall incidents are relatively stable over time, averaging 235 incidents per year, range from 221 to 262 (see diagram 3)	IMS statistics
	Secondary recall incidents are not predictable or constant and vary significantly from 15 to 218 per year (see diagram 3)	IMS statistics
	Public warnings prepared in a timely fashion	Interviews, case study
	Notification of other countries appears proactive	Case Studies

Diagram 3: Recalls Trends by Year

Notes: Increased number of recalls in 2006-07 and 2007-08 are primarily due to a single incident with many secondary recalls. 

Verification of Recall Effectiveness

The involved firm is responsible for determining whether a recall has been effective. CFIA then verifies the effectiveness though recall effectiveness checks (RECs). This is primarily an Area Operations responsibility.

Activities/Outputs	Are they being implemented?	Source
Effectiveness checks by CFIA	Generally well implemented	IMS statistics, Interviews
	Decline in total number of effectiveness checks from 2003-04 to 2005-06 due to implementation of statistically based sampling guidelines (see Diagram 4)	IMS statistics
	Average number of effectiveness checks per recall  stable following implementation of sampling guidelines, indicating consistent implementation of checks (see Diagram 5)	Interviews
	Significant increase in effectiveness checks in 2006-07 in imported products due to a large number of secondary recalls (see Diagram 4)	IMS statistics
	Some difficulties meeting time guidelines contained in FERM, particularly for large, complex recalls	IMS statistics
	Enhanced verification used for high-risk situations	Case studies
	Timeliness varies of Le ministère de l’Agriculture, des Pêcheries et de l’Alimentation du Québec (MAPAQ) summary reporting of effectiveness checks to the retail level	Case studies, Interviews

Diagram 4: Effectiveness Checks

Diagram 5: Average Number of Recall Effectiveness Checks per Recall

 

Follow-up

CFIA staff follow-up with the recalling firm during and after the recall. Follow-up activities may include the disposition of returned and stored product and the review of corrective action plans to reduce the likelihood of a recurrence at a plant. Sometimes, industry-wide initiatives may be indicated, such as policy changes and industry education programs. Area Operations staff lead the activities related to the recalling firm, consulting other branches as required.  Programs staff may be involved in the industry-wide initiatives.

Activities/Outputs	Are they being implemented?	Source
Company follow-up, Product disposition, re-work	Inconsistently tracked in IMS, as some commodity inspection staff do not use IMS. Follow-up was not a mandatory field in IMS prior to 2008	IMS, Interviews
	Operations Branch reports some difficulty completing follow-up activities due to resource pressures, particularly for manufactured and imported products	Interviews
	Unclear point of transition to Programs for follow-up	Interviews, Case studies

 

---

4. Design and Delivery

This portion of the evaluation contains an assessment of the design and delivery of the recall system. It examines the governance structures, roles and responsibilities, processes, and performance monitoring.  Findings are derived from the document reviews, site visits, key interviews and case studies.

4.1 Governance Structures

Are there appropriate governance structures in place to deliver the program activities and how effective are they?

OFSR has a clear mandate and provides a strong coordinating mechanism across branches.  During the period under review, the unit moved within the CFIA organizational structure twice, resulting in different management structures. Currently, the unit reports at a senior level (Associate Vice President, or A/VP), appropriate to its mandate, importance and visibility with the Canadian public. (source - document review, interviews)

The ARCs report within the Operations branch hierarchy. The level and reporting structure of the ARCs varies across the areas. All ARCs have a direct line of communication to OFSR for coordination of food safety investigations and recalls. This provides a direct, efficient line of communication from the field to OFSR while respecting area organizational structure. Some Areas (Ontario, Quebec) have centralized resources to manage food safety investigations and recalls, creating a specialized, dedicated team. The Western Area has three ARCs due to its geographic spread, but is lacking a central point of coordination.  (source - interviews, document review, case studies)

Science Branch is responsible for providing advice on sampling requirements, capacity issues, methodology issues, reporting as well as testing services. Existing Science Branch governance structures are used to manage food recall and emergency response activities. (source – interviews)

Food Safety and Consumer Protection Division of Programs Branch provides the primary link with Health Canada for the health risk assessments.  Where Health Canada guidelines exist, the Division is able to provide a technical assessment. The Division liaises with the commodity programs as required.  (source - interviews, case studies, document review)

A dispute resolution mechanism has been documented and implemented to address non-routine cases. (source – interviews, document review)

The relationship with Health Canada was found to be strong, cooperative and effective for the health risk assessments. In addition, there are a number of forums for interaction between the key federal departments – Public Health Agency of Canada (PHAC), CFIA and Health Canada. Notably, a tri-partite meeting has been established in the past year to review microbiology-related food issues. CFIA attendance includes both the Food Safety and Consumer Protection Division and OFSR.  (source - interviews, case study)

The Foodborne Illness Outbreak Response Protocol (FIORP) guides multi-jurisdictional roles during a foodborne outbreak. MOUs are in place with some provinces and territories to outline the various roles and responsibilities. Case studies found that awareness and execution of these protocols and MOUs with some provinces could be improved. Issues related to extent and timeliness of information sharing under some MOUs need to be resolved, in particular during outbreaks. Some Areas also conduct regular operational meetings and working groups with provincial counterparts to maintain communication. (source - interviews, document review, case studies)

4.2 Roles and Responsibilities

Are the roles and responsibilities of the Food Recall Program clearly documented and understood? How effective is the decision making process throughout the food recall stages? Is it effectively using a risk based approach?

Roles and responsibilities for the food safety and recall system are documented primarily in the FERM. Additional documents related to the recall system include:

• Food Safety Investigation Manual, which details roles and responsibilities during the investigation process.
• FIORP, which outlines the inter-jurisdictional responsibilities for food borne illness outbreaks for PHAC, CFIA, Health Canada, the provinces and territories food inspection and health departments as well as public health departments.
• MOUs with provincial and territorial governments which outline the roles of federal and provincial governments.

Stage	Findings	Source
Food Safety Investigation	Roles and responsibilities are documented in FERM and Food Safety Investigation Manual. Significant revisions to FERM drafted. Final approval pending FIORP clearly identifies roles and responsibilities for multi-jurisdictional outbreaks	Document review, interviews, case studies
	Food safety specialists in some Areas provide specialized support to other commodity inspectors during investigations	Interviews
	Responsibilities for scoping of incident not clear in past. Roles of OFSR, TA, Programs, Program Network updated in FERM, implementation underway	Document review, interviews, case study
	Roles and responsibilities regarding sampling not clear in past. The roles of OFSR, Area Operations, Science Branch - Labs, TA, Programs, and Program Network under discussion	Document review, interviews, case study
	Program Network Specialists inconsistently used in investigations, varies by commodity and by Area	Interviews
	Product not distributed in the marketplace is currently managed through Area inspection staff with commodity program support	Interviews
Risk Assessment / Decision Making	Roles and responsibilities very clearly documented in FERM	Document review, interviews
	Roles of Health Canada and Food Safety and Consumer Protection Division in the recommendation of risk management options slightly unclear – documentation varies from practice. Revised FERM resolves this issue	Document review, interviews
	Decision-making process effective for the majority of incidents	Document review, interviews, case studies
	Non-routine decision making process not well established in past. This process has since been established and documented	Document review, interviews, case study
	Lack of effective dispute resolution mechanism in past has been addressed and is formally documented	Document review, interviews
	Evidence of a risk-based approach in both Health Canada risk assessments and CFIA technical assessments	Document review, interviews
Recall Implementation	Roles and responsibilities clearly documented and understood	Document review, interviews, case studies
Effectiveness Checks	Roles and responsibilities clearly documented and understood	Document review, interviews, case study
Follow-Up	Roles and responsibilities for follow-up generally documented in commodity-specific manuals. Limited documentation in FERM. Revised FERM expands on roles and responsibilities.	Document review, interviews

 

4.3 Processes and Procedures

Are the processes and procedures clearly documented, understood and implemented in each of the five stages of the food recall process?

Processes and procedures for the Food Recall and Emergency Response System are primarily outlined in the FERM. Additional key process documents include:

• Food Safety Investigation Manual;
• Program (commodity-specific) Manuals – such as Meat Program, Fish Program, etc;
• FIORP: To Guide Multi-Jurisdictional Response;
• Guidelines from Health Canada such as the Policy on Listeria monocytogenes in ready-to-eat foods; and
• MOUs with provincial and territorial governments.

Stage	Findings	Source
Food Safety Investigation	Processes documented in FERM and Food Safety Investigation Manual	Document review, interviews, case studies
	Recent initiative to develop detailed procedures for inspectors as a result of the Quality Management System (QMS) audit	Document review, interviews
	Lack of clarity regarding sampling procedures e.g. number of samples required	Document review, interviews, case study
	Lack of a common understanding of priority of an investigation – urgent versus regular hours. Revised FERM includes guidelines.	Document review, interviews, case study
	Some isolated reports of delays in training for new staff in food safety investigation	Interviews
Risk Assessment / Decision Making	Overall processes documented in FERM	Document review, interviews
	Food Safety and Consumer Protection Directorate has developed templates for Request and Response for a Health Risk Assessment	Interviews
	Lack of time guidelines in the past for risk assessment, particularly for potential Health Risk 2 and Health Risk 3 issues had broad impacts – industry, inspectors, OFSR, Food Safety and Consumer Protection, Health Canada. Time standards are under discussion within CFIA and with Health Canada	Document review, interviews, case studies
	Health Canada risk assessments and CFIA technical assessments varied in format and structure; some provide too little information to assist inspectors in discussion with industry. Standard templates have been implemented and include a synopsis for communication with industry	Document review, interviews
	Health Canada not formally established to provide after-hours coverage. Informally very committed and accessible	Document review, interviews, case studies
	Decision Making Guidelines for Public Warnings developed and presented to Policy Committee	Document review, interviews
Recall Implementation	Processes clearly documented in FERM	Document review, interviews
	Processes well understood and consistently applied	Document review, interviews, case studies
Effectiveness Checks	Processes clearly documented in FERM	Document review, interviews
	Processes well understood and consistently applied	Document review, interviews, case studies
	Timelines identified in FERM difficult to meet with larger, complex recalls	Document review, interviews, case studies
	QMS audit of effectiveness checks found few issues	Document review
Follow-Up	Roles and responsibilities for follow-up generally documented in commodity – specific manuals. Limited documentation in FERM. Revised FERM expands on roles and responsibilities	Document review, interviews
	Uneven completion of follow-up activities reported	Document review, interviews
	Tracking of follow-up activities in IMS not consistent	Document review, interviews
	Reported gap in policies and guidelines to deal with repeat offenders. FERM and some commodity-specific procedures have been updated to address this issue	Document review, interviews

 

4.4 Resourcing

Is the level of resourcing adequate given the scope and mandate?

Operations Branch

Inspection staffing levels vary by region and commodity.  Significant pressures exist in some Areas, particularly with food safety inspectors.  Recruitment and retention of staff is a critical issue in Ontario (Toronto region), with other Areas reporting some pressures as well (West). In Ontario, local labour market conditions and the high cost of living are thought to be contributing factors. Despite shortages, food safety investigation and recall activities are assigned priority over other planned inspection activities resulting in some planned activities being less well-delivered, such as the planned projects for the Imported and Manufactured Food program and the Fair Labelling Practices Program. Follow-up activities related to recalls in the Imported and Manufactured Food are also less well-delivered due to these resource pressures. Existing inspection staff are under pressure, particularly during recalls that have large breadth or depth and High Visibility Incidents.

There are six ARC positions across the country -- one in each of Atlantic, Quebec and Ontario and three for the West due to its geographic size. Some Areas have additional staff dedicated to assist in some aspects of investigations and recalls, such as recall effectiveness checks, sample pick up from consumers, etc.

OFSR, currently reporting four vacancies, has historically been appropriately resourced from both the number of resources as well as funding for overtime.

Food Safety Division

This Division has been short of TAs in the past, placing pressure on a limited number of staff to manage the health risk assessments and ensure after-hours coverage.  The Division added a number of technical assessors in 2007/2008, which improved back-up capacity and after-hours coverage. This has also allowed improvement of management practices such as the introduction of a database to track risk and technical assessments.

Science Branch

Science Branch is able to provide the laboratory testing and methodological advice support required by the Food Recall and Emergency Response System. However, when significant numbers of samples are collected and high volumes of tests are required, laboratory operations can be significantly affected in terms of the need for overtime and delays in planned testing activities.

4.5 Performance Monitoring

How effective are the performance monitoring systems for food recall activities? And the mechanisms in which they report and/or provide feedback?

The IMS is the main system for tracking food safety investigations and recalls.  Upgrades to the IMS were implemented in June 2008. Some of the key improvements included additional controls to enhance data quality, increased reporting functionality, and the elimination of a parallel system. Data related to recalls and investigations is available historically to 2000.

Overall, data in IMS is generally good for food safety investigations, recalls, and effectiveness checks. The following limitations are noted:

• As not all of the program commodities use the IMS, some investigations may not be recorded. The extent of this issue has not been fully assessed but is estimated not to be significant;
• Tracking of follow-up activities is uneven because, previously, this was not a mandatory field and some commodities may not have access to or use the system for follow-up.

Performance data related to investigations and recalls are reported on a regular basis for a number of uses:

• The annual CFIA Departmental Performance Report provides information to the public on the number of investigations and recall incidents, CFIA’s performance in meeting the target for timeliness of issuing a public warning for Class I recalls and some description of key high-profile incidents. In past years, the type of triggers for food safety investigations and the type of hazard linked to recalls has also been reported.
• OFSR distributes data annually to commodity programs and the Food Safety Directorate.
• Recall data has been provided to the inter-departmental Science Committee that meets biannually to review food-related risks.
• Some ARCs monitor data for trends and issues in their Areas.
• Ad hoc analyses have been conducted periodically as required.

Within the Food Safety Division, two informal databases exist to track the health risk assessments and the technical assessments related to microbiology and allergens.  There is currently no comparable tracking of the chemical assessments. (source: interviews, document review)

Some food recall-related activities have been brought under the CFIA QMS. The process started in the fall of 2007 for consumer complaints, recalls and effectiveness checks. Initial process mapping and a first round of quality verifications was completed. OFSR is being brought into the process following the unit’s movement to Operations Branch. (source: interviews, document review)

Post-mortems and lessons learned reviews related to non-routine incidents occur at times and ongoing use of these forums is encouraged to support continuous improvement. (source: case studies)

5. Previous Recommendations

To what degree have the recommendations from the previous evaluation been implemented? Are outstanding recommendations still relevant?

The previous review included 22 recommendations. A comprehensive management action plan was developed in April 2005 and progress reports were completed in May 2005 and May 2006.  To date, progress is complete on 16 of the recommendations.  The remaining recommendations are well underway and final approvals or manual updates are required.  See Annex F for details of the recommendations. All of the in-progress recommendations remain relevant.  The recommendations that are in progress and nearing completion are:

• 2.  VP Programs, in collaboration with VP Operations, should ensure that inspection policies and procedures are consistent with the FERM
• 7. The ED Food Safety, in collaboration with other parties, should clarify and communicate the lead responsibility for scoping the immediate food safety issue and the responsibilities of others involved
• 9. The ED Food Safety should implement, where appropriate, common approaches for technical assessment and risk management options and communicate them to various parties involved in food safety investigations and recalls
• 11. The ED Food Safety should develop and implement service standards to enhance timeliness of CFIA technical assessments
• 13. ED Food Safety, in collaboration with Public Affairs (PA) should return to the Program Management Committee (PMC) with options and recommendations for a policy related to public warnings and information sharing on recalls
• 19. Area EDs should continue to implement measures to improve the quality of the information entered in the IMS

6. Conclusions and Recommendations

Overall, the Food Recall and Emergency Response System is operating effectively with very dedicated staff. It has clear accountability and decision making and is supported by good information management systems.

Process improvements are noted in the development of common health risk definitions, guidelines for public warnings, updating of the FERM, and the completion of the Food Safety Investigation Manual. The upgrade of the information management system and the implementation of the quality assurance procedures are also notable achievements.  Improvements in bilingual capacity are evident within OFSR (Operations), the technical assessor group and in documentation.

6.1 Conclusions

Success

Overall, the activities for the Food Recall and Emergency Response System are being implemented and producing the expected outputs and outcomes.

Food safety investigation, including laboratory testing, and recall activities are given priority over other planned surveillance and inspection activities.

Common health risk definitions have been implemented for risk assessments.  Most health risk assessments are completed in a timely fashion. In the past, the lack of agreed-upon time guidelines for risk assessments resulted in different expectations internally, and some health risk 2 and 3 assessments are not completed in a timely manner. Time standards are under discussion within CFIA and with Health Canada.

Recall activities are well-implemented. Public notifications and contact with other countries occur in a timely fashion. While primary recall incidents have been quite stable over time, large-scale events with many secondary recalls are erratic and can place significant pressure on the system.  Historically, these events have been managed well as management reallocates staff to priority activities.  This situation needs to be recognized and planning for extraordinary requirements further considered.

The efficiency of recall effectiveness checks has increased due to the implementation of statistically based sampling guidelines, and the average number of checks per recall is stable.  The time standards contained in FERM for effectiveness checks are considered occasionally difficult to meet in complex recalls.  Enhanced verification is effectively used in high-risk situations.

Follow-up activities are not uniformly tracked as some commodity inspection staff do not use IMS and the field was not mandatory in the system in the past.  The recent system upgrade has made this field mandatory.  Some follow-up activities are unevenly conducted, particularly in the manufactured and imported food sector.

Design and Delivery

Management structures within CFIA are appropriate and effective. The recall structure in the West is dispersed geographically. The relationship with Health Canada was found to be strong and effective. Roles and responsibilities with other departments and other levels of government are guided by MOUs and protocols. Awareness and execution of these could be improved with some provinces. Some issues exist with information sharing between jurisdictions.

Roles and responsibilities are generally clearly documented and understood.  Agreement has been reached on several key areas that required clarification, including follow-up, product disposition and risk management recommendations during the health risk assessment step. Scoping and sampling remain under discussion. Decision making is generally effective and risk-based.

Overall, processes and procedures are well-documented, understood and implemented, with recent revisions to the FERM noted to clarify some specific areas.  Finalization and full implementation of the FERM is pending and fully supported, including sampling and priority setting of an investigation.

While resource levels have fluctuated over the three-year period under review, shortages have been noted in food safety inspection staff in some areas and at times with the technical assessors in the Policy and Programs Branch in headquarters in Ottawa.  The unplanned nature of food safety investigations and recalls exerts pressure across branches, and particularly on inspection staff and laboratory operations, during large or High Visibility Incidents.

The main system related to food safety investigations and recalls, IMS, has recently been upgraded and provides quality data related to the tracking of investigations, recalls and effectiveness checks. Some commodity programs do not use IMS and, therefore, the data may not be complete in terms of investigations and follow-up activities.  Post-mortems or lessons learned discussions related to non-routine incidents occur occasionally and better use of these forums would support continuous improvement.

Recommendations from the Previous Review

A comprehensive management action plan was developed to address the recommendations of the previous review.  Implementation is complete on the bulk of the recommendations. The small number that are nearing completion remain relevant and should be finalized.

6.2 Recommendations

CFIA should finalize and monitor the implementation of the recent process improvements and continue to seek opportunities to enhance the Food Recall and Emergency Response System while retaining the strong accountability, decision making and responsiveness of the system. Staffing issues should be addressed. The key recommendations should be formally tracked and progress reported to senior management to ensure their full implementation.  This point was made during the previous review (Recommendation #22).  The key recommendations (incorporating the remaining recommendations from the previous review):

1. Complete the revisions and implementation of the FERM

• Finalize outstanding areas: sampling, scoping and priority setting
• Distribute manual
• Monitor and refine, if needed, the revised roles and responsibilities

2. Continue to enhance policies/procedures as indicated

• Finalize Standard Operating Procedures for risk assessment including time guidelines incorporating Health Canada’s role (Previous recommendations #9 and #11)
• Finalize common priority setting processes to define and communicate levels of urgency including non-routine decision making, High Visibility Incidents (HVIs)
• Communicate public warnings policy to key partners (Health Canada, PHAC) (Previous recommendation #13)
• Further deploy IMS to commodity based inspectors
• Continue to implement measures to improve the quality of information entered into IMS (Previous recommendation #19)
• Complete the alignment of program manuals with FERM (previous recommendation #2)

3. Review and maintain awareness of existing MOUs and protocols with key partners at the federal and provincial levels

• Resolve issues related to information sharing across jurisdictions during food safety investigations and recalls related to outbreaks
• Review and enhance procedures for information sharing with provincial authorities during investigations, recalls and recall effectiveness checks
• Maintain awareness through activities such as annual reviews, periodic simulated exercises, updating key contact lists

4. Address staffing issues

• Address food safety program resource shortages in Ontario, West
• Ensure appropriate training is provided in a timely fashion
• Review recall structure in Western Area

5. Continue to improve conduct and tracking of follow-up activities

6. Pursue opportunities to improve teamwork and communication internally

• E.g. daily joint situation reports, joint lessons learned from non-routine cases, secondments, co-location of units, information sessions

---

Annex A - Advisory Committee

The members of the Advisory Committee for the evaluation are listed below.

• Bashir Manji, Director, Office of Food Safety and Recall, Operations Branch
• Ken Marcynuk, National Manager, Office of Food Safety and Recall, Operations Branch
• Robert Charlebois, Director, Food Safety, Programs Branch
• Barbara Lee, Director, Food Laboratory Services, Science Branch
• Liz Singh, National Director, Laboratory Operations, Science Branch
• Joanne Roulston, Director, Evaluation, Audit, Evaluation and Risk Oversight
• Zelda Yule, Senior Analyst, Evaluation, Audit and Risk Oversight

---

Annex B – Evaluation Matrix

Evaluation Issue - 1.0 Success

Evaluation Questions	Indicators	Data Sources
1.1 Are recall activities producing the expected outputs?  These implementations include investigations (including samples), recalls, effectiveness checks and follow-up activities.	Volume trends in investigations, samples Trends in recalls (e.g. class, triggers) Effectiveness checks Follow-up activities Timeliness	Issues Management System (IMS) reports Key interviews

Evaluation Issue - 2.0 Design and Delivery

Evaluation Questions	Indicators	Data Sources
2.1 Are there appropriate governance structures in place to deliver the program activities and how effective are they?	Established governance structure in CFIA Clear governance structure with partnering organizations	Document review (FERM, MOUs, FIORP) Key interviews Partner Interviews (HC, PHAC)
2.2 Are the roles and responsibilities of the Food Recall Program clearly documented and understood? (E.g. FRER, ARCs, Area staff, HC).  How effective is the decision-making process throughout the food recall stages?  Is it effectively using a risk-based approach?	Clarity of documents of roles and responsibilities Level of understanding of roles and responsibilities Effective decision making Evidence of a risk-based approach to decisions	Document Review (FERM, MOUs, FIORP) Key internal interviews Partner Interviews (HC, PHAC) Case Studies
2.3 Are the processes and procedures clearly documented, understood and implemented in each of the five stages of the food recall process?	Clarity of documentation Level of understanding of staff processes and procedures Evidence that processes are being followed	Document Review Key internal interviews Case Studies
Is the level of resourcing adequate given the scope and mandate?	Identified gaps and shortfalls Recall timeliness Recall effectiveness checks delivery rate	Key internal interviews IMS reports OPM data
How effective are the performance monitoring systems for food recall activities? And, the mechanisms in which they report and/or provide feedback?	Effective tracking system Use of performance monitoring to make program changes	Document Review Performance reports Key internal interviews
To what degree have the recommendations from the previous evaluation been implemented? Are outstanding recommendations still relevant?	Degree of implementation of recommendations	Management action plan Updates Key internal interviews

---

Annex C - List of Documents and Key Web Sites Reviewed

Policy and Program Documents

Document	Author	Date
Food Emergency Response Manual	Food Recalls and Emergency Response, CFIA	Jun 7, 2002
Business Rules for Use of the Issues Management System (IMS)	--	2007
Foodborne Illness Outbreak Response Protocol to Guide a Multi-Jurisdictional Response	Health Canada	Jul 21, 2006
Decision Making Guidelines – Public Warning	OFSR	Apr 23, 2008
Policy on Listeria monocytogenes in Ready-to-Eat Foods	Food Directorate, Health Products and Food Branch, Health Canada	2004
Standards and Guidelines for Microbiological Safety of Food – An Interpretive Summary	Bureau of Microbial Hazards, Food Directorate, Health Products and Food Branch, Health Canada	July 2006
Health Canada/CFIA Health Risk Definitions – Applied to Microbiological, Nutritional and Chemical Hazards	Health Canada/CFIA	2005
Guidelines for raw ground beef products found positive for Escherichia coli O157:H7	Health Canada	Mar 8, 1999
Injurious Extraneous Material	Health and Welfare Canada	1990
Operational Policy Directive – Health Hazard Evaluation	Health and Welfare Canada	1993
Decision Tree – Timeline for Risk Assessment (draft and associated communications)	Food Safety Division, CFIA	2008
Health Risk Assessment Form (draft)	Food Safety Division, CFIA	2007
Food Recalls: Make a Plan and Action It! Distributors Guide Retailers’ Guide Importers’ Guide Manufacturers’ Guide	Food Recalls and Emergency Response, CFIA
CFIA Web site: FRER, recalls	--
Internal Web site: OFSR, HR	--
Food and Drugs Act	Department of Justice	1985

US Recalls

Document	Author	Date
US Food and Drug Administration – Recalls, Market Withdrawals and Safety Alerts (Web)	--
US Department of Agriculture, Food Safety and Inspection Services, FSIS Recalls (Web)	--

Previous Reviews and Evaluations

Document	Author	Date
Report on Food Emergency Response Review (Slide deck)	AERO (CPRA at the time), CFIA	Mar 2005
Food Emergency Response Review	AERO (CPRA at the time), CFIA	Apr 18, 2005
Food Emergency Response Review Management Action Plan #1302574		Apr 18, 2005
Food Emergency Response Review – Draft Response to Recommendations		May 12, 2005
Food Emergency Response Review – Update on Recommendations (Slide deck)		May 9, 2006
Report on the Food Safety System in Canada (v 5)	Policy and Programs Branch, CFIA	Jan 26, 2007
1999 September Report of the Auditor General: Chapter 15 Management of a Food-borne Disease Outbreak	Office of the Auditor General	Sept 1999
2000 December Report of the Auditor General of Canada: Chapter 25 CFIA Food Inspection Programs	Office of the Auditor General	Dec 2000
Food Safety Assessment: The Food Recall and Emergency Response System of the Canadian Food Inspection Agency	Health Canada	Mar 1999
Food Safety Assessment: Assessment Report of the Canadian Food Inspection Agency Activities Related of the Safety of Imported Fresh Fruits and Vegetables	Health Canada	Jun 2001

Reporting

Document	Auteur	Date
Departmental Performance Reports 2003/04, 2004/05, 2005/06, 2006/07	CFIA	Annual
Sample Health Risk Assessments: Allergen, Listeria, Allergen, Patulin, Chemical	CFIA, Health Canada	Various

Web Sites

CFIA External Web Site

• Food Recalls
• CFIA Consumer Centre: Causes of Food Borne Illness, Food Safety Tips, Specific Products and Risks, Educational Resources, Email Notification Services
• CFIA Newsroom: Food Recalls and Allergy Alerts

CFIA Internal Web Site

• Office of Food Safety and Recall

US Food and Drug Administration

• Recalls, Market Withdrawals and Safety Alerts

US Department of Agriculture, Food Safety and Inspection Service

• FSIS Recalls, Fact Sheets

---

Annex D - Interview Guides

Interview guides were prepared for:

• program managers and implementation staff; and
• other government departments.

These interview guides are contained in the pages which follow.

Food Recall and Emergency Response System
CFIA Internal Interview Questionnaire

Interview conducted by:
Date:
Person(s) interviewed and position(s):

Introduction

1. Which of the following steps of the Food Recall and Emergency Response System are you involved in, and briefly, what is your role:

• 1.1 Food safety investigation
• 1.2 Risk assessment
• 1.3 Recall coordination
• 1.4 Recall effectiveness verification
• 1.5 Recall follow-up

Process Review

2. Food Safety Investigation

• 2.1 Are roles and responsibilities with respect to food safety investigations well documented and understood?  Are they applied consistently?
• 2.2 Are policies and procedures for food safety investigations well documented and understood?  Are they applied consistently?
• 2.3 Are food safety investigations being conducted in a timely and effective manner? If not, why?  What improvements, if any, do you recommend? How is scope managed?
• 2.4 How is coordination with other branches managed? (e.g. lab sampling/testing, program advice, FRER interaction)  Is it managed effectively and consistently?
• 2.5 What training is required to support effective food safety investigation?  Is this training available/taking place?  If not, why?

3. Risk Assessment

• 3.1 What are the roles and responsibilities with respect to risk assessment?  Are these well documented and understood?
• 3.2 What criteria are used in making risk assessment and technical assessment decisions?  Are these well documented and understood? Are these criteria being applied in a consistent manner?  If not, why?
• 3.3 Are risk assessments and technical assessments contributing to timely and effective decision making?  If not, why?
• 3.4 What specific challenges do you face with respect to the risk assessment process?  What improvements would you recommend?

4. Recall Coordination (Implementation)

• 4.1 What are the roles and responsibilities with respect to recall coordination?  Are these well documented and understood?
• 4.2 What policies and procedures are in place for recall coordination?  Are they applied consistently?  If not, why?
• 4.3 Are recalls being conducted in a timely and effective manner?  If not, why?
• 4.4 How is coordination with the other branches/organizations managed?  Is it managed effectively?
• 4.5 How is recall coordination with recalling firms and industry managed?  Is it managed effectively?
• 4.6 What protocols are in place for communication with the media and the public?  Are these protocols effective?  If not, why?
• 4.7 What specific challenges do you face with respect to recall coordination?  What improvements would you recommend?

5. Recall Effectiveness Verification

• 5.1 What are the roles and responsibilities with respect to recall effectiveness verification?  Are these well documented and understood?
• 5.2 What policies and procedures are in place for recall effectiveness verification?  Are they applied consistently?  If not, why?
• 5.3 How is coordination with other branches/organizations managed?  Is it managed effectively?
• 5.4 What specific challenges do you face with respect to recall effectiveness verification?  What improvements would you recommend?

6. Recall Follow-up

• 6.1 What are the roles and responsibilities with respect to recall follow-up?  Are these well documented and understood?
• 6.2 What policies/procedures are in place for recall follow-up?  Are these applied consistently?  If not, why?
• 6.3 What policies/guidelines are in place for dealing with repeat offenders?  Are these being applied consistently?
• 6.4 What processes are in place for the analysis of recall trends?  Does the analysis of recall trends contribute to program changes/redesign?

Other

7. Accountability and Decision Making

• 7.1 Is the governance structure for CFIA’s Food Recall and Emergency Response System well documented and understood?  Are roles, responsibilities and delegation mechanisms clear?
• 7.2 What are the key decision points in a food emergency response situation?  At each of these key points, are decisions being made in a timely and consistent fashion?  If not, why?

8. Resources

• 8.1 Is the level of resourcing adequate to meet the requirements? If not, where are the gaps and what impact are they having? Are recall activities impacting other activities? If so, what and how?

9. Performance Monitoring

• 9.1 Is IMS allowing the effective capture and management of food safety and recall information?  Are the Standard Operating Procedures/business rules for the documentation of food safety investigations and recalls adequate? 
• 9.2 How does performance information related to recalls get analyzed and used within CFIA? Who conducts the analysis and does it contribute to program or policy development?

10. Implementation of Recommendations

• 10.1 What has been implemented to date in terms of the recommendations of the previous evaluation (see next page for list)?

Food Emergency Response System
External Interview Questionnaire

Interview conducted by:
Date:
Person(s) interviewed and position(s):

Introduction

The Canadian Food Inspection Agency is conducting an internal evaluation of the Food Recall and Emergency Response System. The initiative is being led by the Audit, Evaluation and Risk Oversight (AERO) Branch. This evaluation is a follow-up to one conducted in 2005.  The results of the evaluation will be used internally to guide improvements.

We would like to interview you based on the questions outlined below. We anticipate requiring about one hour.

Thank you for your participation in this process.

Interview Questions

1. Briefly describe your involvement in the Canadian Food Recall and Emergency Response System and your contact with CFIA. Are other government departments and/or agencies involved and what is their involvement?
2. What are the governance structures and working relationships in place between the organizations related to food emergency response and are they effective?
3. What are the roles and responsibilities with respect to risk assessment and risk management?  Are these well documented and understood?
4. What criteria are used in making risk assessment and technical assessment decisions?  Are these well documented and understood? Are these criteria being applied in a consistent manner?  If not, why?
5. Are risk assessments and technical assessments contributing to timely and effective decision making?  If not, why?
6. What specific challenges do you face with respect to the risk assessment process?  What improvements would you recommend?
7. How is emergency response and recall information shared with your department and how is it used?

---

Annex E – Case Study Methodology

Four case studies were completed on recent food safety incidents to further identify good practices and key issues.  The case studies were completed through document review only. Documentation was solicited from Operations, Programs, Public Affairs and Science Branches.

The case studies examined the incidents from a common approach as follows:

• Summary Observations
  • Key Issues
  • Good Practices
• Key Milestones
• Process Review
  • Food Safety Investigation
  • Risk Assessment / Decision Making
  • Recall Implementation
  • Recall Effectiveness Checks
  • Follow-up
• Communications
  • Internal, external

The results of the case studies have been incorporated into the evaluation report as an additional line of evidence.

---

Annex F – Previous Recommendations

The table below identifies the recommendations of the previous evaluation along with an assessment of the progress and the continued relevance.

Recommendation	Progress	If not complete, is it relevant?
1. VP Programs, in collaboration with VP Operations, should clarify and communicate the roles and responsibilities of all key parties involved in the food emergency response, including the delegated responsibility for recall decisions	Complete Documented in updated version of FERM	--
2. VP Programs, in collaboration with VP Operations, should ensure that inspection policies and procedures are consistent with the FERM	In progress Updating of program manuals underway	Yes
3. VP Programs should establish an enhanced decision making process for non-routine cases including a dispute resolution mechanism	Complete Documented in updated version of FERM	--
4. Director OFSR, in collaboration with the ARCs, should complete the Food Safety Investigation Manual	Complete Done. Another version underway to streamline training based on feedback. Quebec using own manual to reflect role of MAPAQ	--
5. Area EDs should ensure appropriate training is provided to Area Field Staff	Complete, ongoing Training available and occurring, issues with timeliness	--
6. Area EDs should establish quality assurance procedures to promote continuous improvement in the conduct of food safety investigations related to recalls	Complete QMS initiated Sept 2007. Scope included consumer complaints/investigations, recalls, and effectiveness checks	--
7. The ED Food Safety, in collaboration with other parties, should clarify and communicate the lead responsibility for scoping the immediate food safety issue and the responsibilities of others involved	In Progress Under discussion for inclusion in updated version of FERM	Yes
8. ED Food Safety should continue to work with Health Canada to ensure common health risk definitions are finalized and used as part of the technical/risk assessment	Complete	--
9. The ED Food Safety should implement, where appropriate, common approaches for technical assessment and risk management options and communicate them to various parties involved in food safety investigations and recalls	In progress Templates for Request and Response for Health Risk Assessments developed, piloted. Final review in progress	Yes
10. The ED Food Safety should establish a clear mechanism for communicating with CFIA staff changes to policies, standards and guidelines (including mitigating factors) used during the risk assessment process	Complete	--
11. The ED Food Safety should develop and implement service standards to enhance timeliness of CFIA technical assessments	In progress Time standard discussions ongoing between OFSR, FSD and Health Canada	Yes
12. The ED Food Safety should assess the feasibility of providing Area Field Staff with written information on the risk assessment prior to notifying the establishment	Complete Health Risk Assessment template contains synopsis for use in communications with industry	--
13. ED Food Safety, in collaboration with PA, should return to the Program Management Committee (PMC) with options and recommendations for a policy related to public warnings and information sharing on recalls	In progress Criteria for public warnings drafted and presented to Policy Committee. Final approval pending	Yes
14. ED Public Affairs, in collaboration with OFSR, should continue to explore opportunities to enhance outreach to Canadians on recalls	Complete Some activities implemented, additional underway, continuous activity Listserve continues to grow	--
15. Director of OFSR, in collaboration with the Area ED, should review the current procedures and, where appropriate, additional steps should be implemented to continue to improve the efficiency and effectiveness of the recall verification process	Complete	--
16. VP Programs, in collaboration with VP Operations, should clarify the responsibilities for follow-up, including the development of strategies to systematically address repeat recall offenders and reoccurring food safety issues through all programs	Complete Follow-up responsibilities clarified and documented in updated version of FERM	--
17. Area EDs, in collaboration with ED Food Safety, should develop and implement measures to improve the completion and the tracking of follow-up activities to be conducted by Area Field Staff for the registered and non-registered sector	Complete New version of IMS contains mandatory fields for follow-up	--
18. Director OFSR should continue with improvements to the IMS	Complete New version released 2008	--
19. Area EDs should continue to implement measures to improve the quality of the information entered in the IMS	In progress Some Areas have addressed. QMS may contribute.  New version of IMS has greater data controls. Quality issues with commodity inspectors with less frequent use of IMS. IMS not used by field staff in some programs	Yes
20. ED Food Safety should ensure appropriate bilingual capacity is available in OFSR and for the technical assessments	Complete	--
21. ED Food Safety should ensure key documents are available in both official languages	Complete	--
22. Director OFSR, in collaboration with CPRA, should implement a formal feedback mechanism to ensure necessary improvements are reported and actioned in a timely manner	Complete Management response completed. Follow-up reports to SCARM until May 2006. Additional follow-up in 2008	--