Evaluation of the Feed Program

Management Response

Prepared for Audit, Evaluation and Risk Oversight
Canadian Food Inspection Agency

Final Report
August 2007

---

Table of Contents

• Executive Summary
• 1. Introduction and Context
  • 1.1 Program Description
  • 1.2 Program Approach and Reach
  • 1.3 Program Resources
  • Notes
  • 1.4 Evaluation Context
  • 1.5 Structure of the Report
• 2. Evaluation Methodology
  • 2.1 Evaluation Framework
  • 2.2 Data Collection through Multiple Lines of Evidence
  • 2.3 Data Quality and Limitations of the Evaluation
• 3. Relevance
  • 3.1 Core Feed Program and Enhanced Feed Ban
  • 3.2 Medicated Feeds Activities under APF
  • 3.3 Conclusions
• 4. Success
  • 4.1 Core Feed Program
  • 4.2 Medicated Feeds Activities under APF
  • 4.3 Conclusions
• 5. Design and Delivery
  • 5.1 Core Feed Program
  • 5.2 Medicated Feeds under APF
  • 5.3 Conclusions
• 6. Continuous Improvement
  • 6.1 Overall Program
  • 6.2 Medicated Feeds Activities under APF
  • 6.3 Conclusions
• 7. Conclusions and Recommendations
  • 7.1 Conclusions
  • 7.2 Recommendations
• Annex A – Advisory Committee
• Annex B – Evaluation Matrix
• Annex C – Listing of Documents Reviewed
• Annex D – Interview Guides
• Annex E – Case Study

---

Executive Summary

The objective of the evaluation is to evaluate the success of the Feed Program in achieving its stated objectives; assess its relevance, program design and delivery; and provide recommendations for improvements as necessary. The evaluation is formative in nature and process-based, and recommendations will focus on ways to improve the overall program effectiveness. The purpose of the evaluation is to determine the effectiveness of the Feed Program’s structure, procedures and inspection activities; to determine the success and impacts of the Feed Program relative to its stated programmatic objectives; and to propose improvements to the program’s structure, procedures and inspection activities.

Background

The CFIA administers a national livestock Feed Program to verify that livestock feeds manufactured and sold in Canada or imported are safe, effective and labeled appropriately. The program is reflected in four separate sub-sub activities of the CFIA Management Resources Reporting Structure (MRRS): enhanced Bovine Spongiform Encephalopathy (BSE) programming, feed (including the Agricultural Policy Framework (APF) medicated feeds), product assessment and approvals, and regulation of products of biotechnology.

Objective

The evaluation was conducted during the spring and summer of 2007, and it covered the five-year period from 2001/2002 to 2006/2007. This represents a time of significant change and growth within the Feed Program.  Historically, the core Feed Program was a relatively low visibility program with a limited number of resources in both programs and Operations Branches.  The CFIA received, during this period, additional resources related to BSE and APF programming.

This evaluation is the first assessment of the Feed Program.  An evaluation framework had not been developed for this program, and the first step in this assessment was development of a Logic Model and Evaluation Framework.

The evaluation was conducted using multiple lines of evidence, including:

• document and file review;
• review of web publications;
• staff and management interviews;
• area and facility visits;
• case studies; and
• external stakeholder interviews.

The evaluation provides conclusions on four areas of interest: relevance, success, program design and delivery and continuous improvement. Four recommendations are presented to address the most significant issues found.

Conclusions

Overall, the Feed Program is a key contributor to the CFIA mandate and has competent and dedicated staff; however, effective service delivery has been challenged due to a lack of clear vision and direction, management shortcomings and a poor working relationship with the regulated parties.  The activities under the APF funding have significantly failed to meet established timelines.

Relevance

The program is consistent with government and Agency priorities and fills an important role for ensuring food safety and the health of Canada’s livestock population.  The vision of the program has not been adequately articulated and agreed upon.

Success

Mill and rendering plans inspections occur broadly at the targeted frequency but retail and on-farm inspections are occurring inconsistently across the areas. Inspections are supportive of the program objectives. Product registrations are not meeting the defined service standard.  Feed ban activities are on schedule to meet the July 12, 2007, rollout of the enhanced feed ban. Medicated feed activities and outputs of the APF MOU are significantly behind schedule. Significant effort is still required to bring the Medicated Feeds Regulations to implementation. The poor relationship with ANAC, outdated Feeds Regulations, insufficient IT systems and delays in the Medicated Feeds Regulations are limiting the success of the program.

Design and Delivery

The Health of Animals Regulations are current, but the Feeds Regulations are outdated. The feed inspection program is based on a process-based model; however, it is not risk-based.  Resources are appropriate and planning and priority setting is occurring at the tactical level. Strategic level planning and priority setting across the Feed Program is limited. Feed Program coordination between internal CFIA units is poor. The historical direct line of communication between the Feed Section and the inspectors is no longer effective or appropriate.  Product assessment procedures and sampling are well-documented and understood. Processes, frequency and monitoring are clearly understood and followed for commercial mills and rendering plants, but there is a lack of clarity related to the frequency and follow-up procedures for retail establishments and for farms which mix feeds.

Continuous Improvement

The CFIA has implemented a quality management system which has identified key areas for improvement; however, implementation of the identified recommendations need a greater focus. Overall weaknesses exist in performance monitoring for the purposes of informing program decisions. There is some evidence of leveraging best practices originating within CFIA, but there is limited use of best practices from other regulatory departments within Canada or internationally.  Risk assessment and mitigation are evident in specific elements of the program; however, formal risk assessment is limited in the overall design of the inspection programs.

Recommendations

The most pressing actions for the Feed Program are to resolve strategic direction and planning issues that have resulted from a long period of relative inactivity in program management, coupled with explosive recent growth in the program. The key recommendations are:

1. Develop and communicate a strategic policy direction for the Feed Program

   For the core Feed Program, a strategic plan should be developed that articulates a clear policy direction and prioritizes the various initiatives. Specific actions should include:

   • Establishment of clear executive engagement and support;
   • Commitment to integrating a HACCP approach with the core Feed Program and the Medicated Feeds Regulations;
   • Communication of the Feed Program policy direction and vision:
     • Internal to CFIA, in particular down the reporting chain of command; and
     • With key stakeholders, including feed and rendering facilities, industry associations, HC, AAFC, farming community, and retailers.
2. Strengthen program leadership and leverage executive support for key strategic initiatives

   For the core Feed Program, strengthen program leadership to:

   • Resolve governance in terms of decision making authority, roles and responsibilities;
   • Develop defined roles and responsibilities, including membership and objectives of the industry liaison Working Group; and
   • Resolve the organization context for the Feed Program in terms of either a Division or a Section.

   For the core Feed Program, develop comprehensive performance monitoring and reporting to monitor the effectiveness of the program and inform decision making.  Place more emphasis on the implementation of the findings from the QMS process.

   For the APF, strengthen accountability and complete the activities outlined in the APF related to the development and implementation of Medicated Feeds Regulations as a high priority.  Engage and leverage executive support toward rapid closure of outstanding tasks. Develop a strategy to complete the activities in light of the expiration of the APF funding in March 2008.

3. Promote an ongoing, constructive professional relationship with stakeholders

   For both the core Feed Program and the APF activities, the Agency must build on the recent positive steps taken to engage key stakeholders including the regulated parties and industry associations.  The Agency needs to:

   • Work more diligently to nurture a culture of mutual respect and trust;
   • Promote the principles of effective regulatory consultation – openness, honesty and forthrightness, inclusiveness, transparency – as laid out by the Privy Council in the Guidelines for Effective Regulatory Consultations; and
   • As a first step, task the industry liaison Working Group to develop a strategy to engage stakeholders constructively and demonstrate commitment to the process.
4. The CFIA should modernize the program design over time

   In light of the significant recent and ongoing changes in the program, modernize the core Feed Program design in a staged, risk-based approach.  This should include the following activities:

   • Review existing inspection, sampling, labelling and registration activities from a risk assessment basis;
   • Finalize key decisions regarding HACCP and its role in program delivery;
   • Review, develop and promulgate inspection frequencies and criteria for retail and on-farm inspections;
   • Engage the Office of the Chief Information Officer to enhance IT tools, especially for reporting; and
   • In the longer term, update the Feeds Regulations to reflect the current environment.

---

List of Acronyms and Abbreviations

AAFC

Agriculture and Agri-Food Canada

AMP(s)

Administrative Monetary Penalty

ANAC

Animal Nutrition Association of Canada

APF

Agricultural Policy Framework

BSE

Bovine Spongiform Encephalopathy

CBS

Canadian Biotechnology Strategy

CFIA

Canadian Food Inspection Agency

CGC

Canadian Grain Commission

CIO

Chief Information Officer

CMIB

Compendium of Medicated Ingredient Brochures

CMS

Client Management System

CRSB

Canadian Regulatory Strategy for Biotechnology

DPR

Departmental Performance Report

EU

European Union

FDA

US Food and Drug Administration

GMP(s)

Good Management Practice(s)

HACCP

Hazard Analysis and Critical Control Point

HC

Health Canada

HR

Human Resources

KPI

Key Performance Indicators

MCAP

Multi-Commodity Activities Program

MFP

Medicated Feed Program

MOU

Memorandum of Understanding

MRRS

Management Resources and Results Structure

OCIO

Office of the Chief Information Officer

OECD

Organization for Economic Cooperation and Development

OFFSRP

On-Farm Food Safety Recognition Program

QMS

Quality Management System

QUAD

Quadrilateral (Canada, US, New Zealand and Australia)

SGS

Société Générale de Surveillance (goes by initials only)

SRM

Specified Risk Material

TB

Treasury Board

TBS

Treasury Board Secretaria

USDA

US Department of Agriculture

VDD

Veterinary Drugs Directorate (of Health Canada)

---

1. Introduction and Context

1.1 Program Description

Under the authority of the federal Feeds Act and Health of Animals Act, CFIA administers a national livestock feed program to verify that livestock feeds manufactured and sold in Canada or imported into Canada are safe, effective and are labelled appropriately. The Feed Program works to ensure safe feeds in order to maintain the safety of meat, milk and egg products.

For the purposes of this evaluation, the CFIA activities in the Feed Program have been broadly broken down into the following categories:

• program design and management
• product/establishment assessment, permitting, and licensing
• verification and compliance activities
• laboratory testing and analysis
• stakeholder education and awareness

Program Design and Management includes the assessment and development of regulations and standards related to feed. This includes standard setting for feed and feed ingredients, standards for establishment for permits, licences and certification.  This also includes the development of the associated inspection programs for implementation within the Agency and entails the development of procedures manuals, training packages, standard operating procedures, and systems development in support of the program.

Product/Establishment Assessment, Permitting, and Licensing includes activities related to the evaluation and approval of ingredients for use in livestock feeds and the evaluation and registration of specialty products of specific safety or efficacy concern. It also addresses the licensing of establishments that prepare animal feeds.

Verification and Compliance activities relate to both product and establishment verification.  The Feed Program monitors feeds via random sampling and analysis for the presence of residues of chemicals, pesticides, contamination by heavy metals, biologicals (e.g. mycotoxins and salmonella) and to verify drug guarantees in feeds.  In addition, the Agency carries out facility inspections at commercial feed mills, farms that mix feeds, retail outlets and rendering facilities.  In 2003, 1,200 inspections were performed according to the tracking system used by Operations Branch. In 2006, that number has increased to over 2,600, as an additional 216 staff were added. Compliance actions are taken where instances of non-compliance are detected and may include the use of warning letters, product detention, administrative monetary penalties (AMPs) and prosecutions.  Investigations are conducted in response to detections of contamination of meat, milk or eggs and producer complaints related to feed, conducted at both commercial feed mills and on farm.  Label reviews of medicated feeds are conducted to verify that the proper level of medication is provided and that all applicable cautions and warnings are provided.

Laboratory Testing and Analysis is conducted as required to analyze for residues, contamination and drug levels.  Between 20 and 25 staff are involved in the analysis of feed samples with the majority of this work done in the Carling laboratory in Ottawa. This analysis is supported by methods development to expand the scope of hazards which can be covered, improved methodology for quicker turnaround time, developing multi-residue methods and improved sensitivity.

Stakeholder Education and Awareness is conducted to ensure that industry is consulted about potential changes to standards or regulations, and, that regulations and standards are communicated.

The immediate outcomes of the Feed Program are animal feed regulations that are efficiently and effectively designed and managed.  Industry awareness, understanding and compliance with the Feeds Regulations are also expected outcomes.  The feed industry should experience fair, transparent and effective product evaluation and establishment licensing/permitting while at the same time appropriate regulatory controls against feed products are that are non-compliant.

Enhanced BSE Program and the Repositioning the Canadian Beef and Cattle Industry and Strengthening Animal Feed Restrictions Program.

Since the discovery of bovine spongiform encephalopathy (BSE) in Canada, additional resources have been allocated and enhanced feed ban controls are being implemented. Specifically, CFIA received additional funds as part of the enhanced BSE program that was targeted at the Feed Program, along with the Meat Hygiene and Animal Health Programs.  Funding for strengthening animal feed restrictions was included initially for two years.  Since that time, funding has been requested and received to maintain the program on an annual basis, with permanent funding recently approved.

From an animal feeds perspective, Canada implemented a feed ban in 1997 to limit the potential spread of BSE, should it exist in North America. The ban is intended to prohibit the feeding of ruminant animals (e.g. cattle, sheep, deer) with animal proteins of mammalian origin. In 2003, CFIA moved to publish regulations to enhance the feed ban in July 2006 to require the removal of bovine specified risk material (SRM) from the animal feed chain.  The regulations begin to come into force in July 2007.

The immediate outcomes are that the Agency develops an increased capability and capacity to verify that SRM is identified and segregated from animal feed and that prohibited animal proteins are not fed to ruminants.

Agricultural Policy Framework (APF)

Related to the APF, CFIA also received additional resources in 2004 for the Medicated Feed Program (MFP) and the On-Farm Food Safety Recognition Program (OFFSRP). This evaluation will examine the Medicated Feed Program component only, initially budgeted at $17.0M over five years.

In the winter of 2000, CFIA published proposed Regulations Respecting the Making of Medicated Feed, under the federal Health of Animals Act. The regulations were based on Hazard Analysis Critical Control Point (HACCP) principles and were to require licensing and increased control measures by anyone mixing medicated animal feeds in Canada. The initiative provided for implementing controls aimed at reducing potential errors in the food production chain. The proposed regulations were to provide a mechanism for CFIA to facilitate trace-backs in the event of a possible contamination.  These regulations went to Gazette I in 2000, but were not pursued at that time.

In 2003, CFIA committed to completing the following activities related to medicated feeds under an MOU for the APF Initiatives Development signed between AAFC, CFIA and Treasury Board Secretariat (TBS):

• promulgate regulations establishing minimum standards for the manufacturers of medicated feed for food-producing animals;
• develop inspection protocols and compliance strategies designed to achieve compliance with the Medicated Feeds Regulations for delivery by certified CFIA inspectors and accredited 3rd party inspectors;
• CFIA inspections will be augmented by the On-farm Food Safety Program administered by national producer groups; and
• establish standards and supporting the design and implementation of an inspection and licensing system for these feed mills and farms by way of evaluating their manufacturing procedures and records.

Feed Program Logic Model

Core Feed Program Outputs

Program Design and Management	Program Delivery			Education and Awareness
	Product/ Establishment Assessment, Permitting, Licensing	Verification and Compliance Activities	Laboratory Testing and Analysis
Standards for feed ingredients Policies Nutritional criteria for the exemption of feeds from registration	Approval of ingredients Registering specialty products Rejections	Commercial feed mill, rendering plant, retail and on-farm inspections Investigations Label review of medicated feeds Monitoring Compliance actions - warning letters, detentions, AMPs, prosecutions Export certificates	Analysis for residues, contamination and drug levels Methods development	Stakeholder consultation Education Communication

Medicated Feed Outputs

Program Design and Management	Program Delivery			Education and Awareness
	Product/ Establishment Assessment, Permitting, Licensing	Verification and Compliance Activities	Laboratory Testing and Analysis
Regulations promulgated in Gazette I and II National training and inspection program Licensing office operational Lab accreditation system Systems development - MCAP, CMS, licensing dbase	Licences, rejections	Feed mill, on farm and feed retail inspections Compliance actions - warning letters, detentions, AMPs, prosecutions	Lab accreditation system Accredited labs Third party lab testing results	Stakeholder consultation Education Communication

Enhanced BSE Outputs

Program Design and Management	Program Delivery			Education and Awareness
	Product/ Establishment Assessment, Permitting, Licensing	Verification and Compliance Activities	Laboratory Testing and Analysis
Regulations promulgated in Gazette II Program design for new elements Risk assessments Training materials, sessions Additional recruited, trained, equipped  inspection staff	Permits	Inspections Compliance actions - warning letters, detentions, AMPs, prosecutions	Suite of laboratory analytical methods Sampling, laboratory testing Methods development	Stakeholder consultation Education Communication

Immediate Outcomes

Program Design and Management	Program Delivery			Education and Awareness
	Product/ Establishment Assessment, Permitting, Licensing	Verification and Compliance Activities	Laboratory Testing and Analysis
Animal feed regulations are strengthened Efficient and effective design and management of Feed Program	Feed industry experiences a fair, transparent and effective product registration and producer licensing process	Industry compliance with feed regulations, enhanced BSE control measures and Medicated Feeds Regulations Increased level of verification that SRM  is identified and segregated from animal  feed Increased level of verification that prohibited animal protein is not fed to ruminants	Accurate and timely lab results	Industry awareness of feeds and  Medicated Feeds Regulations Industry awareness of BSE control measures Industry adoption of GMP

Intermediate Outcomes

• Transmission of BSE related diseases to humans is prevented
• Feed products meet requirements of federal acts and regulations
• Feed industry complies with federal acts and regulations
• Industry is supported in adopting risk management practices

Agency Strategic Outcomes

• A sustainable animal resource base
• Protection from preventable health risks related to food safety or the transmission of animal diseases to humans

1.2 Program Approach and Reach

Within CFIA, the core groups involved in the program are the Feed Section of the Animal Health and Production Division of Programs Branch, Operations Branch and Science Branch, both labs and risk assessment. Interaction with a number of other functional units occurs within CFIA, including Regulatory Affairs, HR, Finance, IT and International Affairs.

Other key federal players include Agriculture and Agri-food Canada (AAFC) and Health Canada (HC). Internationally, the Feed Program works with the Association of American Feed Control Officials, the US Food and Drug Administration (FDA) and the US Department of Agriculture (USDA). The provinces and territories have limited direct involvement with the federal Feed Program. Industry associations such as the Animal Nutrition Association of Canada (ANAC) and producer associations such as the Canadian Cattlemen’s Association, the Canada Pork Council, the Chicken Farmers of Canada, and the Dairy Farmers of Canada are also key stakeholders.

The reach of the program covers all of the rendering facilities in Canada (approximately 35), the commercial feed mills (estimated between 300 and 550), animal feed retailers (up to 1,300) and farms that mix their own livestock feeds. In addition, CFIA has a role in monitoring the import of animal feeds and provides export certification for exporters.

1.3 Program Resources

The Feed Program resources are reflected in four separate sub-sub-activities of the CFIA Management Resources Reporting Structure (MRRS):

• enhanced BSE programming;
• feed (including APF – Medicated Feeds);
• product assessment and approvals; and
• regulation of products of biotechnology (includes Canadian Regulatory System for Biotechnology (CRSB) and Canadian Biotech Strategy (CBS).

Of these four sub-sub-activities, only one sub-sub activity, Feed, contains resources related to the Feed Program alone. The other sub-sub-activities include other programs such as Meat Hygiene, Animal Health and Fertilizer.

Additional resources that have been received by the program as part of the following larger submissions:

• CFIA received an additional $28.2M over 2004/2005 to 2005/2006 under the Repositioning the Canadian Beef and Cattle Industry and Strengthening Animal Feed Restrictions to augment the Feed Program with 216 FTEs.  This funding was extended on an annual basis for two years, with the ongoing funding recently approved.
• The total amount of resources committed to CFIA under the APF was $17.0M for a five year period starting on April 1, 2003, for the Medicated Feed Program (MFP). As of January 31, 2007, actual expenditures for the MFP have amounted to $2.994M.

The following table summarizes the actual expenditures recorded in the financial system related to three sub-sub-activities of the Program Activity Architecture: Feeds – including APF Medicated Feeds; Product Assessments; and Regulation of Products of Biotechnology. While not comprehensive, the table illustrates the growth of the program over time.

	2002/2003	2003/2004	2004/2005	2005/2006	2006/2007
Programs	932.3	1,164.9	1,873.2	5,779.7	6,455.0
Operations	2,028.8	2,434.3	2,812.2	3,553.7	3,341.6
Science	2,161.9	1,804.5	2,111.8	5,717.2	5,811.3
Fin, Admin & IT			65.1	649.7	521.1
HR	86.9	17.9	17.4	0.1	--
Other		1.8		97.7	9.8
Total	5,210.0	5,423.4	6,879.6	15,798.2	16,138.8

Notes:

1. For 2002/03 to 2004/05 includes Feed Program activity 4120 under PRAS.
2. For 2005/06 and 2006/07 includes PAA sub-sub-activities 7722 (Feeds – including APF Medicated Feeds), 7731 (Product Assessments, and 7732 (Regulation of Products of Biotechnology).
3. Does not include feed expenditures that may be coded under 7522 Enhanced BSE Programming.

1.4 Evaluation Context

The five-year period selected for this evaluation represents a time of significant change and growth within the Feed Program.  Historically, the Feed Program was a relatively low visibility program with a limited number of resources in both Programs and Operations.  The Feed Section of the Animal Health and Production Division operated very independently of the rest of CFIA.

The finding of BSE in Canada in 2003 raised the visibility of the program both within the Agency and among Canadians and became an overriding priority for CFIA. A number of new programs were developed to respond to this finding, and significant resources were received by CFIA across a number of programs. Most significantly for the Feed Program, the Repositioning the Canadian Beef and Cattle Industry and Strengthening Animal Feed Restrictions initiative provided resources in support of the enhanced feed ban regulations. International delegations were hosted on several occasions post-2003 to review Canada’s feed controls. For example, in 2005, the USDA’s review of Canadian feed controls concluded that Canada’s feed ban has significantly reduced the potential spread of BSE through the animal feed system.

Also in 2003, CFIA received resources under APF to design, develop and implement Medicated Feeds Regulations.

1.5 Structure of the Report

This report contains the following sections:

• Section 2 summarizes the evaluation methodology, including the evaluation framework, data collection methods and limitations of the evaluation;
• Section 3 presents the findings and conclusions related to the evaluation issue of relevance of the Feed Program;
• Section 4 includes the findings and conclusions for the success of the Feed Program;
• Section 5 outlines the findings and conclusions related to the design and delivery of the Feed Program;
• Section 6 provides the findings and conclusions for the evaluation issue of continuous improvement; and
• Section 7 summarizes the conclusions and the recommendations resulting from the evaluation.

2. Evaluation Methodology

2.1 Evaluation Framework

The objective of the evaluation is to evaluate the success of the Feed Program in achieving its stated objectives; assess its relevance, program design and delivery; and provide recommendations for improvements as necessary.  The evaluation is formative in nature and process-based, and recommendations will focus on ways to improve the overall program effectiveness.  The purpose of the evaluation is:

• to determine the effectiveness of the Feed Program’s structure, procedures and inspection activities;
• to determine the success and impacts of the Feed Program relative to its stated programmatic objectives; and
• to propose improvements to the program’s structure, procedures and inspection activities.

An Advisory Committee was formed to provide advice to the evaluation process at key points in the evaluation.  Annex A contains a list of the members of the Committee.

Scope of the Evaluation

The evaluation will include the core Feed Program as well as the activities related to feeds under the Canadian Regulatory Strategy for Biotechnology (CRSB) / Canadian Biotechnology Strategy (CBS), the Agricultural Policy Framework (APF) and the Enhanced BSE Programming. It will cover the period from 2001/2002 to 2006/2007.

Evaluation Issues

A list of evaluation issues was developed based on the TB Evaluation Policy, the Terms of Reference for the evaluation and the issues that the Agency would like investigated.  These are summarized below and the detailed performance indicators and data sources are outlined in Annex B.

Relevance:

• Are there clear and relevant vision and objectives for activities under this program?
• Does the program continue to be consistent with government-wide priorities and the CFIA mandate?
• Is there a continued need for the program?

Success:

• Are planned activities being implemented and producing the expected outputs?
• Are completed and planned activities meeting, or likely to meet articulated objectives?
• What are the impediments to success, if any?
• To what extent has CFIA complied with the terms and conditions set out in the MOU between CFIA and AAFC with respect to medicated feeds?
• Have there been any unexpected outputs and outcomes?

Design and Delivery:

• Are there appropriate resources, planning, priority setting, management and decision making structures in place to meet the objectives?
• Has the program coordinated activities effectively with other areas of CFIA, other federal departments, relevant provincial agencies, and international agencies, including QUAD and the EU?
• Are processes/procedures for product assessment, establishment inspection and monitoring documented and clearly understood? Are they followed?
• Is the Feed Program supported by adequate and appropriate regulations, resources and management support?
• Is the program using a HACCP-based approach and why or why not?
• Are there mechanisms to provide assurance that processes and procedures are followed and are they effective?

Continuous Improvement:

• Are ways to improve program delivery, quality and consistency being considered and implemented in support of continuous improvement?
• Is the program considering and implementing best practices and lessons learned from other regulatory programs, other Federal/Provincial governments and international partners?
• Is risk assessment and mitigation considered in ongoing program design and management?

2.2 Data Collection through Multiple Lines of Evidence

Data collection consisted of the following elements:

• document review including manuals, program files, training materials, inspection forms, Departmental Performance Report (DPR), Feed Ban Review, APF MOU, Quarterly APF Progress Reports, etc. (See Annex C for a listing of the key documents);
• interviews with CFIA staff (See below for a list of the groups of interviewees, and Annex D for the interview guides);
• interview with functional managers/staff (e.g. Finance, HR, Office of the CIO); and
• interviews with key external stakeholders including other government departments, industry representatives and facility owners and management;
• file review;
• site visits to 3 area/district offices: Calgary, Guelph and Sainte-Hyacinthe;
• visits to selected feed mills, rendering plants and retail establishments; and
• case study (see Annex E).

Interviews by Group	Number
Internal – Programs
Feeds Program	5
Program Network	1
Senior Management	2
Internal – Operations
Inspectors	8
Supervisors, Inspection Managers	6
Regional Program Officers	2
Regional Directors	3
Senior Management	5
Other	3
Internal – Other
Functional Units – IT, Finance, Regulatory and Parliamentary Affairs, Science	4
External
Other Government Departments	2
Industry Associations	1
Feed Mill, Rendering Plant Management/Owners	7
Grand Total	49

2.3 Data Quality and Limitations of the Evaluation

There is a lack of performance information related to the Feed Program.  There are no commonly accepted definitions for facility compliance and the data contained in MCAP has limitations in terms of completeness and is difficult to analyze.  This has resulted in a greater reliance on qualitative data sources.

Due to time and budget limitations, site visit to Atlantic was not possible; however, this was replaced with telephone interviews.  A limited number of external stakeholders were interviewed.

The Feed Program is in a state of change and as such, a number of initiatives are in progress and the final outcomes are not yet realized.  Where possible, the evaluation notes the initiatives and their likely contribution to success.

3. Relevance

This portion of the evaluation assesses the relevance of the program in the context of CFIA’s mandate and mission, Government of Canada priorities and its perceived relevance to regulated parties, and other stakeholders. Findings are derived primarily from key staff and stakeholder interviews and reviews of relevant government and Agency plans and priorities as well as the applicable acts and regulations.

3.1 Core Feed Program and Enhanced Feed Ban

Are there clear and relevant vision and objectives for activities under this program?

The key objectives of the program are well accepted as safer food, animal protection and consumer protection although different stakeholder groups rank the importance of the three differently. All three objectives are recognized by all stakeholder groups. (interviews)

The program lacks a firmly articulated vision and objectives statement that is issued by an appropriate authority. No such document was found in the document review. (document review)

The Feed Program attempted to develop and implement a risk-based approach using a HACCP model in the late 1990s; however, this was dropped due to competing priorities. This lack of success on the Agency-led approach of risk-based inspection can be attributed, in part, by a lack of a recognized vision that clearly outlined the future role of HACCP and the roles and responsibilities of the stakeholders. (interviews)

The commercial feed industry received mixed communications on the degree to which inspections would change after adoption of their own HACCP program, FeedAssure™. Different messages were communicated to industry resulting in the assumption that by adopting FeedAssure™, they would possibly be subject to significantly fewer inspections and less rigorous inspections. (interviews)

The objectives and vision for the Enhanced Feed Ban initiative are clearly articulated and well understood. (interviews, document review)

Operations staff including inspectors, inspection managers and inspection supervisors had varying degrees of understanding of the future roles of inspections in support of the Feed program. They were unsure of whether inspections frequencies would increase, decrease or stay the same. They were unsure of the evolution of on-farm inspections. They expressed difficulties in communicating to regulated parties (mill owners) the rationale for Good Management Practices (GMPs) in the inspection. (interviews)

In a Case Study of a new form rolled out during the period of the evaluation, Operations Branch reversed a Programs Branch release of a new inspection form. In part, this could be tied to a disagreement on the final vision for the program and an inability to compare the new form against the stated vision to validate if it was indeed a sufficient improvement and advancement of the vision. (Case Study)

Does the program continue to be consistent with government-wide priorities and the CFIA mandate?

The Feed program can be found within published Agency priorities. The activities of the core Feed program can be traced to specific requirements in the Health of Animals and Feeds Acts. All outcomes can be linked with the overall strategic objectives of CFIA as identified in the MRRS. The Program Activity Architecture contains four specific sub-sub activities related to the Feed program: enhanced BSE programming, feed (including medicated feeds APF), product assessments and approvals (non-biotechnology), and regulation of products of biotechnology. (document review)

Several recent investments have been made into the program (APF, Enhanced BSE), demonstrating the fit of the program to government priorities. In particular, the APF funding shows the growing importance of the Feed Program in the overall policy framework of the government. (document review)

Senior executives all agreed that the Feed Program was a critical element of the Agency’s programming. Likewise, inspectors, inspection managers and regional manager officers as well as team members of the Feeds Section agreed that the program links well with agency priorities. (interviews)

The Agency reports on some of the Feed Program compliance rates in the Departmental Performance Report (DPR) and that inclusion presumes that the efforts of the program are visible to parliamentarians and therefore considered useful and consistent with other government programs. (document review)

Is there a continued need for the program?

The published compliance rates of high-risk tasks at mills and rendering plants exhibited an initial improvement, and have remained fairly constant over the last two years while inspection numbers have been rising in the past two years. This suggests that the program is still needed. (document review).

Compliance Rates by Year	2002/2003	2003/2004	2004/2005	2005/2006
Mills	92%	97%	95%	96%
Rendering Plants	90%	97%	93%	93%

Source: CFIA DPR

Inspectors and inspection managers feel that the inspections generate true worth and need to be continued. This was particularly true for rendering plants and mills where Specified Risk Material (SRM) is used. Industry, while concerned with the level of regulation, recognizes the value of a regulatory program to contribute to food safety, animal health and trade requirements.  The need for the amount of regulation around product registration is questioned by industry. (interviews)

Recent events such as melamine poisoning in feeds and dioxin contamination of feed additives have demonstrated the continuing need for the program (interviews)

Inspections of mills and rendering plants are an essential part of the control of BSE. (document review)

3.2 Medicated Feeds Activities under APF

Are there clear and relevant vision and objectives for activities under this program?

Under the APF, the vision for the Medicated Feeds activities shows a process-based inspection regime, supported by licensing of feed mills using medications, independent testing laboratories and validation of scales and mixing equipment. The MOU among Agriculture and Agri-Food Canada (AAFC), the Treasury Board Secretariat (TBS) and the Agency is quite clear and lays out expectations for activities of the Agency under the agreement. (document review)

AAFC does monitor the Agency’s progress on its commitments under the APF. They have focused on other areas of the agreement and not pressed the Agency on the time lapse for development of new Medicated Feeds Regulations until recently. (interviews)

In interviews with Feed Section staff, the expectations of the MOU seemed to be well understood as was the failure to meet timings within the MOU. (interviews)

Does the program continue to be consistent with government-wide priorities and the CFIA mandate?

The government’s Agricultural Policy Framework (APF), which outlines the current priorities related to agricultural policy, included provisions and funding to develop and implement Medicated Feeds Regulations under the Health of Animals Act.  This remains a key priority for CFIA as outlined in its 2007/2008 Report on Plans and Priorities. Medicated Feeds are a recognized activity under the Feeds sub-sub activity of the MRRS. (document review)

Is there a continued need for the program?

The requirements of the MOU have not yet been met. There is still a need for this activity as there have been no mitigating factors since the MOU has been developed which would remove its need or relevance. (document review)

3.3 Conclusions

We find that while the overall objectives of the Feed Program are clear, the vision is not well-articulated or understood.  The objectives of the program are well understood and accepted. The future vision for the various components of the Feed Program varies considerably. There is a different vision between the executive and managerial levels of the Agency, between the mill owners and the Agency and between Programs Branch and Operations Branch staff, in general. The Enhanced BSE and APF mandated portions of the program have a well understood vision.

The Feed Program is a critical component of the Agency’s mandate to ensure safe foods and to protect animal health. The various components are consistent with government priorities and are mandated by the Health of Animals Act and the Feeds Act.

Overall, the various components of the Feed Programs fill a recognized and valued need. They are also critical to the eradication of BSE from the bovine herd.

4. Success

This portion of the evaluation contains an assessment of the success of the Feed Program in meetings its stated objectives. It examines the activities that are performed against the plans, the outcomes of those activities and the overall impact of the program. Findings are derived primarily from the document reviews, site visits, and key staff and management interviews.

Key performance indicators (KPI) are reported quarterly since 2005/2006. The operational data is maintained within the Multi-Commodity Activities Program (MCAP) and traditionally it has been difficult to get statistics from that system and the data is not very reliable. For example, a simple report detailing the time between opening an inspection and closing one showed records that have a closing date before the opening data. There are insufficient data edits upon entry to ensure reliable data. The CFIA reports on compliance rates for high-risk tasks at feed mills and rendering plants, however, this is not fully accepted across the program as a definition of compliance.  Only recently have non-compliances been categorized as minor, major and critical with definitions of each categorization in a new inspection form not yet released.

4.1 Core Feed Program

Are planned activities being implemented and producing the expected outputs?

The following table lists the activities for the Feed Program components except the Medicated Feed portions as listed in the Logic Model. The final column provides a quick assessment of whether they are being implemented. More detailed text follows for items in bold in the middle column.

Component	Activities/Outputs	Are they being implemented?
Program Design and Management	Standards for feed ingredients Policies Nutritional criteria for the exemption of feeds from registration	The CMIB addresses medication standards. Feed ingredient standards are set out in schedules under the regulations Some minor policies are being produced beyond the enhanced BSE policies No significant activity for exemptions
Product/Establishment Assessment, Permitting, Licensing	Approval of ingredients Registering specialty products Rejections	Approvals have a backlog, but that is being reduced. From 2005 to 2007, less than half of all registrations met the performance target of 90 days, due to resourcing levels  Specialty products are approved through the registration process using procedures developed within the Feed Section Rejections occur for approximately 5% - 10% of registrations
Verification and Compliance Activities	Commercial feed mill, retail and on-farm inspections Investigations Label reviews Monitoring Compliance actions - warning letters, detentions, AMPs, prosecutions Export certificates	Approximately 80% of facility inspections occur according to plan, with the bulk of mill and rendering inspections completed and on farm and retail inspections varying by region due to a lack of understanding or commitment to firm inspection guidance for frequency and incomplete lists of retailers and farms Inspections in total doubled in 2006 and into 2007 Investigations are performed when required Label reviews are being completed during inspections Monitoring is done through inspections and sampling activities Compliance actions taken for non-conformities Export certificates issued in a timely fashion
Laboratory Testing and Analysis	Analysis for residues, contamination and drug levels	Backlog exists for feeds testing but not to the extent that they undermine the statistical requirement for process testing
Education and Awareness	Stakeholder consultation Education Communication	Various outreach activities occur. Stakeholder groups feel they are not consulted on program changes Significant BSE activity: consultation sessions, pamphlets produced, various town hall meetings

Policies

Enhanced BSE regulations have resulted in new policies. Other than this, the only policies issued in recent years focus on single issues, as follows:

• Import Policy for Bovine Animals and their Products;
• Addition of Pet Food as an Ingredient for Use in Livestock Feeds; and
• Feeding of Poultry Manure to Cattle

Policies on broader issues such as sequencing and flush out at mills or recognition of HACCP programs, were limited. (document review)

Approval of Ingredients

Ingredient approval through registrations is performed at headquarters. Between 90% and 95% of these registration requests are approved. The Feed Section tracks the time to complete the approval process and they have failed to meet the service standard of 90 days but have shown a steady improvement in recent years. In 2005/2006 the KPI showed an approximately 38% compliance rate with the service standard. In 2006/2007, this increased to approximately 42%. Historical data before 2005 was not available. The tracking process considers the input from HC and this time is not “charged” to the Agency. (document review, interviews)

There is a backlog of renewal registrations. The Feed Section prioritizes new approvals ahead of renewals. The backlog is being reduced over time and existing approved products are allowed to continue being marketed pending renewal of their registration. This produces a minor risk of a dangerous product remaining on the market for longer but the risk is small as the product has been approved once before. The practice allows industry to continue operations despite the backlog. (interviews)

Commercial Feed Mill, Retail and On Farm Inspections

KPI reports from 2006/2007 indicate that approximately 80% of all inspections in the plan are actually performed. Interviews with operations staff suggests that the bulk of mill inspections and all rendering plant inspections occur according to the established frequency. No inspection frequency has been set for on-farm and retail inspections and this varies across areas. (interviews, document review)

The following table lists inspection forms submitted to the MCAP system:

2003	2004	2005	2006	2007 (Jan to June)
1,225	1,145	1,121	2,635	1,252

In visits to three regions, a great disparity was observed in how farms that mix feeds are selected for inspection. There is also no comprehensive database of farms for inspection although each regional office appears to have its sources for potential farms to be inspected. (site visits, interviews)

There was a report of one district office deciding not to conduct the inspections until further work was done on the protocols. Their rationale was that the inspections are unwelcome to the producers and the expectations of the inspection not sufficiently clear for the inspector to explain to an uncooperative producer. (site visit and interview)

Analysis for Residues, Contaminations and Drug Levels

Science Branch acknowledges that there is a backlog of tests in support for the Feed Program. There are no other certified labs that can address the backlog and the lab would require more resources both for tests and methods development to fully address this backlog.  (interview)

Stakeholder Consultation

Large mill owners are represented through the Animal Nutrition Association of Canada (ANAC). Historically, dialogue between CFIA and ANAC has been strained at times. Currently, dialogue with ANAC occurs at two levels. A Working Group consisting of Executive Directors has been tasked with executive dialogue with ANAC and the Feed Section also maintains a technical-level dialogue. During the evaluation, the team received reports of a number of consultation occasions between both the Feed Section and ANAC and also the Working Group and ANAC. (observation, interviews)

Industry has expressed concerns that many of the above mentioned communication activities, particularly around the new Medicated Feeds Regulations, have been one-sided and that their opinions were not fully considered. Concerns were expressed about the disconnect between strategic direction and practical implementation; industry accepts the overall strategic objectives but feels the proposed technical solutions do not consider all alternatives and the viewpoint of industry. (interviews)

Fewer opportunities exist for consultations with retailers. Producer consultations are done through the national associations. (interviews)

Significant effort has been put into stakeholder consultation in support of the Enhanced BSE Feed Ban. This effort included creation of pamphlets, town hall meetings, Web postings, etc. Industry expressed appreciation for the BSE communication activities of the Task Force. (interview)

Are completed and planned activities meeting, or likely to meet articulated objectives?

The following table lists the outcomes for the Feed Program components excepting the Medicated Feed portions as listed in the Logic Model. The final column provides a quick assessment of whether they are being met. More detailed text follows for items in bold in the middle column.

Component	Outcomes	Are they being met?
Program Design and Management	Animal feed regulations are strengthened Efficient and effective design and management of Feed Program	Feed regulations around BSE are strengthened Management has been uncoordinated and ineffective
Product/Establishment Assessment, Permitting, Licensing	Feed industry experiences a fair, transparent and effective product registration and producer licensing process	Industry has expressed concerns about transparency and timeliness of registrations. Rendering permits issued in timely fashion
Verification and Compliance Activities	Industry compliance with feed regulations, enhanced BSE control measures Increased level of verification that SRM  is identified and segregated from animal  feed Increased level of verification that prohibited animal protein is not fed to ruminants	Enhanced BSE control measures not yet implemented. Reported compliance rates for mill inspections have not changed Not yet implemented Not yet implemented
Laboratory Testing and Analysis	Accurate and timely lab results	Lab results are considered accurate but not timely Areas for method development both enhanced sensitivity as well as expanded scope of testing has been identified as a gap
Education and Awareness	Industry awareness of feeds regulations Industry awareness of BSE control measures	Commercial mills are aware of existing feeds regulations. Producers and retailers have lower levels of awareness Industry well aware of BSE measures

Efficient and Effective Design and Management of Feed Program

Management involves the recognition of goals and objectives and the coordination of approaches to meet these goals. The Feed Program lacks goals at the outcome level. Any goals that exist are for outputs only. Two examples of well established goals are the 90 day completion of an ingredient registration and the goal to visit all high-risk mills (those that use restricted materials) four times per year. While high-risk task compliance rates are reported in the DPR, there are no compliance rate targets and or meaningful tracking and analysis year by year of compliance. (interviews, document review)

The Feed Program has missed many significant deadlines for items such as training materials, licensing procedures for mills and the new inspection forms. Target dates have been reset several times.  Although external factors have played a significant part in this, the program management has demonstrated an inability to overcome these external influences. (interviews)

The program has a history of lacking executive oversight. It has been left “on its own” during periods when there were no significant issues. The National Manager, Feed Section, reports directly to the Executive Director, bypassing the Animal Health and Production Division Director at executive request. (interview)

Operational management within the Feed Section appears sound with a great many task assignment documents, internal status reports, written procedures and protocols and decision making forums. (interviews)

Feed Industry Experiences a Fair and Effective Product Registration

ANAC members feel that the registration process is not transparent. They cite examples where a registration is accepted for one producer and rejected for another without sufficient explanation of the differences between the two. There is also a perception that some registration requirements are onerous. An example was given where whole eggs are accepted as an ingredient but egg components (e.g. egg yolks) still required evaluation and registration. It was also suggested that CFIA does not, in its registration process, consider rulings in other countries. The registration process in Canada is described by ANAC as being onerous and, as a result, Canada is the last country manufacturers go to for registrations, sometimes opting not to sell in Canada to avoid the process. (interviews)

Industry Compliance with Feeds Regulations

It is generally believed that compliance rates have been steady over the years. An exact definition of compliance has not been established and agreed upon by the various parts of the Agency. As such, there is no track record of industry compliance rates that is authoritative and comparable year to year. Estimates were presented that ranged from below 50% (considering all inspection tasks) to as high as 95%, representing no major non-conformance for high-risk tasks. (interviews, document review).  The Agency does report on compliance rates for high-risk tasks at rendering plants and feed mills. Historical rates reported in the DPR are as follows:

Compliance Rates by Year	2002/2003	2003/2004	2004/2005	2005/2006
Mills	92%	97%	95%	96%
Rendering Plants	90%	97%	93%	93%

Accurate and Timely Lab Results

Test results for residues that might be used to retain products arrive too late for the inspectors to take action. By the time the tests are done and Reports of Analysis completed the product has typically been consumed. (interviews)

Industry Awareness of Feeds Regulations

Commercial mills and renderers are aware of the Feeds Regulations.  Inspectors observed that retailers and producers who mix feeds on farm have a lower level of awareness of the objectives and the Feeds Regulations. (interviews)

What are the impediments to success, if any?

There are three significant impediments to success for the Feed Program. They are:

1. History of a poor relationship with ANAC: The relationship has been described by both CFIA and ANAC officials as confrontational at times. It is characterized by a lack of effective two-way dialogue which often leaves unresolved issues. For example, the Agency has established a sequencing table for medicated ingredients to advise what ingredients can follow after another without the need for flushing. ANAC disagrees with some of the table and feels it is too stringent but an effective dialogue to identify the issues and resolve them has not been possible. Feed Section staff believe ANAC is not always technical enough to engage in detailed discussions and ANAC feels that the Feed Section is too low organizationally for strategic discussions. However, ANAC has noted that their technical expertise is provided by member input, not Association staff, and they deal with other levels of the Agency for strategic discussions. (interviews, case study)
2. Outdated Feeds Regulations (1983): The Feeds Regulations show signs of ageing. They do not adequately address new products (such as genetically modified organisms (GMOs)), new technologies and trends (such as using feed to impact manure used for fertilizer) and their associated risks. It is based on product assessment and not process audits and it lacks sufficient tools such as administrative monetary penalties (AMPs) to effectively enforce. In short, it needs to be modernized.  (document review, interviews)
3. Insufficient Information Management and Technology Systems: The technology systems that support the Feed Program have failed to meet operational and reporting requirements. At present, there are no active projects to modernize the Multi-Commodity Activities Program (MCAP) for feeds. The Laboratory Sample Tracking System (LSTS) used for samples is insufficient for Feed program reporting and as a result, they re-enter the data and add one or two data fields to enable some reporting using the Feed Analysis Tracking System (FATS). Overall, there is a dearth of performance information for this program, data quality issues with the MCAP information, and no plans to improve the situation. In addition, the on-farm inspections have no comprehensive database of clients. (interviews)

Have there been any unexpected outputs or outcomes?

The only unintended outcome identified is a potential restructuring of the mill and rendering business models in Canada. Some renderers, in order to more cost effectively comply with BSE regulations regarding SRM, built separate facilities for SRM handling and non-SRM handling so that they do not need to segregate lines. ANAC reported one example of two feed mills that have traded clients to serve only ruminant or non-ruminant animals.

4.2 Medicated Feeds Activities under APF

Are planned activities being implemented and producing the expected outputs?

The following information summarizes the project environments and the projects’ planned and reported progress as contained in the Annual Reports.

Activity/ Outputs	2003/2004	2004/2005	2005/2006	2006/2007
Publish Regulations - Gazette Part I: Sept 1/04 - Gazette Part II: Dec/04	Amendments to Regulatory Proposal	Cost benefit analysis (CBA) for RIAS initiated Regulations reviewed and approved by DoJ	CBA ongoing	CBA ongoing Modifications to Regulatory Proposal completed Plan to move forward approved by Executive Policy Committee (Jan/07)
Publish Regulations - Gazette Part I: Sept 1/04 - Gazette Part II: Dec/04	Consultation sessions with industry	Consultation sessions across Canada (11) Consultations with variety of industry associations	Ongoing industry awareness activities Plan to determine stakeholder consultation strategy for 06/07	Ongoing industry awareness activities
National training and inspection program - April/05	--	Development of inspection forms, training modules, process Training sessions held	Ongoing development of training modules and tools Revisions to inspection forms Ongoing training sessions	Ongoing development of training modules and tools
National training and inspection program - April/05	--	Inspections initiated based on GMPs	Inspections ongoing based on GMPs	--
Functional Licensing Office - Jan/05	--	Staff identified for  HQ Licensing Office and 6 national Feed Program Specialists	Preliminary discussions on options for licensing Application licence, SOPs on hold pending regulation publication	Consultations with internal and external stakeholders to determine effective licensing options
Lab Accreditation System	--	Lab accreditation system initiated Lab methods development and processes	Lab accreditation system initiated No applications received	No applications received and none are anticipated
MCAP/CMS Systems Development	--	CMS	MCAP work on hold	Program working with CFIA consultant on required MCAP system changes Planning Web-based medicated feed mill licensing database

Actual expenditures are well below budgeted, with funding to expire in March 2008.

In $000s	2003/2004	2004/2005	2005/2006	2006/2007	2007/2008	Total
Budget	$4,070.0	$2,810.0	$3,300.0	$3,460.0	$3,360.0	$17,000.0
Actual	$366.0	$724.3	$1,121.4	$782.9*	N/A	$2,994.6

* Spending to Jan 31, 2007

Publish Regulations

Current target date is fall, 2007 for Gazette I. Initial target was 2004. This date has been changed several times. The most recent delays are due to concerns about the ability of industry to absorb additional regulations.

National Training and Inspection Program

Much of the work on this program has occurred. Training program is integrated across all other elements of the Feed Program. Training of inspectors is ongoing. National Inspector Certification Program not yet implemented.

Functional Licensing Office

Staff have been identified, yet continue to work on program design. The office is not operational.

Lab Accreditation System

To date, no labs have submitted applications for certification. Science Branch believes the testing, which requires significant equipment investment with no federal incentive does not provide sufficient return on investment at present so no certification requests are expected. (interviews)

MCAP/CMS Systems Development

Requirements being gathered but have not yet been communicated to OCIO. (interviews)

Are completed and planned activities meeting, or likely to meet articulated objectives?

The following table lists the outcomes for the Feed Program that are relevant for medicated feeds as listed in the Logic Model. The final column provides a quick assessment of whether they are being met. More detailed text follows for items in bold in the middle column.

Component	Outcomes	Are they being met?
Program Design and Management	Animal feed regulations are strengthened Efficient and effective design and management of medicated Feed Program	The animal feed regulations have not yet been strengthened The new elements are fairly well designed but failure to meet timelines implies poor management
Product/Establishment Assessment, Permitting, Licensing	Feed industry experiences a fair and effective licensing process	Not yet implemented
Verification and Compliance Activities	Industry compliance to Medicated Feeds Regulations	Not yet implemented. A clearly articulated needs analysis is required for the gaps and priorities linked to a science branch action plan.
Education and Awareness	Industry awareness of Medicated Feeds Regulations Industry adoption of GMPs	Mills are aware of proposed changes Industry has had mixed responses to the GMPs proposed by CFIA

Efficient and Effective Design and Management of Feed Program

The Medicated Feed Program has missed many significant deadlines. New regulations were due in 2004 and have still not been accomplished. Several external factors have played a significant part in this delay, including the finding of BSE, and most recently a decision to hold back publication until parliament recommences in the fall due to absorption capacity of industry. Development of the initial cost benefit analysis for the new regulations was poorly coordinated and communicated, resulting in an analysis that is not universally accepted within CFIA. (interviews)

Industry Adoption of GMPs

There are GMPs as tasks within the mill inspection forms, primarily related to the anticipated Medicated Feeds Regulations. At present, and until the new Medicated Feeds Regulations take place, these GMPs receive a mixed reception. Some mill owners appreciate the advice but most feel that they are redundant to the FeedAssure™ practices, unsupported by legislation and regulation and so are ignored or tolerated. This is exacerbated by the continued delays in implementation of the new regulations. (interviews)

What are the impediments to success, if any?

The three impediments described for the overall program also apply for the MFP. In addition, the following is the major impediment to progress in medicated feeds:

Delays in the Medicated Feeds Regulations: The time to implement the revised medicated regulations has resulted in the addition of non-mandatory inspection tasks based on GMPs. As an interim measure, these GMPs helped to familiarize mill operators with proposed regulations. They have, however, caused a loss of credibility as the delays in the implementation continue. The inspection tasks are recommendations only and as such are often ignored by the mill owners. Other program components such as staffing, licensing, standard operating procedures are dependent on the regulations being implemented. (interviews)

To what extent has CFIA complied with the terms and conditions set out in the MOU between CFIA and AAFC with respect to medicated feeds?

The table below outlines the sections of the MOU that has specific terms and conditions that CFIA should be complying with and the degree of compliance.

Section	Terms and Conditions	Are they being met?
3 Results and Outcomes	Activities, Outputs, Outcomes	Significant delays – see tables above for details.
6 Roles and Responsibilities	5 year work plan, annual work plans Quarterly reports	No evidence of a 5 year workplan. Annual plans available for 2005/06 and 2006/07 Quarterly reports available for: 2003/04: Q4 2004/05: Q1/2, Q3, Q4 2005/06: Q3 2006/07: Q1/2, Q3
9 Performance Reporting	RMAF and RBAF Reporting template	Logic Model developed. Reporting on activities and financial status Reporting template developed
10 Communication	Include reference to APF Bilingual	Not fully assessed – limited public communications available Not fully assessed – limited public communications available
11 Audit	Audits per RBAF	Partial audit completed in 2007
12 Evaluations	Evaluation per RMAF	Evaluation underway

In general, most conditions of the MOU have been partially met.

Have there been any unexpected outputs or outcomes?

The delayed implementation of the Medicated Feeds Regulations has given the Agency the opportunity to phase in the GMPs over time. There has been a mixed reaction to this. At present, because the GMPs are not mandatory, only some mills have been willing to adopt them.

4.3 Conclusions

Mill and rendering plans inspections appear to occur at the targeted frequency (twice, four times or once per year). Retail and on-farm inspections are occurring inconsistently across the areas and have no clearly established frequency targets.  Product registrations are processed but are not meeting the defined service standard.  Medicated feed activities and outputs are significantly behind schedule. Feed ban activities are on schedule for a July 12 rollout of the enhanced feed ban.

The completed and planned activities are likely to meet many of the articulated objectives. Significant effort is still required to bring the Medicated Feeds Regulations to implementation and to effectively engage stakeholders and manage their expectations. The low priority assigned to feeds activities by the labs also needs to be addressed to reduce the backlog of test results.

The impediments to success are the history of a poor relationship with ANAC, the outdated Feeds Regulations, insufficient information management and technology systems and delays in the Medicated Feeds Regulations.

There are some gaps in the compliance with the terms and conditions of the MOU.

The restructuring of the commercial feed mill and rendering industries to reduce the need for segregation has been an unexpected outcome.

5. Design and Delivery

This portion of the evaluation contains an assessment of the design and delivery of the Feed Program and its underlying subcomponents. It examines the management processes, coordination across other areas of CFIA and other stakeholders, the regulatory base, and the approach to inspection. Findings are derived primarily from the document reviews, site visits, and key staff and management interviews.

5.1 Core Feed Program

Are there appropriate resources, planning, priority setting, management and decision making structures in place to meet the objectives?

Resources have increased significantly since 2003 due to the new program funds related to Enhanced BSE activities.  Operations has expanded the number of inspectors, inspection managers and regional program officers across the country.  Product registration within the Feed Section has had temporary resource shortages as resources were redirected to other priority activities such as contamination issues.  (document review, interviews)

Shortages with lab resources may exist despite additional resources received related to BSE, as bottlenecks appear to exist in lab analysis.  These bottlenecks are generally restricted to monitoring plans as opposed to directed samples. Monitoring plans are designed to reflect the state of the marketplace and are considered as an initial phase. Test results that are outside of acceptable levels lead to directed sampling and testing, which are given priority.  (interviews)

Planning and priority setting are occurring at the tactical level within organizational units such as district and area Operations offices, Feed Section, and the Feed Ban Task Force. Detailed operational planning (e.g. inspections, sampling) in the past was communicated directly from Feed Section to inspectors; however, program growth and the movement to more traditional lines of communication (through Ops Coordination) has created some gaps in the planning process. (interviews)

One of the key planning documents, I-3-93, was not updated over the period of 2003 to May 2007. Planning guidelines in the 2003 version were outdated. Sampling targets have not been reviewed or updated since 2005/2006. (document review)

Strategic level planning and priority setting across the Feed Program and its various subcomponents is limited. This has resulted in planning and coordination issues such as the problems with the new inspection form roll-out, the delivery of just-in-time training, and the delays in the Inspector Certification program.  By comparison, the model used for the implementation of the enhanced BSE regulations demonstrated good planning and collaboration. (interviews, case study)

Senior management oversight of the Feed Program has increased in recent years, but was limited at times during the period under review due to competing priorities.  The finding of BSE in Canada in 2003 catapulted the Feed Program and the feed industry to the forefront of management and the Canadian public’s attention. Since 2003, management interest has remained high with respect to the core Feed Program and the enhanced BSE-related activities.  (interviews)

Decision-making structures within the Feed Section and within the Operations chain of command are clear.  Issues exist with decisions that affect more than one organizational unit. With the creation of the Working Group to liaise with ANAC, roles and responsibilities between the Programs, Operations, and the Working Group have become less clear resulting in mixed communications to ANAC and inspectors and confusion around program authority. (interviews, case study, document review)

A number of options for the organizational positioning of the Feed Section are under review within the Agency to provide a greater profile and alignment of the program. This remains unresolved at this time. (interviews)

Has the program coordinated activities effectively with other areas of CFIA, other federal departments, relevant provincial agencies and international agencies, including QUAD and EU?

Generally, coordination between areas of CFIA has been poor.  Historically, the Feed Program was a very self-contained, independent unit within CFIA.  The growth and increasing profile of the Feed Program has created challenges in engaging other units.  The historical direct line of communication between the Feed Section and the inspectors is no longer effective or appropriate and has not been effectively replaced with the traditional, formal lines of coordination.  Functional organizations such as CIO and Regulatory and Parliamentary Affairs have not been effectively engaged in the Feed Program, despite the significant regulatory and system changes required.  (interviews)

The Regional Program Officer positions being staffed within Operations are improving the coordination between Programs and Operations and across districts and regions. (interviews, case study)

The Enhanced Feed Ban Task Force has been successful in providing a coordinated focal point for the implementation of a significant regulatory change. (interviews)

The program has had a good working relationship with its two other key government departments – HC and AAFC.  There are some concerns with HC with respect to the timeliness of technical decisions and inspection support for compliance findings such as illegal drugs. More work is required to maintain the HC’s priority response to technical support required by CFIA. (interviews)

There is limited contact with provincial agencies; however, this is a good relationship when coordination is required. (interviews)

CFIA participates appropriately in a number of international forums related to feed such as the Association of American Feed Control Officials, CODEX, OECD and QUAD. (interviews)

Are processes/procedures for product assessment, establishment inspection and monitoring documented and clearly understood? Are they followed?

Product assessment procedures are well-documented and understood within the Feed Section.  For applicants, registration checklists, a guide (T-3-20 - Application for Feed Registration and Ingredient Approval) and application form are available on the CFIA Web site.  However, industry reports that the process is not well understood. This is supported by the finding that often applications for product registration do not meet the information requirements and are sent back to the submitter.  Industry feels that there is a lack of transparency in the process and that the reasons for refusal are not well-communicated.  When applications are returned, the Feeds Section communicates that the service level commitment is not considered until the completed form is returned (i.e the Agency is “off the clock” when awaiting proper documentation from the applicant and also when awaiting input from Health Canada).  (interviews, document review)

Establishment inspection is guided by the inspection form which combines the inspection tasks with the standards, creating a long and very detailed inspection form.  Inspection forms are available for commercial mills, rendering plants, retail outlets and on-farm where feed is mixed.  Processes, frequency and monitoring are clearly understood and followed for commercial mills and rendering plants.  There is a lack of clarity related to the frequency and follow-up procedures for retail establishments. Inspection processes, frequency and follow-up are also not clear for farms which mix feeds. (interviews, document review, site visits, file reviews)

Sampling procedures and frequencies appear to be well-understood.  Procedures are documented in the I-3-93 which was recently updated after a gap from 2003. Planning guidelines in the 2003 version were outdated. Sampling targets have not been reviewed or updated since 2005/2006. (document review)

Is the Feed Program supported by adequate and appropriate regulations, resources and management support?

The key regulations that the Feed Program is based on includes the Health of Animals Regulations, the Feeds Regulations, and the Food and Drug Act and Regulations.  The Health of Animals Regulations are current and have been amended over time as required, with the most recent set of amendments related to the Enhanced Feed Ban to come into force on July 12, 2007.  The Feeds Regulations (1983) have had some amendments over time as well and will also have amendments related to the Enhanced Feed Ban. There are a number of issues with the current Feeds Regulations as they are product-focused, they do not reflect developments in the feed industry, there is a lack of clarity and limited enforcement power. No issues were identified with the current Food and Drug Regulations (document review, interviews)

The Commercial Feed Mill Inspection Form (2002) currently in use lists the authority for Task 16 Evaluate the condition of the pest control product storage area as the Pest Control Products Regulations Section 43. In this regard, CFIA inspectors are assisting, informally, the Pest Control Regulatory Agency by identifying issues involving pesticide use. It is unclear if CFIA has a mandate under this set of regulations and as such, this task has been removed from the revised inspection form. (document review, site visit)

Management support, in the form of executive leadership, has been inconsistent at times. The National Program Manager, for example, reports to the Executive Director who has an extremely large portfolio of programs. Executives interviewed acknowledged that they had not had the opportunity to devote as much attention in the past to this program as required. (interviews)

Is the program using a HACCP-based approach and why or why not?

HACCP is a systems approach to processing often applied to food safety that aims to prevent the physical, chemical or biological contamination of food products. HACCP approaches tend to have three key components:

• A pre-requisite program that is developed by industry and approved
• A HACCP plan which includes conducting a hazard analysis, determining the Critical Control Points (CCPs), establishing critical limits, establishing a system to monitor control of the CCP, establishing the corrective action to be taken when monitoring indicates that a particular CCP is not under control, establishing procedures for verification, and documenting all procedures and records appropriate to these principles and their application. This plan typically is developed by industry and approved by a third party.
• Ongoing system reviews and audits are then usually conducted by a third party

Industry, led by ANAC, has developed and implemented a voluntary HACCP-based system called FeedAssure™. ANAC reports that since 1999, it has been implemented by approximately 180 feed mills and related facilities representing an estimated 70% of commercially manufactured livestock and poultry feeds. The mills are certified and audited by a third party, SGS. (document review, interviews)

The Feed Program historically was end-product focused relying on significant sampling of product. More recent changes have resulted in an inspection system based on reviews of a number of processes in mills that have been identified by CFIA as important to ensuring the production of safe feed.  However, the program does not include the pre-requisite program or the hazard analysis and controls by industry of a HACCP approach. The inspection process carried out by CFIA is identical at FeedAssure™   and non-FeedAssure™ certified mills. Of interest, in 1999/2000 there was consideration and some development on a HACCP-based inspection system for the Feed Program which was abandoned due to competing priorities. (document review, interviews, site visits)

The Agency has not promoted, internally, an Agency-wide approach for recognition of industry-led HACCP programs within regulated parties to encourage implementation within the Feed Program. Senior executives have consistently acknowledged the direction of integrating risk-based approaches into inspection programs but this approach has yet to be articulated in a manner that will allow for program modernization of the Feed Program. In this regard, the Feed Program with its recent collaboration with the sponsors of FeedAssure™ is helping to establish that approach. (document review, interviews)

The CFIA and ANAC have embarked on a recent initiative to compare and harmonize where possible the CFIA inspection program and FeedAssure™.   While there are still several key issues to be resolved, this initiative should result in greater alignment and potential streamlining of the inspection and auditing activities. ANAC reports that the discussions have helped them improve their HACCP-based system. (document review, interviews)

Are there mechanisms to provide assurance that processes and procedures are followed and are they effective?

CFIA has implemented a quality management system (QMS) and the Feed Program was one of the pilot areas.  A national QMS feed team was formed in 2005 and two cycles of reviews on the commercial feed mill inspections have been completed.  While implementation of recommendations has been limited to date, program staff, inspectors and inspection managers understand the potential value and express commitment to the program. (document review)

In addition, the Feed Ban Review of 2005 examined specific feed ban-related activities. Also, this Evaluation of the Feed Program is the first formal evaluation of the program. (document review)

5.2 Medicated Feeds under APF

Are there appropriate resources, planning, priority setting, management and decision making structures in place to meet the objectives?

While significant funds are available under the APF, capacity limitations and other priorities have limited progress. The project is currently being led from within the Feed Section. Accountability for the task has been assigned to a manager and the organization of the Feed Section allows for priority setting and decision making.  There was no evidence of project tracking and formal project management practices to ensure that schedules and budgets are tracked and deviations are addressed proactively. (interviews, document review)

Has the program coordinated activities effectively with other areas of CFIA, other federal departments, relevant provincial agencies and international agencies, including QUAD and EU?

The medicated feed regulation effort suffered from poor coordination with Regulatory Affairs, in part resulting in an ineffective first submission to Gazette I and dissatisfaction with the initial cost-benefit analysis.  They also have not effectively communicated system change requirements to the OCIO. While specific funding is identified for CMS and MCAP enhancements, no major initiatives are underway to enhance MCAP for the Feed Program. (interviews)

AAFC has reported good working relationship with the Feed Section with increasing concern in recent times over the lack of progress on the APF activities.  Many of the specific activities of the APF such as 5-year workplans, quarterly reports and a RMAF/RBAF suggesting that coordination of activities has not been optimal. (interviews, document review)

Are processes/procedures for product assessment, establishment inspection and monitoring documented and clearly understood? Are they followed?

This is not applicable at this time as the medicated feed program has not yet been implemented.

Is the Feed Program supported by adequate and appropriate regulations, resources and management support?

Regulations are under development. Significant and sufficient resources have been allocated over the five-year period under the APF. The Agency has been unable to attain sufficient progress to date to fully expend resources available. The medicated feeds initiative is now receiving increased management support as a priority of the Agency. In the labs there have not been any increases of resources to the core feed program to address a growing backlog of uncompleted tests. (interviews, document review)

Is the program using a HACCP-based approach and why or why not?

The new regulations take a process-based approach and as such are complementary to HACCP. However, the role of the new regulations and the existing, ANAC sponsored, FeedAssure™ have not been clearly established and opinions vary as to how the two will function. It has been suggested by industry that some CFIA inspection tasks are redundant in a FeedAssure™ mill.

Recent attempts to harmonize the FeedAssure™ program and the CFIA inspection program are making progress to bring the two mechanisms closer together. (interviews)

Are there mechanisms to provide assurance that processes and procedures are followed and are they effective?

This is not applicable at this time as the medicated feed program has not yet been implemented. Current QMS will capture processes and procedures at commercial feed mills.

5.3 Conclusions

Program resources are appropriate, having increased significantly since 2003.  Within science branch there are insufficient resources resulting in a testing backlog and an increase in the need for new method development. Planning and priority setting are occurring at the tactical level but strategic level planning and priority setting across the Feed Program is limited. Senior management oversight of the Feed Program has been increasing in recent years, but was limited at times during the period under review.  Decision-making structures within the Feed Section and within the Operations chain of command are clear; however, issues exist with decisions that affect more than one organizational unit.  The organizational positioning of the Feed Section remains unresolved.

Coordination on the Feed Program between units within CFIA is poor.  The historical direct line of communication between the Feed Section and the inspectors is no longer effective or appropriate and has not been fully replaced with more formal lines of coordination. The Enhanced Feed Ban Task Force has been successful in coordinating activities.  The program has good working relationships with other government departments and international agencies.

Product assessment procedures and sampling are well-documented and understood. Processes, frequency and monitoring are clearly understood and followed for commercial mills and rendering plants.  However, there is a lack of clarity related to the frequency and follow-up procedures for retail establishments and for farms which mix feeds.

The Health of Animals Regulations are current and have been amended over time as required. The Feeds Regulations are outdated, product-focused, do not reflect developments in the feed industry, lack clarity and have limited enforcement power. Executive leadership and oversight has been inconsistent at times, but is now showing improvements.

The feed inspection program is based on a process-based model; however, it is not HACCP-based and does not follow a risk-based approach.  The CFIA and ANAC have embarked on a recent initiative to compare and harmonize where possible the feed inspection program and FeedAssure™, the industry-led HACCP program. This initiative should result in greater alignment and make the program more complementary to HACCP.

CFIA has implemented a quality management system (QMS) and the national QMS feed team was formed in 2005 and activities are ongoing. Program staff, inspectors and inspection managers understand the potential value and express commitment to the program.

6.0 Continuous Improvement

This portion of the evaluation contains an assessment of the degree that continuous improvement has been implemented related to the Feed Program and its underlying subcomponents. It examines the use of best practices and risk assessment in the ongoing design and management of the program. Findings are derived primarily from the document reviews, Web-based document review, and key staff and management interviews.

6.1 Overall Program

Are ways to improve program delivery, quality and consistency being considered and implemented in support of continuous improvement?

CFIA has implemented QMS and the Feed Program was one of the pilot areas. A national QMS feed team was formed in 2005 and two cycles of reviews on the commercial feed mill inspections have been completed.  The process included a review of current processes, root cause analysis for quality loss and resulting recommendations.  While implementation of recommendations has been limited to date, program staff, inspectors and inspection managers understand the potential value and express commitment to the program. (document review, interviews)

Policy clarification form and processes developed to provide a structured method of communication for policy clarification between operational staff and programs, but this has not yet been implemented. (document review)

Feed Regional Program Officers are also playing a role in continuous improvement through provision of technical advice, dealing with issues that cross regions and seeking guidance from Program Network when required. (interviews)

Within the Feed Section in Ottawa, the feed product registration resources meet regularly as a team to consider ways to improve the process, such as triage.  Also, a “Blitz” process has been in place within the Feed Section for over 10 years. It is a formal method to elicit technical input on specific issues primarily related to product registration and assists in developing consistent responses to common problems. (document review)

Performance monitoring is completed on a very limited basis. Key performance indicators have been developed and are being reported; however, there remain issues with definition and there appears to be a decreasing level of commitment to the KPI in terms of monitoring, analysis and action planning. The annual Departmental Performance Report (DPR) includes compliance rates for feed ban-related tasks at commercial feed mills and the number of feed registration submissions received and approved.  The number of days to complete product registration is tracked and monitored against a standard.  While all facility inspections are entered into the MCAP system, reporting capabilities are limited and analysis of this data is not being done on a regular basis to inform program decisions. (document review)

Is the program considering and implementing best practices and lessons learned from other regulatory programs, other Federal/Provincial governments and international partners?

The proposed Medicated Feeds Regulations are building on the process-based approaches of other programs within CFIA. (interviews)

Program staff and management expressed a common opinion that there are no countries that are widely considered to have developed and implemented a gold standard feed regulation system. No evidence of leveraging of best practices from other Federal/Provincial governments or international organizations was found during the evaluation. (interviews)

The program design does consider international standards as required. For example, the Codex Good Animal Feeding Practices have been used to support the development of regulations and policies, e.g. medicated feeds.  In May 2007, Canada was recognized as a country with controlled BSE risk by OIE, demonstrating the meeting of OIE standards.  (interviews, document review)

Is risk assessment and mitigation considered in ongoing program design and management?

There is some evidence of risk assessment and mitigation in specific elements of the feed program.  For example, the frequency of mill inspections varies from two to four times per year depending on whether the mill is low or high risk, i.e. whether the mill deals with prohibited material.  (document review)

Some areas are using a risk assessment approach to determine priorities for on-farm inspections considering factors such as high-risk products, high volume production and multiple species, however, this is not consistent across all areas.  (interviews, site visits)

The product registration process includes safety assessments, a form of risk assessment at the product level. In addition, risk assessments are done jointly with HC for events related to contaminants and cross-contamination. (interviews)

However, formal risk assessments are not available for key program inspection activities. Specific questions have been raised on the risk assessment and risk mitigation value of some activities including:

• retail inspections, particularly where the retail facility only sells bagged feed and no bulk feeds;
• label reviews at retail establishments;
• sampling program as currently implemented;
• frequency of product renewals, currently at a frequency of every three years; and
• on-farm inspections.

6.2 Medicated Feeds Activities under APF

The medicated feeds activities remain in the development phase at this time and as a consequence, no continuous improvement activities are in place.

6.3 Conclusions

The Feed Program has adopted the QMS and performed two cycles of reviews on the commercial feed mill inspections. While implementation of recommendations has been limited to date, program staff, inspectors and inspection managers understand the potential value and express commitment to the program. Performance monitoring is completed on a very limited basis. Reporting capabilities are limited and analysis of this data is not being done on a regular basis to inform program decisions.

The proposed Medicated Feeds Regulations are building on the process-based approaches of other programs within CFIA through FSEP. The program design also considers international standards as required.

Risk assessment and mitigation are evident in specific elements of the program; however, formal risk assessment is limited in the overall design of the inspection programs. Formal risk assessments are not available for key program inspection activities. Specific questions have been raised on the risk assessment and risk mitigation value of some activities.

7.0 Conclusions and Recommendations

7.1 Conclusions

Relevance

The program is consistent with government and Agency priorities and fills an important role for ensuring food safety and the health of Canada’s livestock population. The vision of the program has not been adequately articulated and agreed upon.

Success

Mill and rendering plans inspections occur at the targeted frequency but retail and on-farm inspections are occurring inconsistently across the areas. They are supportive of the program objectives. Product registrations are not meeting the defined service standard.  Medicated feed activities and outputs of the APF MOU are significantly behind schedule. Significant effort is still required to bring the Medicated Feeds Regulations to implementation. Feed ban activities are on schedule for a July 12 rollout of the enhanced feed ban. The poor relationship with ANAC, outdated Feeds Regulations, insufficient IT systems and delays in the Medicated Feeds Regulations are limiting the likely success of the program.

Design and Delivery

The Health of Animals Regulations are current, but the Feeds Regulations are outdated. The feed inspection program is based on a process-based model; however, it is not fully HACCP-based.  Resources are appropriate and planning and priority setting is occurring at the tactical level. Strategic level planning and priority setting across the Feed Program is limited. Coordination on the Feed Program between units within CFIA is poor. The historical direct line of communication between the Feed Section and the inspectors is no longer effective or appropriate.  Product assessment procedures and sampling are well-documented and understood. Processes, frequency and monitoring are clearly understood and followed for commercial mills and rendering plants, but there is a lack of clarity related to the frequency and follow-up procedures for retail establishments and for farms which mix feeds.

Continuous Improvement

The CFIA has implemented the QMS initiative which has identified key areas for improvement; however, implementation of the identified recommendations needs a greater focus. Overall weaknesses exist in performance monitoring for the purposes of informing program decisions. There is some evidence of leveraging of other programs within CFIA, but limited use of best practices from other regulatory departments within Canada or international partners. Risk assessment and mitigation are evident in specific elements of the program; however, formal risk assessment is limited in the overall design of the inspection programs.

7.2 Recommendations

The most pressing actions for the Feed Program are to resolve strategic direction and planning issues that have resulted from a long period of relative inactivity in program management, coupled with explosive recent growth. The key recommendations are outlined below.

1. Develop and communicate a strategic policy direction for the Feed Program

   For the core Feed Program, an integrated plan should be developed that prioritizes the various initiatives and be consistent with the articulated policy direction.  Specific actions should include:

   • Integration of the approach to HACCP with the core Feed Program and the Medicated Feeds Regulations;
   • Executive engagement and support;
   • Communication of this policy direction and vision internally in particular down the reporting chain of command; and
   • Communication of policy direction and vision with key stakeholders including feed and rendering facilities, industry associations, HC, AAFC, farming community, and retailers.
2. Strengthen program leadership and leverage executive support for key strategic initiatives

   For the core Feed Program, strengthen program leadership to:

   • Resolve governance in terms of decision making authority, roles and responsibilities;
   • Develop defined roles and responsibilities, including membership and objectives of the industry liaison Working Group; and
   • Resolve the organization context for the Feed Program in terms of either a Division or a Section.

   For the core Feed Program, develop comprehensive performance monitoring and reporting to monitor the effectiveness of the program and inform decision making.  Place more emphasis on the implementation of the findings from the QMS process.

   For the APF, strengthen accountability and complete the activities outlined in the APF related to the development and implementation of Medicated Feeds Regulations as a high priority. Engage and leverage executive support toward rapid closure of outstanding tasks. Develop a strategy to complete the activities in light of the expiration of the APF funding in March 2008.

3. Promote an ongoing, constructive professional relationship with stakeholders

   For both the core Feed Program and the APF activities, the Agency must build on the recent positive steps taken to engage key stakeholders including the regulated parties and industry associations.  The Agency needs to:

   • Work more diligently to nurture a culture of mutual respect and trust; and
   • Promote the principles of effective regulatory consultation - openness, honesty and forthrightness, inclusiveness, transparency - as laid out by the Privy Council in the Guidelines for Effective Regulatory Consultations.
   • As a first step, task the industry liaison Working Group to develop a strategy to engage stakeholders constructively and demonstrate commitment to the process.
4. The CFIA should modernize the program design over time

   In light of the significant recent and ongoing changes in the program, modernize the core Feed Program design in a staged, risk-based approach.  This should include the following activities:

   • Review existing inspection, sampling, labelling and registration activities from a risk assessment basis;
   • Review, develop and promulgate inspection frequencies and criteria for retail and on-farm inspections;
   • Finalize key decisions regarding HACCP and its role in program delivery;
   • Engage the Office of the Chief Information Officer to enhance IT tools, especially for reporting; and
   • In the longer term, update the Feeds Regulations to reflect the current environment.

Annex A – Advisory Committee

The members of the Advisory Committee for the evaluation are listed below.

• Yvon Bertrand,
• Barbara Lee, Director, Food Laboratory Services
• Billy Hewett, Executive Director, Program Policy Directorate
• Freeman Libby, Director, Feed Ban Task Force
• Robert Charlebois, Director, Food Safety
• Robert Shepherd, Special Advisor, Audit, Evaluation and Risk Oversight

Annex B – Evaluation Matrix

Evaluation Issues: 1.0 Relevance - The continued relevance of the program, including whether or not it is consistent with CFIA and government priorities.

Evaluation Questions	Indicators	Data Sources
1.1 Are there clear and relevant vision and objectives for activities under this program?	Existence of seminal documents and statements Awareness and consistency of staff of vision and objectives	Document review Staff interviews
1.2 Does the program continue to be consistent with government-wide priorities and the CFIA mandate?	Alignment with CFIA mandate, mission statements and priorities Consistency with government-wide objectives and priorities Regulations and regulatory initiatives	Document review (DPR, RPP, Regulatory Initiatives, Throne Speeches, Ministerial statements) Interviews with CFIA executives
1.3 Is there a continued need for the program?	Continued relevance as risk prevention/mitigation Recognition of worth by stakeholders	Key interviews related to risk Interviews with stakeholders (e.g. feedlots) Document review (e.g. regulatory initiatives, contaminant events/safety assessments, MC/TB submissions)

Evaluation Issues: 2.0 Success - An assessment of the success of the program; to what extent it has achieved its objectives?

Evaluation Questions	Indicators	Data Sources
2.1 Are planned activities being implemented and producing the expected outputs?	Existence of activity plans, work plans Performance of activities against plan (quantity, quality and timeliness) Counts/timeliness of outputs: evaluations/approvals, inspections, compliance, etc. against planned numbers/standards	Document review Staff interviews
2.2 Are completed and planned activities meeting, or likely to meet articulated objectives?	Compliance rates - product, establishment   Degree of adoption of GMP by industry Export acceptance of Canadian feed/animals/animal products Regulatee awareness of regulations and regulated industry’s responsibilities Regulated bodies perception of fairness of regulations, processes Operations/Lab quality and timeliness	Stakeholder interviews and evidence Performance reports Document review of export approvals and lists of approved export countries and products
2.3 What are the impediments to success, if any?	Identification of specific impediments	Document review Staff interviews
2.4 To what extent has CFIA complied with the terms and conditions set out in the MOU between CFIA and AAFC with respect to medicated feeds?	Comparison of commitments and actual performance Comparison of CFIA performance against AAFC expectations as expressed in MOU	Performance reports Document review of MOU Staff interviews Interviews with AAFC
2.5 Have there been any unexpected outputs and outcomes?	Evidence of unintended outputs and outcomes	Document review Staff and stakeholder interviews

Evaluation Issues: 3.0 Design and Delivery

Evaluation Questions	Indicators	Data Sources
3.1 Are there appropriate resources, planning, priority setting, management and decision making structures in place to meet the objectives?	Existence of governance bodies and appropriate terms of reference and evidence of active engagement Evidence of reliable information available to decision-makers (e.g. program delivery information (quantity and quality), resource data (programs, operations, laboratory), submission scientific safety and efficacy evidence, regulated sector manufacturing information, compliance data indicators)	Interviews with program managers System review of decision support systems Document review (e.g. performance reports, qualitative assessments of program information, terms of reference for governance bodies and records of decision)
3.2 Has the program coordinated activities effectively with other areas of CFIA, other federal departments, industry, relevant provincial agencies, and international agencies, including QUAD and the EU?	Tone and effectiveness of correspondence Evidence of a positive collaboration with OGDs and other governments Evidence of a positive relationship with industry Extent of coordination activities	Document review (e.g. correspondence or case/issue files) Interviews with OGDs, regulated parties, industry associations and International Relations staff
3.3 Are processes/procedures for product assessment, establishment inspection and monitoring documented and clearly understood? Are they followed?	Clear documentation of processes and procedures including availability of reference tools and materials for inspectors,  evaluators and laboratory   staff Degree of staff understanding of processes and procedures Evidence that processes are followed (i.e. no missed steps)	Document review - procedure manuals and reference tools (e.g. interpretation bulletin) Process mapping and review File review of registration processes and inspections including rejection and litigation Staff interviews
3.4 Is the Feed Program supported by adequate and appropriate regulations, resources and management support?	Appropriate resource levels Completeness of regulations	Document review (Acts and Regulations) Budget review and comparison to operational plans Staff interviews
3.5 Is the program using a HACCP-based approach and why or why not?	Evidence of HACCP-based principles in program documentation	Document review - procedure manuals Staff interviews
3.6 Are there mechanisms to provide assurance that processes and procedures are followed and are they effective?	Existence of quality assurance and management processes Evidence that there are mechanisms/procedures to identify errors and corrective actions implemented	Document review (e.g. procedure manuals and file reviews) Staff Interviews

Evaluation Issues: 4.0 Continuous Improvement

Evaluation Questions	Indicators	Data Sources
4.1 Are there ways to improve program delivery, quality and consistency being considered and implemented in support of continuous improvement?	Evidence of corrective action identified in response to process deviations or inadequacies Evidence of consultation with stakeholders to solicit ways to improve effectiveness Evidence of evaluation of performance monitoring and activity to improve	Staff and stakeholder interviews Review of program documents
4.2 Is the program considering and implementing best practices and lessons learned from other regulatory programs, other Federal/Provincial governments and international partners?	Evidence of consultation with partners to solicit ways to improve effectiveness	Staff and stakeholder interviews Review of program documents
4.3 Is risk assessment and mitigation considered in ongoing program design and management?	Evidence of risk assessment Evidence of actions in response to risk assessments	Staff interviews Review of program documents

Annex C – Listing of Documents Reviewed

Category	Document	Date
Acts, Regulations, Schedules	Feeds Act
	Feeds Regulations, 1983
	Health of Animals Act
	Health of Animals Regulations
	Schedule I - Tables 1, 2, 3 and 4
	Schedule II - Tables 1 and 2
	Schedule V - index
	Schedules IV and V
Policy Documents	PD-001 - Addition of Pet Food as an Ingredient for Use in Livestock Feed	2003
	PD-002 - Feeding of Poultry Manure to Cattle	2003
	BSE - Import Policy for bovine animals and their products - AHPD-DSAE-IE-2005-9-1	2005
	BSE - Guidelines for returning various Canadian Animal products refused - AHPD-DSAE-IE-2003-9-2	2003
	Dir-95-03 Guidelines for the assessment of novel feeds: plant sources	2004
	Draft guidelines for the assessment of novel feeds - microbial products	1996
	Policy Regarding the Disposal of Animals, Their Products or By-products Through Commercial Inedible Rendering	2007
CFIA Departmental Performance Reports	Departmental Performance Reports 2005/06 2004/05 2003/04
	Data Verification Form and Notes	2006
Reports, Studies, Presentations	CFIA Feed Ban Review	2005
	Government Accountability Office: Mad Cow Disease FDAs Management of the Feed Ban has Improved, but Oversight Weaknesses Continue to Limit Program Effectiveness	2005
	The Feed Program	2004
	Feed Program Overview: Building an Integrated Program	Undated
	CFIA Enhanced SBSE Activities and CFIA Repositioning the Canadian Beef and Cattle Industry and Strengthening Animal Feed Restrictions 2005/06 Annual Performance Report	2006
	Benefit-Cost analysis of Regulatory Options for Medicated Feeds in Canada – Draft Final Report	2004
Agricultural Policy Framework	Memorandum of Understanding	2003
	APF Performance and Financial Reporting Template
	Status Report on CFIA’s Agricultural Policy Framework Initiatives	2007
	Summary of the APF Allotment	2007
	Medicated Feed Logic Model	2005
	Workplans: 2006/07 2005/06	2005 – 2006
	Progress Reports 2006/07: Q1/2, Q3 2005/06: Q3 2004/05: Q1/2, Q3, Q4 2003/04: Q4	2004 – 2007
Facility Inspection Forms	Feed Mill Inspection Form	2002
	Feed Mill Inspection Form - Partial Inspection - BSE	2006
	On Farm Inspection Form	2002
	Rendering Plant Inspection Form	2004
	Rendering Plant Inspection Form - Partial Inspection - BSE	2006
	Feed Retail Inspection Form	2006
	New Feed Mill Form	2007
	Feed Mill Inspection Form	2002
	Med Feed Alcohol Ethanol Inspection Form	2006
CFIA – ANAC FeedAssure Review	Meeting Minutes	2007
	FeedAssure Prerequisite Program Gap Analysis and ANAC Response	2007
	Science-based Questions – CFIA – ANAC HACCP Recognition	2007
	On-site Verification of Generic Feed HACCP System	2007
ANAC Publications	Situation Report on the Implementation of the Enhanced Feed Ban	2006
	Medicated Feeds in Canada – A Way Forward	2007
Follow-Up Inspection Guides	Feed Inspection Non-Compliance Label Guide	2006
	Feed Inspection Non-compliance Establishment Guide	2006
	Guide to Conducting Follow-Up Inspections of Non-Compliant Samples (Draft)	2006
	On Farm Residue & Compliant Inspection (Draft)	2006
	Dioxin Traceback Guidelines	2005
Inspector Tools - Miscellaneous	Facility Inspection - Questions and Answers
	Veterinarian Prescription Checklist	2006
	Label Verification - Excel Spreadsheet	2006
	Medication Sequencing Guide	2005
	Monensin Sodium Calculations
	Registrant List
	MCAP User Guide	2003
	National Feed Inspection Program document - I-3-93 Draft	2007
	National Feed Inspection Program document - I-3-93	2003
	National Feed Inspection Program Changes	2003
	National Feed Inspection Programs - 2003-2004 (English)	2003
	Policy Clarification Request (Informed Filler)	2007
Inspector Certification Training Modules	Feed Labelling - Module	2006
	Feed Program - Module	2006
	Import - Module	2005
	Product Sampling - Module	2005
	Production Records - Module	2006
	Inspection Comments Module	2006
	Feed Inspector Training Module - TOXICOLOGY- draft	2007
On-the-Job Training Outlines	On The Job Training (OJT) - Retail	2005
	Final - OJT Appendix - Retail	2006
	Final - OJT Appendix - On Farm Feed Mill	2006
	Final - OJT Appendix - Feed Mill	2006
	December OJT - On-Farm	2005
	December - OJT - Feed Mill	2005
Training - Miscellaneous	Commercial Feed Mill- On Farm - Retail Inspection Module	2006
	Feed Export Certification Module	2006
	Self Directed Learning - Answers	2005
	Training Program Meeting Report - English	2005
	Detailed Feed Training Program	2005
	Feed Training Program Overview	2005
	Training Letter	2005
	Meeting Next Steps	2005
	National Training Initiative Programs Submission Forms: Feed 1 - Intro to Feed Training Feed 2 - Feed Facility and Intro to Audit Training Feed 3 - Feed Labelling Training Feed 5 - Rendering Plant Inspection Training Feed 6 - Operational Management Training-	2006
	National Training Initiative Quarterly Reports: Intro to Feed Q2 Feed Labelling Q2 Rendering Plant Inspection Q2	2006
Quality Management System (QMS)	Rendering Plant Permit Control Procedures	2003
	NVT Feed Recommendations	2007
	QLI Fishbone	2007
	QMS National Verification Task List	2007
	Process Flow	2007
	Verification Criteria Feed Sampling	2007
	Criteria Feed Export Certificate	Undated
	Verification Criteria Feed Labelling	2007
	Feed Label Process Flow	2007
	Feed Sample Process	2007
	NVT Activity Update Presentation	2006
	QMS Feed presentation Ottawa	2005
	Feed Mill National Verification 2005/06 Summary Report	2006
Form Letters	Feed Non-compliant Sample Letter-Program 60A (Draft)	Undated
	Feed Compliant Sample Letter-Program 60A (Draft)	Undated
	Feed Non-compliant Sample Letter-Program 15B (Draft)	Undated
	Feed Compliant Sample Letter Program 15B (Draft)	Undated
	Feed Non-compliant Sample Letter-Program 15D (Draft)	Undated
	Feed Compliant Sample Letter-Program 15D (Draft)	Undated
	Feed Non-complaint Sample Letter-Program 61 (Draft)	Undated
	Feed Compliant Sample Letter-Program 61 (Draft)	Undated
	Feed Non-compliant Sample Letter-Program 15A (Draft)	Undated
	Feed Compliant Sample Letter-Program 15A (Draft)	Undated
	Feed Non-compliant Sample Letter-Program 15B (Draft)	Undated
	Draft Free Sale Certificate	Undated
Product Registration Checklists	List of Registration Checklists	Undated
	16 Registration Checklists	2003
Product Evaluation Standard Operating Procedures	Feed Application Procedures (Revised)	2003
	Product Registration System - User Guide	2000
	Orientation Manual	2003
	Assessment table products short form (Blank Form)	Undated
	Biotech Nutrition Evaluation Form (Blank Form)	Undated
	Safety/Efficacy Assessment - Biotech Feeds (Blank Form) Long Form	Undated
	Feed Toxicology SRP for Safety Assessments	Undated
	Safety Assessment Table (Blank Form) - Tox	Undated
	Safety Assessment of Viable Microbial Strains (Blank Form) Short Form	Undated
Workplans	Feed Workplan	2005 - 2006
Trade Memoranda	Index of Trade Memoranda	May 2000
	34 Memos	Various 1990 to 2001
Compendium of Medicating Ingredient Brochures	56 Brochures	Various 1998 to 2007
Blitz Documents	Blitz Rules	1995
	Index of Blitz	1995 – 2006
	Index of Legal Opinions	1983 - 2006
Hazard Analysis and Critical Control Point	HACCP Plan Audit Checklist (Blank Form)	1998
	Appendix 1 - Feed Establishment Document Tracking Form	1998
	Appendix 2 - Feed Establishment - Pre-requisite Program Review Worksheet	1998
	Appendix 2 - Pre-requisite Program Audit Checklist (Blank Form)	1998
	Pre-requisite Program Review -  Written Program Incompletes	1998
	Corrective Action Request	1998
	Appendix 4 - Feed Establishment - HACCP Plan Review Worksheet	1998
	Appendix 3 - Guidelines for the Use of the Health Risk Assessment Model	1998
	HACCP Generic Model - Dry Livestock Feed (Larger Scale Establishments)	1998
	Process Flow Diagram - Form #3A Product Name:  Dry Livestock Feed	1998
	Process Flow Diagram - Product Name: Dry Feed for Food Producing Animals	1998
	HACCP Generic Model - Dry Feed for Food-Producing Animals	1998

Key Web Sites:

• CFIA: www.inspection.gc.ca
• ANAC: www.anac-anac.ca/
• US Government Accountability Office: www.gao.gov/
• US Food and Drug Administration: www.fda.gov/

Annex D – Interview Guides

Interview guides were prepared for:

• program managers and implementation staff;
• science branch;
• regulatory and parliamentary affairs;
• information management;
• other government departments; and
• industry representatives.

These interview guides are contained in the pages which follow.

Feed Evaluation Interview Guide
Internal CFIA

Introduction:

TDV Global, a management-consulting firm, has been engaged by Audit, Evaluation and Risk Oversight (AERO) Branch of CFIA to conduct an evaluation of the Feed Program including the Medicated Feeds component of the Agricultural Policy Framework and the enhanced BSE programming. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

We would like to discuss the program with you, based on the areas outlined below that you are involved in. We anticipate requiring approximately 60 to 90 minutes.

Opening

1. Please describe your involvement with the Feed Program.

Relevance

2. What is your understanding of the objective and the vision of the Feed Program?

3. What is the value of the program to the agricultural industry and Canadians in general?

Success

4. Do the program’s activities result in safer or more reliable feed? How do you know?

5. How successful has the implementation of the medicated feeds portion of the program been as set out in the MOU? What have been the key accomplishments and issues?

6. How successful has the implementation of the enhanced BSE portion of the program been? What have been the key accomplishments and issues?

7. Are you aware of any unintended outcomes or outputs from the program?

Program Design and Delivery

8. Describe the major decisions faced in the operation of the program. How are those decisions made; are they effective?

9. How do you work with the following groups - what activities are coordinated and how? Are there any problems?

1. Other government departments
2. International partners
3. Other branches/areas within CFIA

10. How do you know the procedures for your activities for the Feed Program? Do you have manuals or other documentation that you use? When did you last receive formal training on procedures?

11. Are documented procedures appropriate? Do you ever have to deviate from them?

12. Are the regulations in place sufficient for you to carry out your mandate?

13. What activities, if any, are not performed due to lack of financial or executive support?

14. To what degree has the program embraced HACCP-like procedures?

Continuous Improvement

15. Are ways to improve program delivery being considered and implemented?

16. Have you considered and implemented best practices observed internationally? If so, in what areas?

17. How do you assess risk in considering your activities? Is there formal risk management?

Feed Evaluation Interview Guide
Science Branch

Introduction:

TDV Global, a management-consulting firm, has been engaged by Audit, Evaluation and Risk Oversight (AERO) Branch of CFIA to conduct an evaluation of the Feed Program including the Medicated Feeds component of the Agricultural Policy Framework and the enhanced BSE programming. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

We would like to discuss the program with you, based on the areas outlined below that you are involved in. We anticipate requiring approximately 60 minutes.

Opening

1. Please describe your involvement with the Feed Program.

Relevance

2. What is your understanding of the objective and the vision of the Feed Program?

Success

3. Do the program’s activities result in safer or more reliable feed? How do you know?

4. How successful has the implementation of the medicated feeds portion of the program been? What have been the key accomplishments and issues?

5. How successful has the implementation of the enhanced BSE portion of the program been? What have been the key accomplishments and issues?

6. Are you aware of any unintended outcomes or outputs from the program?

Program Design and Delivery

7. How do you know the procedures for your activities for the Feed Program? Do you have manuals or other documentation that you use?

8. Are documented procedures appropriate? Do you ever have to deviate from them?

9. What activities, if any, are not performed due to lack of financial or executive support?

Continuous Improvement

10. Are ways to improve program delivery being considered and implemented?

11. Have you considered and implemented best practices observed internationally? If so, in what areas?

12. How do you assess risk in considering your activities? Is there formal risk management?

Feed Evaluation Interview Guide
RPA

Introduction:

TDV Global, a management-consulting firm, has been engaged by Audit, Evaluation and Risk Oversight (AERO) Branch of CFIA to conduct an evaluation of the Feed Program including the Medicated Feeds component of the Agricultural Policy Framework and the enhanced BSE programming. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

We would like to discuss the program with you, based on the areas outlined below that you are involved in. We anticipate requiring approximately 60 to 90 minutes.

Opening

1. Please describe your involvement with the Feed Program.

Relevance

2. What is your understanding of the objective and the vision of the Feed Program?

3. What is the value of the program to the agricultural industry and Canadians in general? How well does it reflect G of C priorities?

Success

4. How successful has the implementation of the medicated feeds portion of the program been as set out in the MOU? What have been the key accomplishments and issues?

5. How successful has the implementation of the enhanced BSE portion of the program been? What have been the key accomplishments and issues?

Program Design and Delivery

6. How does the Feed Program work with other government departments and central agencies- what activities are coordinated and how? Are there any problems?

Feed Evaluation Interview Guide
CIO

Introduction:

TDV Global, a management-consulting firm, has been engaged by Audit, Evaluation and Risk Oversight (AERO) Branch of CFIA to conduct an evaluation of the Feed Program including the Medicated Feeds component of the Agricultural Policy Framework and the enhanced BSE programming. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

We would like to discuss the program with you, based on the areas outlined below that you are involved in. We anticipate requiring approximately 60 minutes.

Opening

1. Please describe your involvement with the Feed Program.

Success

2. How successful has the implementation of the medicated feeds portion of the program been as set out in the MOU? What have been the key accomplishments and issues?

3. How successful has the implementation of the enhanced BSE portion of the program been? What have been the key accomplishments and issues?

4. Are you aware of any unintended outcomes or outputs from the program?

Program Design and Delivery

5. How do you know the procedures for your activities for the Feed Program? Do you have manuals or other documentation that you use? When did you last receive formal training on procedures?

6. Are documented procedures appropriate? Do you ever have to deviate from them?

Continuous Improvement

7. Are ways to better support program delivery being considered and implemented?

8. Have you considered and implemented best practices observed internationally? If so, in what areas?

Feed Evaluation Interview Guide
External - OGDs

TDV Global, a management-consulting firm, has been engaged by Audit, Evaluation and Risk Oversight (AERO) Branch of the Canadian Food Inspection Agency (CFIA) to conduct an evaluation of the Feed Program including the Medicated Feeds component of the Agricultural Policy Framework. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

We would like to discuss the program with you, based on the areas outlined below. We anticipate requiring approximately 60 minutes.

Opening

1. Please describe your involvement with the Feed Program.

Relevance

2. What is your understanding of the objective and the vision of the Feed Program?

3. How does the program benefit the agricultural industry and Canadians?

Success

4. Do the program’s activities result in safer or more reliable feed?

5. How successful has the implementation of the medicated feeds portion of the program been as set out in the MOU? What have been the key accomplishments and issues?

6. How successful has the implementation of the enhanced BSE portion of the program been? What have been the key accomplishments and issues?

7. Has the program resulted in any unintended outcomes or outputs that you are aware of?

Design and Delivery

8. What are the key coordinating mechanisms between your department and CFIA? How effective are these?

Continuous Improvement

9. Is CFIA open to suggestions on how to improve the program? Are ways to improve program delivery being considered and implemented?

Feed Evaluation Interview Guide
Industry

Introduction:

TDV Global, a management-consulting firm, has been engaged by the Canadian Food Inspection Agency (CFIA) to conduct an evaluation of the Feed Program. The evaluation will assess the relevance, success and design and delivery of the program. Recommendations will focus on ways to improve the overall program effectiveness.

Opening

1. Please describe your involvement with the CFIA Feed Program.

Relevance

2. What is your understanding of the objective and the vision of the Feed Program? Is it clear what the role of CFIA is and what it is trying to achieve?

3. How does the Feed Program benefit you and your organization?

Success

4. Do the program’s activities result in safer or more reliable feed?

5. Is CFIA open and fair in its procedures? Is it timely in its decisions and schedules?

6. Has the program resulted in any unintended outcomes to you?

Program Design and Delivery

7. Are the regulations that are enforced by CFIA relevant and up to date?

8. How does CFIA communicate the regulatory requirements to you? Is the information clear, timely? Do you have enough information to understand what is required of you from a regulatory perspective? How effective has the communication on regulatory changes been in the past?

9. Do you feel that CFIA has adequate and appropriate resources to effectively manage the Feed Program?

Continuous Improvement

10. Is CFIA open to suggestions on how to improve the program? Are ways to improve program delivery being considered and implemented?

Annex E – Case Study

Under the Feed Program, a number of different types of facilities are inspected. Currently, there are four separate inspection forms in use:

• Commercial feed mill form: Full and partial inspection (2002)
• Rendering Plan Form: Full and partial inspection
• On-farm Form
• Retail Inspection Form

The current version of the forms was implemented in about 2002. The forms have been under revision for several years with multiple attempts to implement revised versions. This case study reviews the processes, impacts and challenges of these attempts.

History

All of these forms follow a similar structure and content. In general, the forms include a significant amount of detail, essentially containing the material that would typically be included in an inspection manual. Each form has an initial section for general information about the facility, a number of task inspection sections and a summary section. Each task includes the following elements:

• Task: number and name
• Rating type: Compliance, Good Manufacturing Practice or Not Applicable
• Standard: description of what elements must be met to achieve a satisfactory rating
• Comments: unstructured format

Type of Facility	# of Pages	# of Tasks
Commercial Feed Mill - Full - Partial	52 18	86 17
Rendering Plant - Full - Partial	29 20	34 20
On-Farm Inspection	46	59
Retail Inspection	24	26

The prescriptive nature of the form was intended to assist the significant number of new inspectors that were hired over a short period of time. It has been reported that the intent was to separate out the manual from the form itself at some point in the future.

The 2002 version incorporated many of the procedures that were to be implemented under the Medicated Feeds Regulations as Food Manufacturing Practices (GMPs). Instruction to inspectors and common practice was to consider these GMPs as optional until such time as the regulations came into force.

Proposed Revisions

A process was embarked on early in 2005 to identify and implement modifications to the form. One of the key changes was to identify minor deviations, major deviations and critical deviations. Feed Section led the initiative at the time and garnered the input of experienced feed inspectors. There are conflicting messages regarding the level of influence the experienced inspectors had on the form design. The revised forms were used for training of the new inspectors that were hired in 2005/2006.

2006 Pilot

A revised form was developed and a pilot of the revised form was scheduled for 2006. Industry responded negatively and contacted senior CFIA management. The pilot was partially completed.

2007 Roll-out

In late 2006 and early 2007, Programs and Operations worked together to reduce the size of the form and include the previously recommended changes. Tasks were reduced from 86 to about 42. Agreement was secured between Programs and Operations to implement the form on a time-limited basis (from January 2007 to April 2008) when additional changes would be required. On March 28, 2007 a new form was distributed by programs to inspectors with instructions to begin using the form on April 1, 2007. The form had some differences from the one agreed to between Programs and Operations in January. At the same time, a request was sent to Operations to review the new form with ANAC.

Additional training to inspectors was not provided at this time. Inspectors in the areas reviewed the form and in some cases forwarded the form to the mills where they would be conducting inspections in early April.

Feed mill managers contacted ANAC regarding the form and ANAC requested that the form be withdrawn. Over the next two weeks, inspectors received notification through Operations not to use the form; however, some inspections were completed on the new form. In other cases, CFIA inspectors were informed by mill management that CFIA had withdrawn the form and it was not to be used.

Impacts

Inspectors’ Perspective

Inspector’s feel their credibility with industry is being impacted as industry seems to know more than they do about CFIA policies. There are reports of inspectors losing confidence in CFIA headquarters staff. Find the uncertainty around the forms demoralizing.

Feed Section Perspective

Feed section is also frustrated with the inability to implement changes to the inspection form that have been under development for an extended period. Degree of industry involvement in the process is also of concern.

Industry Perspective

Industry saw the process as a failure of communication. Consultations on the new form in the early stages were not felt to be a two-way dialogue. Expressed concerns were not felt to be addressed. Some feed mill owners contacted ANAC for clarification and information as they had not received advance notification of implementation of the form. ANAC has recently decided to focus efforts elsewhere as they do not see the value in future discussions on the form.

Lessons Learned

Issues Identified	Issues, Consequences
Communication with inspectors: there is more than one line of communication with inspectors from headquarters regarding program design and management	Conflicting communications to inspectors Lack of coordination in communications to inspectors Loss of credibility of headquarters from the perspective of the inspectors Confusion amongst inspectors as to the direction of the program, future of the form, reasons for withdrawal
Timeliness of communication with inspectors: ANAC was able to communicate faster with mill management regarding the withdrawal of the form than the CFIA could to the inspectors	Loss of credibility of inspectors with regulated parties
Relationship with the regulated party: Lack of proactive communication with regulated parties regarding upcoming changes	Negative response from industry to senior management of CFIA
Authority for program design: Form is implemented and withdrawn in a very short time period	Program accountability and decision making processes are not clear Frustration on the part of all staff involved

Indicators for the Evaluation

The following table lists the evidence gathered from this case directed to the evaluation issues. Only the questions relevant to Case Studies are listed from the Evaluation Framework.

Evaluation Issue	Evidence/Indicators
Success 2.1 Are the planned activities being implemented and producing the expected outputs?	No, the revised forms for inspection have not been rolled-out in a timely manner.
Success 2.3 What are the impediments to success, if any?	The relationship with the regulated industry (ANAC and industry in general) has been an impediment to success. ANAC has been involved in the new form roll-out on at least two occasions in a negative, reactive manner. Proactive communication with industry was not successful in the initial roll-out and was not evident in the second roll-out.
Design and Delivery 3.1 Are there appropriate resources, planning, priority setting, management and decision making structures in place to meet the objectives?	No, decision making structures are not clear, resulting in confusion over who has the authority to implement program changes.
Design and Delivery 3.2 Has the program coordinated activities effectively with other areas of CFIA, other federal departments, relevant provincial agencies, and international agencies, including QUAD and the EU?	No, the form roll-out was not well communicated between headquarters and the inspectors, or between programs and Operations in headquarters.