Summative Evaluation of the Canadian Shellfish Sanitation Program (CSSP)

Final Report

The Canadian Shellfish Sanitation Program (CSSP) is a federal food safety program jointly administered by the Canadian Food Inspection Agency (CFIA), Fisheries and Oceans Canada (DFO) and Environment Canada (EC).  The CSSP is an important program which protects Canadians from potential harm associated with the harvesting and consumption of bivalve shellfish.

As part of its on-going efforts to ensure that the Canadian Shellfish Sanitation Program continues to provide improved protection from food safety risks associated with shellfish, the Government of Canada commissioned an independent evaluation of the program in 2006. The report of this evaluation was issued on July 3, 2007.

The findings have confirmed good operational delivery of the program, and that it is effective in protecting consumers. In response to some of the recommendations in the report, CSSP partners have developed a management response and are currently implementing an action plan to strengthen the CSSP.  

The Government of Canada, through the CSSP, is working hard with industry, provincial governments and other stakeholders to enhance the effectiveness of the CSSP. The Stratos Report and the CSSP Management Response and Action Plan can be found below.

---

Table of Contents

• EXECUTIVE SUMMARY
  • 1. INTRODUCTION TO THE EVALUATION
  • 1.1 BACKGROUND TO EVALUATION
  • 1.2 BACKGROUND ON THE CANADIAN SHELLFISH SANITATION PROGRAM
  • 1.3 PROGRAM OBJECTIVES
  • 1.4 ROLES AND RESPONSIBILITIES
  • 1.5 EVALUATION OBJECTIVES AND SCOPE
  • 1.6 EVALUATION APPROACH AND METHODOLOGY
• 2. FINDINGS BY KEY EVALUATION QUESTIONS
  • 2.1 OUTCOMES
  • 2.2 PRESSURES, RISKS AND LIABILITIES
  • 2.3 GOVERNANCE
  • 2.4 PROGRAM DELIVERY
  • 2.5 PROGRAM COSTS
  • 2.6 PROGRAM ALTERNATIVES
• 3. CONCLUSIONS
  • 3.1 PROGRAM OUTCOMES
  • 3.2 RISKS AND LIABILITIES
  • 3.3 GOVERNANCE, ADMINISTRATION AND COORDINATION
  • 3.4 PROGRAM DELIVERY
• 4. RECOMMENDATIONS
  • 4.1 GOVERNANCE
  • 4.2 PROGRAM RISKS
  • 4.3 PROGRAM SCOPE AND POLICY
  • 4.4 PROGRAM STRUCTURE
  • 4.5 RESEARCH SUPPORT
  • 4.6 PERFORMANCE TRACKING AND REPORTING
• Appendices - Working Papers:
  • Appendix A - Formative Evaluation of the Canadian Shellfish Sanitation Program
  • Appendix B - Program Profile for the Canadian Shellfish Sanitation Program
  • Appendix C - Cost Analysis for the Canadian Shellfish Sanitation Program Evaluation
  • Appendix D - Alternative Approaches for the Canadian Shellfish Sanitation Program (Not for publication)
  • E. List of interviewees (Interviewee names not for publication)
• CSSP Evaluation Report: Management Response and Action Plan

---

Executive Summary

Evaluation Overview

This evaluation was undertaken in response to a request from senior management at the Canadian Food Inspection Agency (CFIA), Environment Canada (EC), and Fisheries and Oceans Canada (DFO), to conduct an independent and objective summative evaluation of the effectiveness and efficiency of the current Canadian Shellfish Sanitation Program (CSSP).

As a summative evaluation, this assessment was designed to:

• Determine progress in delivering identified outputs and achieving the anticipated results related to performance and delivery of the CSSP between 2000/01 and 2005/06;
• Provide information to assist the CSSP partners in addressing alternative governance and delivery options and developing recommendations for improvement in program delivery and management; and
• Assess the degree to which appropriate programs, processes, and procedures have been put in place to ensure effective and efficient expenditure of resources.

The evaluation process involved gathering multiple perspectives across multiple lines of enquiry, including:

• Document review and interviews with responsible program personnel and managers from the three federal partners;
• Interviews with 57 national and regional program staff;
• Interviews with 25 key informants having in-depth knowledge of the program;
• A workshop dedicated to exploring implementation and delivery alternatives; and
• A cost analysis of the program's key costs and expenditures.

This evaluation is "evidence-based." That is, its conclusions and recommendations are based on objective, quantitative and documented evidence to the fullest extent possible. Where possible, findings and conclusions were drawn from documented evidence, however, information obtained from interviews was incorporated into the evaluation to provide context for formulating findings and conclusions when documented evidence was not available.

Departmental personnel validated all findings from this evaluation and a series of presentations were held to debrief program managers in all three partner organizations.

CSSP Overview

The CSSP dates back to 1924-25, when regulations were developed in response to an epidemic of typhoid fever following the consumption of contaminated oysters in the United States. Currently, the three federal partners manage the CSSP jointly, and the day-to-day operations of the program are governed by:

• A Memorandum of Understanding that describes the roles and responsibilities of the three federal partners;
• Bilateral Agreements with major trading nations (e.g. United States of America, European Union); and
• A comprehensive Manual of Operations, which serves as a reference document and provides the operational policies and procedures for government staff involved with duties related to the classification and patrolling of shellfish harvesting areas and the harvesting, processing and distribution of shellfish.

The partner departments/agency work collectively to achieve two major program objectives:

• Continue to protect the public from the consumption of contaminated shellfish (bivalve molluscs more specifically), by controlling the recreational and commercial harvesting of shellfish within Canada, as well as of shellfish imported to Canada; and
• Fulfill Canada's international obligations to meet the terms of bilateral agreements (e.g. United States of America, EU. etc)

Conclusions and Recommendations

The evaluation found that the context in which the CSSP operates continues to evolve, and the risk drivers associated with the program are increasing. Funds available for program delivery have not been increasing and, as a result, there is a need to improve the analysis and rationalize resource allocations. This effort has recently been challenged due to the structure for delivery being split between the three federal partners. With an increased awareness of the food safety mandate of the program, there is a need to identify and implement changes to the program. The current approach is placing constraints on effecting that change, and the following recommendations are provided with the purpose of enabling the government to increase the responsiveness of the CSSP to these emerging challenges.

The following broad conclusions arise from the detailed findings presented in this evaluation report. With respect to the specific objectives established for this evaluation, the following conclusions and associated recommendations¹ are put forward:

Governance

The evaluation has identified a number of areas where improvements can be made in program governance. These were primarily in the cross cutting areas of program management and coordination, as opposed to operational delivery. Some of the impacts are reflected in operational issues, such as inconsistent delivery of program activities across the regions, or differing levels of effort in and across the regions and programs. However these inconsistencies can be reduced through stronger and more responsive policy direction. This will require strong and pro-active leadership and program coordination than currently exists.

There are also a number of issues or activities that require a more focussed approach, such as addressing enquiries from international trading partners, dealing with international standards and guidelines (e.g. CODEX), coordination of efforts with the provinces, and establishing appropriate accountability mechanisms to ensure that the program is being implemented as intended. To address these needs, we make the following recommendation:

Program Risks

In order to ensure that the CSSP remains relevant and effective, there is a need to undertake a systematic assessment of the risks associated with shellfish food products. This risk assessment should be completed on a priority basis in order to inform the development of the program policy and scope statement. The risk drivers identified in this evaluation could be a starting point for such an analysis, but should also take into consideration the experiences of other jurisdictions involved with managing shellfish risks, and the views of other experts and scientific specialists.

Such a risk assessment would provide ongoing benefits for setting program priorities.

Program Scope and Policy

The evaluation found that there are an increasing number of risk drivers that are putting pressure on the program. These pressures are difficult to address under the current design and policy framework within which the program is delivered. Operational staff are required to make day to day decisions within the mandates of their respective organizations, and within the resources and operational guidance in place for the program. With the lack of clarity around program priorities, some inconsistencies in program delivery have crept into program delivery across the country. Also, with increasing internal resource constraints and external pressures, it is becoming more difficult to meet client group expectations for increased coverage or service from the program.

Recent efforts to redesign the program have also been stalled due to the difficulty in defining exactly what the program is intended to cover or address. In order to move forward, it is critical that the policy framework for the program be clarified. This will require a thorough analysis of the government's legal and other obligations with respect to the shellfish program, as well as development of a clear vision or directional guidance document for the program.

Program Structure

The CSSP currently is conducted under the guidance of the MOU and the Operations Manual. These documents provide substantive guidance on the operational requirements of the program. The immediate challenges facing the program relate more to the cross-cutting management and administrative requirements to ensure the program meets objectives that go beyond the mandates and priorities of the three program partners.

The challenges of managing programs that cut across more then two or three departments and agencies is not unique to the CSSP. Numerous government programs rely on a coordinated effort from a number of partnering departments and agencies. Treasury Board has recognized these challenges, and has provided specific guidance to departments and agencies on the preparation of Horizontal Results-based Management and Accountability Framework's (RMAF) to help guide such efforts. In order to address the challenges facing the CSSP, a horizontal RMAF should be prepared, recognizing that the most significant short term value to be obtained from such an exercise is the review and discussion required to come to a common understanding of how the constituent parts contribute to the achievement of broader program objectives and results.

Research Support

A key challenge in the CSSP is the scientific and biological complexity of the risks involved. Detecting biotoxins or pathogenic organisms, determining the risk they present to shellfish consumers, and putting in place the testing or sampling protocols necessary to reduce those risks is dependant on a high level of scientific capability. Supporting the development of that capacity through research focussed on shellfish is necessary if the program is to maintain its effectiveness. Over the past few years the research effort has diminished due to resource constraints, and there is a need to redefine the nature of that research effort and determine how it can best be re-focussed on the needs of the program. At present there has been no systematic assessment of what those research needs are, although scientific and technical staff are well aware of many areas of investigation where Canada needs to rely on the work of other countries to stay current with developing science and technology. Additional effort is required to better define the appropriate level of Canadian investment in shellfish related research.

Performance Tracking and Reporting

In conducting this evaluation, it was evident that there is a significant gap in the information available on resource expenditures and it was difficult to accurately describe the level of activities and the associated resources due to the lack of budgeting and financial tracking capability for CSSP related activities. There is also a gap in the performance assessment framework for the program, hindering the ability to assess performance over any period of time.

Future planning and decision making efforts will require significantly enhanced financial and performance information.

1. Introduction to the Evaluation

1.1 Background to Evaluation

This evaluation was undertaken in response to a request from senior management at the three partner departments and agency, Canadian Food Inspection Agency (CFIA), Environment Canada (EC), and Fisheries and Oceans Canada (DFO), to conduct an independent and objective summative evaluation of the effectiveness and efficiency of the current Canadian Shellfish Sanitation Program (CSSP).

As a summative evaluation, this assessment was designed to:

• Determine progress in delivering identified outputs and achieving the anticipated results related to performance and delivery of the CSSP between 2000/01 and 2005/06,
• Provide information to assist the CSSP partners in addressing alternative governance and delivery options and developing recommendations for improvement in program delivery and management², and
• Assess the degree to which appropriate programs, processes, and procedures have been put in place to ensure effective and efficient expenditure of resources.

This document contains the results of the "Summative Evaluation of Canadian Shellfish Sanitation Program" and is organized into four main sections:

• Section 1 outlines the purpose and objectives of the evaluation, provides an introduction to the CSSP, and includes a summary of the evaluation methodology,
• Section 2 documents the evaluation findings, from the perspective of each of the three core federal departments,
• Section 3 provides the evaluation conclusions, and
• Section 4 provides recommendations.

In addition, the report includes a number of Appendices containing more detailed information on key partner program outputs, a CSSP cost analysis report, and a summary report on an alternatives workshop report held on March 15, 2007.

1.2 Background on the Canadian Shellfish Sanitation Program

The CSSP is a long-standing Government of Canada activity. The program dates back to 1924-25, when regulations were developed in response to an epidemic of typhoid fever following the consumption of contaminated oysters in the United States, which killed 150 people. Canada passed regulations under the Fish Inspection Act in 1925 requiring that imported oysters be accompanied by a certificate showing that they were a "safe food product". The states of New York and Massachusetts extended certification requirements to shipments consigned to their markets and in April of 1948, a formal agreement was signed between the USA and Canada.

Initially, the Department of National Health and Welfare and the Department of Fisheries were the federal agencies responsible for administering the 1948 agreement in Canada. Health and Welfare was responsible for:

• Paralytic shellfish poison bio assays and recommendations for closures;
• Certification of the British Columbia Shellfish Program;
• Issuance of plant certificates; and
• Surveys of shellfish growing areas in the Atlantic Provinces and Quebec.

The Department of Fisheries was responsible for:

• Inspection of plants and products;
• Management of paralytic shellfish poison control programs in the Atlantic Provinces and B.C.;
• Inspection of U.S. imports;
• Patrol of closed areas;
• Preparation and promulgation of shellfish closure regulations; and
• Land surveys for installation of shellfish area closure boundary markers.

Responsibilities for the shellfish program were reassigned in 1971, when Environment Canada - Fisheries Service (EC) assumed operational responsibility for all of the CSSP except for the bioassay portion of Paralytic Shellfish Poisoning (PSP) control, which remained the responsibility of Health and Welfare Canada. Environment Canada's Fisheries Service was designated the Canadian agency responsible for administering the 1948 agreement. In addition, EC was responsible for sanitary control of shellfish growing areas and harvesting and processing of shellfish in B.C.

In a subsequent reorganization in 1979, shellfish control activities were assigned to the newly formed department of Fisheries and Oceans Canada (DFO), with the exception of water quality assessment and classification of shellfish growing areas, which remained within EC. In 1988, the bioassay portion of the program (previously the responsibility of Health and Welfare Canada) was incorporated into the responsibilities of DFO to improve closure response times.

In February 1990 a Memorandum of Understanding (MOU) was signed between the Department of the Environment and the Department of Fisheries and Oceans defining their respective roles and responsibilities for the Canadian Shellfish Sanitation Program. The Canadian Food Inspection Agency (CFIA) was created in 1996 and the MOU was revised in 2000 to reflect the transfer of Fish Inspection Directorate activities from DFO to the new Agency.

The CSSP is currently managed jointly by the Canadian Food Inspection Agency (Fish Inspection Branch), Fisheries and Oceans Canada (Resource Management and Conservation and Protection Branches), and Environment Canada (National Marine Water Quality Monitoring Program). Although not considered a formal partner for implementing the CSSP, Health Canada also plays an important role in developing the standards for water quality and product.

The current legal basis for the CSSP is derived from the Fisheries Act (sections 36 to 42), and the Management of Contaminated Fisheries Regulations in DFO, and the Fish Inspection Act and the Fish Inspection Regulations in CFIA. However, the day-to-day operations of the program are governed by:

• A Memorandum of Understanding that describes the roles and responsibilities of the three partner departments/agency (CFIA, DFO, EC);
• Bilateral Agreements with major trading nations (e.g. United States of America, European Union); and
• A comprehensive Manual of Operations, which serves as a reference document and provides the operational policies and procedures for government staff involved with duties related to the classification and patrolling of shellfish harvesting areas and the harvesting, processing and distribution of shellfish.

1.3 Program Objectives

The partner departments/agency work collectively to achieve two major program objectives:

• Continue to protect the public from the consumption of contaminated shellfish (bivalve molluscs more specifically), by controlling the recreational and commercial harvesting of shellfish within Canada, as well as of shellfish imported to Canada.
• Fulfill Canada's international obligations to meet the terms of bilateral agreements (e.g. United States of America, EU. etc).

1.4 Roles and responsibilities

As a program jointly managed by the three federal partners, the overall responsibility for delivering the program lies with the President of the CFIA and the Deputy Ministers of Environment Canada and Fisheries and Oceans. The roles and responsibilities for each partner are formally defined in the Memorandum of Understanding (MOU), which was signed on March 1, 2000. The following section summarizes the respective roles of each partner and the horizontal or cross departmental/agency approaches put in place to deliver the program.

1.4.1 Formal Roles Established Under the MOU

Canadian Food Inspection Agency

Roles and Responsibilities
The Agency's primary responsibility is to make certain that a comprehensive program for ensuring food safety is in place and operating throughout all of Canada. CFIA's role with respect to the CSSP is defined in the Fish Inspection Act, the Fish Inspection Regulations, and the Food and Drugs Act. The Agency's key responsibilities related to these pieces of legislation are defined in the MOU and include:

• Registration and inspection of shellfish processing and depuration facilities for exporting;
• Licensing of importers and inspecting imported shellfish;
• Administering the marine biotoxins monitoring program, which include harvest area monitoring;
• As required, the administering of other harvest area monitoring programs, not covered by EC, such as testing of shellfish for Vp (Vibrio parahaemolyticus) in BC and salmonella for exports to the EU.
• Recommending the closure and openings of harvesting areas to DFO based on the results of the CFIA monitoring programs and maintaining records;
• Laboratory QA/QC/Certification;
• International liaison; and
• Acting as the primary contact for external audits on the CSSP.

CFIA's CSSP-related activities are captured under two major programs: the Marine Biotoxin Monitoring Program and the Quality Management Program. As the federal agency responsible for the management of the Marine Biotoxin Monitoring Program, CFIA routinely tests shellfish samples for Paralytic Shellfish Poisoning (PSP), Amnesic Shellfish Poisoning (ASP), and on a limited basis Diarrhetic Shellfish Poisoning (DSP). CFIA also tests for salmonella for shellfish exported to the EU.

The Agency plays a prominent role in regulating the processing and preparation of shellfish for the market, including post-harvest monitoring and sampling. The Quality Management Program (QMP), an inspection program based on Hazard Analysis Critical Control Point (HACCP) principles, is the main control system used by CFIA to monitor the safety and quality of fish and seafood. The Fish Inspection Regulations require all federally registered fish processing plants in Canada to develop and implement a Quality Management Program (QMP) plan. As described in the Fish Inspection Regulations, all establishments that process fish and seafood for export or inter-provincial trade must be registered with the government of Canada.

CFIA Delivery and Management
The Programs Branch is responsible for developing program policies, reference standards, liaising with other federal government departments and other foreign governments. Within the Programs Branch, the Fish, Seafood and Production Division (FSPD) is the division responsible for the overall management and co-ordination of CSSP-related activities at CFIA. The FSPD is part of the Program Branch and consists of a National Capital Region, West and East section.

The Science Branch focuses on testing lab samples taken as part of the Marine Biotoxin Program and the QMP, conducts laboratory evaluations and carries out limited research-related activities.

Operations Branch inspectors conduct compliance verification of registered processing facilities, take product samples for testing by the Science Branch and take the appropriate enforcement actions. Currently there are approximately 200 inspectors in the Operations Branch that are involved in CSSP activities.

Department of Fisheries and Oceans

Roles and Responsibilities
Under CSSP, DFO is responsible for ensuring that bivalve molluscan shellfish are harvested from approved growing areas (i.e., from open areas or closed areas where there is a licence for relay or depuration operations). The authorities under which DFO participates in the CSSP are from the Fisheries Act and the Management of Contaminated Fisheries Regulations.

DFO's key CSSP responsibilities under the Memorandum of Understanding include:

• Opening and closing shellfish growing areas;
• Posting the closure signs in harvesting areas, patrolling and enforcing shellfish closures in accordance with the Fisheries Act;
• Controlling shellfish relaying operations and harvesting for depuration operations;
• Implementing DFO portions of jointly developed Management Plans for "Conditionally Approved" areas;
• Providing information to stakeholders on opening and closures of shellfish growing areas;
• Maintaining records of the opening and closure of shellfish growing areas, as well as records of enforcement patrols; and
• Consult with CFIA and EC prior to the commencement of any developmental or exploratory shellfish fishery or the issuance of any new licences or permits.

DFO Delivery and Management
At the national level, CSSP is led by the Director Resource Management. In each Region, the Regional Director Fisheries and Aquaculture Management leads the program, with support from the Resource Management Branch and the Conservation and Protection Branches. The program is delivered at the regional level as part of the Department's Integrated Fisheries Management Plans. There is limited senior management involvement in CSSP in HQ and the Regions, as the program is operationally based and highly decentralized.

Environment Canada

Roles and Responsibilities
Environment Canada's CSSP responsibilities are implemented through the National Marine Water Quality Monitoring Office. Primary responsibilities include undertaking sanitary and water quality surveys in order to make recommendations on the classification of shellfish growing areas. Secondary responsibilities include the promotion of pollution prevention and remediation of shellfish growing areas.

EC does not have any specific regulatory responsibilities related to the CSSP. However, EC has administrative responsibility for Section 36-42 of the Fisheries Act, related to water pollution control under the General Provision and Regulations under the Act. As well, Environment Canada has a mandate to monitor and report on environmental quality under the Canadian Environmental Protection Act, Part 3.

Environment Canada's key responsibilities under the Memorandum of Understanding include:

• Conducting comprehensive surveys, annual reviews and re-evaluations of shellfish growing waters;
• Conducting shoreline surveys and determining sources of point and non-point pollution;
• Maintaining records, data bases, sectoral maps, survey reports, central files and other documents in support of classification action and administrative reviews by internal and external auditors;
• Providing QA/QC for laboratories and sampling; and
• Promoting pollution prevention, regulatory compliance, remediation and restoration of shellfish growing areas, together with federal / provincial / municipal agencies and other stakeholders.

EC Delivery and Management
The National Marine Water Quality Monitoring Office (previously called the Shellfish Water Quality Protection Program), is located in the Science and Technology Branch of Environment Canada. This is one of several water quality monitoring programs within the Science and Technology Branch, but the Office's sole mandate is to deliver on the Department's responsibilities under the MOU of the CSSP.

Environment Canada has Marine Water Quality Monitoring Offices located in each of the three CSSP Regions (Pacific, Quebec and Atlantic), as well as in the National Capital Region. The National Marine Water Quality Monitoring Office is led by a Manager, and each Regional Office is led by a Head who is supported by Area Coordinators and other scientific / technical staff.

While the outputs for EC in each region are the same, the delivery approach of Environment Canada's program varies slightly between each region. For instance, with regard to the Water Quality Monitoring function, EC in Atlantic tends to use primarily employees and students, as well as agreements with a Provincial government and community groups; Quebec completes the water quality monitoring through contractors only; and Pacific uses contractors, EC employees and volunteers. In contrast, all three regions conduct the shoreline surveys using Environment Canada employees exclusively. With respect to the frequency of water quality monitoring, sampling timelines are variable due to geographic and climate factors.

In Prince Edward Island there is a unique method of delivering on the CSSP responsibilities. In this province, the Department of Environment, Energy and Forestry collects and analyzes water samples under an equal cost-sharing Agreement with Environment Canada. Furthermore, under this Agreement, all shellfish growing areas in PEI are sampled annually.

1.4.2 National and Regional Program Coordination

A CSSP Steering Committee consists of the Assistant Deputy Ministers of DFO and EC and the Vice-President of CFIA. This committee provides oversight and direction to the program. The ADMs Steering Committee established the National Interdepartmental Shellfish Committee (NISC) with responsibility for coordinating the implementation of the MOU. The role of the NISC is as follows:

Table 1 - Roles of the NISC

• Review and consider presentations by provincial/territorial governments, First nations, shellfish industry and other stakeholders on CSSP-related matters;
• Review the regional assessment/activity reports;
• Discuss and assess CSSP implementation to address national shellfish-related legislative, regulatory, policy, and/or procedural issues of mutual concern, including proposed amendments to the CSSP Manual;
• Establish sub-committees and working groups, as required, to deal with specific issues and develop appropriate policies and/or procedures for dealing with them;
• Providing policy recommendations to the CSSP DGs Operations Committee for decision/action and overseeing and/or facilitating the implementation aspects of decisions made by the CSSP DGs Operations Committee;
• Enhance over interdepartmental communication and co-ordination in the delivery of the CSSP; and
• Prepare an annual CSSP report, as established in the CSSP interdepartmental MOU, Section 5b.

CFIA provides the Chairperson for NISC, which reports to, and takes direction from, the Director Generals (DGs) Operations Committee to facilitate the operation of the NISC. NISC meets twice a year (once in person and once by teleconference) to fulfill its obligations as specified in the MOU and the NISC Terms of Reference (ToR). Recommendations from the NISC and the annual report on program delivery are forwarded to the Director General of CFIA, DFO and EC for review and approval.

The Director Generals Operations Committee consists of Director Generals from each of the three federal partners and meets regularly to provide guidance and information to the NISC with respect to CSSP activities and operations.

The NISC is supported by Regional Interdepartmental Shellfish Committees (RISCs) for the Pacific, Quebec, and Atlantic³ regions; these committees consist of members from all three departments at the regional level, relevant provincial authorities, and industry representatives. The purpose of the RISCs is to provide advice to CFIA, DFO and EC senior management and the Chair of NISC on matters pertaining to the classification of shellfish growing areas for bacterial contamination and biotoxins and management of shellfish resources in the regions. Roles of the RISC are as follows:

Table 2 - Roles of the RISC

• To review shellfish growing area survey reports and make decision on classification recommendations in the region;
• To advise in writing the Regional Director General of DFO pertaining to the classification of shellfish growing areas based on bacterial contamination;
• To review and discuss changes to biotoxin survey requirements/location and/or closures;
• As issues arise, to review the policies, procedures, criteria and regulations affecting the implementation and the regional application of the CSSP, including making recommendations to the NISC if required;
• To make recommendations to the senior management of CFIA, DFO and EC regarding regional shellfish growing area survey needs and priorities;
• To review submissions from interested parties for potential referral and/or presentation to the NISC;
• To recognize provincial shellfish working groups to identify survey requirements for bacterial and biotoxin on a provincial basis and to identify policy issues related to the delivery of CSSP in consultation with industry and other stakeholders; and
• To prepare an annual CSSP interdepartmental regional report, for submission to the NISC by June of each year.

1.5 Evaluation Objectives and Scope

The purpose of the evaluation was to assess the efficiency, effectiveness and limitations of the CSSP from a broad, whole-of-program perspective, as well as from the perspective of each of the three partner departments (CFIA, DFO, EC) and the three broad delivery regions (Atlantic, Pacific, and Quebec).

Objectives

The objectives of the evaluation were to examine and make recommendations with respect to:

1. The degree to which the expected outcomes associated with the program objectives are being realized;
2. The risks and liabilities faced by the Government of Canada in managing the CSSP and striving to deliver on those program objectives;
3. The effectiveness of the current governance/administration/coordination structures, processes put in place to manage risks, and support program delivery and decision making; and
4. Program design and the impact the chosen program delivery structure has on overall program performance.

To build the necessary evidence to comment on the evaluation's objectives, the evaluation focused on answering a number of key questions that apply to the program as a whole, the activities of the individual partner departments, and the individual delivery regions within those departments. The key questions assessed by the evaluation included:

1. Success/Achievements:
   1. To what extent has the CSSP met its objective and expected outcomes?
   2. How have unintended results (positive and negative) been addressed?
   3. What are the risks and liabilities associated with the delivery of the CSSP? How effective have program strategies been at mitigating them?
2. Governance/Rationale:
   1. To what extent is the mandate of the Program consistent with the mandates and roles of the partner departments as well as with current Government of Canada (GOC) priorities?
   2. To what extent is the governance of the Program being delivered in an effective manner?
   3. What are the impacts of the current Program structure on Program delivery and performance?
   4. Are the roles and responsibilities, including authorities, involved in the CSSP clear, well understood and delivered as intended?
   5. What are the major external and internal forces influencing the performance of the CSSP? What changes in Program design/delivery would be required in order to adjust?
   6. Are there any gaps, overlaps, additional players or other considerations that need to be taken into account in the program design and delivery?
3. Expenditure Analysis/Alternatives:
   1. What does it cost to deliver the CSSP as a whole, and each of its major activities on a per unit basis? Which costs are borne by each of the federal departments? Which costs are borne by others? Where are the most pressing cost-pressures for the Program? Are there program elements that are not being delivered as a result of resource limitations? What are the options for providing the required resources?
   2. What alternative approaches to the current design and delivery of the CSSP would allow the Program to meet its objectives and mandate more effectively? What might the cost implications be?

Scope

The evaluation covered program delivery over the six-year period from April 1, 2000 through March 31, 2006. Recent developments, contextual factors and other issues arising in the period beginning April 1, 2006 through October 31, 2006 were also considered and documented as appropriate.

From an organization and process perspective, the scope of the evaluation included:

• The activities and outputs of cross-departmental governance and administrative support structures;
• The activities and outputs of regional, national and international entities established to support program delivery and decision making;
• Canadian Food Inspection Agency program and activities directly related to the CSSP;
• Environment Canada programs and activities directly related to the CSSP; and
• Fisheries and Oceans Canada programs and activities directly related to the CSSP.

Geographically, the scope of the evaluation included all of Canada and all Canadians. However, a particular emphasis was placed on the 3 regions and six provinces with active commercial and recreational shellfish operations, namely: Atlantic Canada (PEI, NB, NS, NL), Quebec (Gaspésie, Lower St. Lawrence, Magdalene Islands, and North Shore), and Pacific (BC).

1.6 Evaluation Approach and Methodology

The evaluation is summative in nature. It measures progress towards achieving results, and the effectiveness and efficiency of the current Canadian Shellfish Sanitation Program (CSSP).

The evaluation was organized in accordance with the evaluation questions as provided in Appendix A.

This evaluation is "evidence-based." That is, its conclusions and recommendations are based on objective, quantitative and documented evidence to the fullest extent possible. Where possible, findings and conclusions were drawn from documented evidence, however, information obtained from interviews was incorporated into the evaluation to provide context for formulating findings and conclusions when documented evidence was not available.

The evaluation was conducted in accordance with the work plan described in the Evaluation Plan prepared by Stratos Inc. The major project phases included:

• Phase I: Evaluation Planning
• Phase II: Data Collection and Review
• Phase III: Analysis and Development of Findings
• Phase IV: Debriefing
• Phase V: Reporting

The evaluation process involved gathering multiple perspectives across multiple lines of enquiry, including:

• Document review and interviews with responsible program personnel and managers from the three federal partners;
• Interviews with 57 national and regional program staff;
• Interviews with 25 key informants having in-depth knowledge of the program;
• A workshop dedicated to exploring implementation and delivery alternatives; and
• A cost analysis of the program's key costs and expenditures.

Departmental personnel validated all findings from this evaluation, and a series of presentations were held to debrief key managers in all the partner departments/agency.

Where possible, our conclusions and findings were drawn from documentary evidence. However, we have relied to a large extent on information obtained from interviews with key informants and program staff.

2. Findings By Key Evaluation Questions

This section provides evaluation findings for each of the key evaluation questions outlined in Appendix A. The key findings for these questions are described under the following six headings:

1. Program Outcomes - Section 2.1
2. Pressures, Risks and Liabilities - Section 2.2
3. Governance, Management and Coordination - Section 2.3
4. Program Delivery - Section 2.4
5. Program Costs - Section 2.5
6. Program Alternatives - Section 2.6

The material in Section 2 is organized in a consistent manner, documenting:

• A brief introductory overview of the key area of analysis and where possible, the identification of the expected outcomes associated with the program area as articulated in the program profile (see Appendix B);
• A discussion of the key issues and challenges identified that may limit the federal government's ability to achieve the expected outcomes; and
• An assessment of the extent to which anticipated intermediate outcomes are being realized or are likely to be realized, based on progress to date.

2.1 Program Outcomes

2.1.1 Overview

The Memorandum of Understanding recognizes that the responsibility for implementing the CSSP is shared among three federal partners and that these partners are responsible for fulfilling the program's overarching purpose:

• To provide reasonable assurance that molluscan shellfish are safe for consumption as food by controlling the harvesting of all molluscs within the tidal waters of Canada.

Furthermore, the program's objectives are to:

• Protect the public from the consumption of contaminated shellfish (specifically bivalve molluscs), by controlling the harvesting of shellfish within Canada, as well as shellfish imported to Canada;
• Fulfill Canada's obligations under the Bilateral Agreement between the United States Public Health Service and Health Canada; and
• Fulfill Canada's obligations under other international agreements.

The MOU also highlights the broad expectation that the involved parties will work cooperatively with each other and commit to strive to enhance the efficiency and effectiveness of CSSP, participate in ongoing monitoring programs and implement improvements as required.

Expected Outcomes

The following expected outcomes were identified for the program as well as for all three federal partners (see Appendix B - Program Profile).

Ultimate Program Outcomes

• Reasonable assurance that commercially harvested shellfish in Canada are safe for consumption.
• Reasonable assurance of the safety of the recreational and traditional shellfish harvest.
• The environmental conditions of shellfish growing areas are properly assessed to provide assurance that shellfish are harvested from safe areas and meet export requirements.

Intermediate Partner-Specific Outcomes

2.1.2 Issues and Challenges

There are a number of issues and challenges that affect how the CSSP can achieve its expected outcomes.

Inconsistent Implementation Across All Management Regions
It is recognized that the current CSSP program does not provide coverage for all areas of the country where shellfish are being harvested for personal consumption by recreational and Aboriginal harvesters and the full range of program activities as prescribed in the MOU are not being applied in all respective management regions. Staff indicated that this is primarily due to limited resources that affect the federal partners' abilities to conduct all of the activities prescribed in the MOU in all areas. These resource limitations affect the partners' abilities to deliver on a number of core activities such as sampling for pathogens, inspecting federally registered fish processing facilities, patrolling harvest areas, and raising the public's awareness with respect to health and safety. Due to Canada's vast geography and the remoteness of some harvesting areas, it would not be possible to prevent the harvesting of shellfish from all closed areas.

We also note that the current policy and scope statement for the program is not clear in terms of how broad the coverage of the program should be. This is creating difficulty in determining where additional program effort is warranted.

Furthermore, interviews with program staff and managers from all three partners highlight that inconsistent implementation is posing a potential risk to the health and safety of Canadians, to the economic viability of commercial shellfish activities, and to domestic and international commercial markets.

Internal Coordination Issues
In terms of developing a coordinated, federally consistent approach to implementation of the activities of the MOU, the evaluation found evidence that CSSP partners experience challenges with respect to planning and coordinating their CSSP responsibilities and activities. With the responsibilities shared across three departments and spanning three broad management regions, core activities are delivered inconsistently across the regions and within the departments. Roles and responsibilities between partners are occasionally misunderstood (e.g. responsibility for monitoring products after relay from contaminated areas). Furthermore, partners are not coordinating activities under a common, comprehensive plan for the program. More detailed information on coordination issues and challenges as they pertain to the program's governance and delivery is included in sections 2.3 and 2.4.

Program Performance Assessment Framework
Assessing program outcomes and results is complicated in that there is no agreed upon performance assessment framework in place for the program as a whole. Partner departments prepare annual performance summaries, but these tend to be a compilation of data and information on activities undertaken by the three partners. No overarching indicators exist for tracking and reporting on program results.

2.1.3 Assessment

In the areas where the program is being operated, the CSSP is providing reasonable assurance that commercially harvested shellfish are safe for consumption and that the federal partners are fulfilling their obligations under their respective areas of responsibility. Interviews with program managers and senior staff emphasized that the CSSP is meeting the program's ultimate outcomes and objectives as prescribed in the MOU. Additional evidence to support the assessment that the program is meeting its expected outcomes include: the program's track record in preventing serious illnesses or outbreaks; US and international trade markets remaining open; Canada is meeting other countries' shellfish sanitary standards for exported products; and continued consumer confidence in the safety of consumer shellfish products.

Despite the reasonable assurance that Canadian shellfish are safe for domestic and international consumption, the program cannot provide 100% assurance. It is important to note that coverage of the program does not extend to all areas of the country. In some cases, shellfish are being harvested for domestic consumption (including recreational, subsistence and/or ceremonial purposes) from areas that are unclassified or closed. Commercial harvesting from closed or unclassified harvesting is prohibited.

Although the last recorded case of shellfish fatality occurred in 1987 (due to the presence of domoic acid in PEI mussels), there have been other incidents of shellfish-related illnesses over the past 20 years that have affected the health and well-being of Canadians, including:

• 2004: 79 cases of Norovirus in BC
• 1998: 100+ cases of Vp outbreak in BC
• 1992: 150 cases of Norovirus in Quebec
• 1987: last recorded incidents of shellfish related fatalities
• Most years: There are Paralytic Shellfish Poisoning (PSP) reported illnesses linked to the consumption of self-harvested shellfish

2.2 Pressures, Risks and Liabilities

2.2.1 Overview

The Government of Canada has been monitoring the harvest and sale of bivalve molluscs since it was determined that the 1924 outbreak of typhoid fever in the United States was attributable to the consumption of shellfish. This event provided the impetus to develop a greater understanding and awareness of the human health risks associated with shellfish. The outbreak also propelled the Government of Canada to establish the CSSP in 1948 to address related human health concerns and maintain a healthy trade relationship with the United States. Since its inception in 1948, the program has experienced a shift in priorities that has forced the program to evolve and adapt to address other incidents beyond its original focus, causing it to become more responsive to new and emerging risks and liabilities. These risks and liabilities are caused by internal and external pressures which shape the federal partners' ability to conduct activities consistently and effectively throughout all shellfish management areas.

2.2.2 Issues and Challenges

External and Internal Pressures
Historically the CSSP focused primarily on conducting activities that maintained the health and viability of the wild commercial harvest intended for domestic consumption and international export. Currently the CSSP is affected by a number of external pressures that influence the federal partners' ability to meet new and/or increased demands imposed by the aquaculture industry, the Aboriginal, subsistence, and recreational harvesters, and the increasing expectation to meet national and international standards of care requirements. Specific pressures identified during the evaluation include:

• Aboriginal harvesters are demanding that the CSSP be delivered in areas where the CSSP has not traditionally been delivered, such as Nunavut and Northern British Columbia, and in support of the Government of Canada's fiduciary responsibilities to Aboriginal peoples in Canada;
• Canada's emerging aquaculture industry, most notably in British Columbia, Nova Scotia, and Prince Edward Island, is creating additional demands for services on the CSSP partners;
• International trading partners are placing additional pressures on the CSSP to demonstrate that the CSSP maintains standards comparable to their shellfish programs. The United States and the European Union conduct regular audits on the CSSP to ensure that the program is equivalent to the US National Shellfish Sanitation Program guidelines and the European Commission shellfish standards respectively. The findings from these audits are putting pressure on the CSSP and its limited resources;
• Increased coastal development is affecting the sanitary water quality in some shellfish growing areas;
• The Government of Canada's standard of care in protecting health and providing access to safe resources has been influenced by touchstone events such as the Krever Inquiry, the Marshal Decision, and the Walkerton Incident; and
• The Food and Agriculture Organization's Codex Alimentarus Commission's food standards and guidelines for testing shellfish are placing additional pressures on the program to monitor and test for more biotoxins.

Internally within the departments and agency, the CSSP is affected by limited resources, as well as the demands to deliver the CSSP in conjunction with other programs. Evidence provided by program managers and senior staff emphasizes that scarce resources are limiting opportunities to open new harvest areas, conduct scientific research, and carry out public awareness campaigns.

While the consolidation and specialization of CFIA lab resources has resulted in benefits associated with specialization of equipment and specialized staff resources, some staff have suggested that the consolidation of lab resources for analyzing samples, has resulted in some delays in obtaining sample results. For example, the current turnaround time for testing for Diarrhetic Shellfish Poisoning in Atlantic Canada is seven days. This delay has been attributed to the centralization of laboratory services to the Halifax area. In some cases, this delay poses a risk that contaminated shellfish will have entered the market before test results are available. Staff commented that while laboratory services are delivered on the basis of their available resources and all the programs they were delivering, CSSP is an operational program that requires timely test results. With no dedicated CSSP resources, laboratories were not always in a position to accommodate the CSSP operational delivery requirements.

Aside from the activities carried out by Environment Canada's Marine Water Quality Monitoring Office, the other federal partners conduct their CSSP activities in conjunction with other programs or within other priority areas. Evidence provided by CSSP program managers and staff highlights the fact that competing departmental priorities affect the resources and time available to meet CSSP requirements. As a result, the CSSP is not a high priority for all of the federal partners, (e.g., at DFO, approximately 5% of all patrol hours are dedicated to CSSP).

Changing environmental conditions, such as the increased presence of marine biotoxins in new areas or their observed presence in Canadian waters for longer periods of time during the year, are also placing increased pressure on CFIA. Although the Canadian Food Inspection Agency regularly tests molluscs for Paralytic and Amnesic Shellfish poisons, and occasionally for DSP in the Pacific Region, current resource levels limit CFIA's ability to be proactive and determine whether the additional biotoxins identified by the Codex Committee on Fish and Fishery Products, namely Azaspiracidic Poison (AZP) or Neurotoxic Shellfish Poison (NSP) are present in Canadian waters. These limitations are increasing the likelihood that contaminated shellfish may enter the market.

Risks and Liabilities
The major risk and liability associated with the CSSP are the potential to negatively affect public health (death or illness) and lost economic opportunities due to the presence of contaminated shellfish (i.e. pathogens or biotoxins) in the marketplace.

Contaminated shellfish have the potential to enter the local markets because delivery of the CSSP cannot provide complete assurance that all harvested and consumed shellfish were harvested from classified growing areas and processed through certified processing facilities. Furthermore, public surveys indicate that there is a low level of public understanding around the safety of all shellfish for consumption. There is no ability to track or monitor all products harvested for personal consumption, and limitations in the ability to track product sold intra-provincially. These factors could increase the possibility of contaminated shellfish reaching consumers. Program staff also indicated that recreational and Aboriginal harvesters are the most likely to consume shellfish harvested from unclassified or closed areas because current activities focus on commercial harvesters.

From a commercial perspective, risks and liabilities concerning public health can have negative political and economic effects. When contaminated shellfish negatively affects domestic and international public health, relationships with export partners are weakened and the Canadian shellfishery and aquaculture industry suffers. Additionally, the Government of Canada's reputation is at stake if Canadians and/or trading partners believe that the government cannot deliver a national program that ensures the safety of products in the domestic and international markets. The impact of this risk was seen during the Domoic Acid outbreak in 1987 when the entire fishing industry almost collapsed due to concern over the risks of shellfish consumption. The fishing industry is worth more than CDN $5 billion a year, provides more than 130,000 jobs, and is the economic mainstay of approximately 1,500 communities in rural and coastal Canada. Although the CSSP maintains a good commercial track record in terms of providing shellfish that is safe for consumption, there have been a few incidents where products with contaminant levels slightly above the program's standards have entered the market.

Workload related pressures also create potential shellfish related risks. Evidence provided over the course of the evaluation also noted that only 56% of planned Compliance Verifications audits for processing plants' Quality Management Programs (QMP) are completed on an annual basis. It should be noted that the target for completion of audits applies to all establishments and encompasses all fish related products, and that the vast majority of these establishments do not process shellfish. In addition, the risk model employed by CFIA Operations to determine the priority for compliance verification does result in a higher rate of compliance verification for establishments that produce higher risk products including shellfish. However, the overall level of compliance verification audits conducted does make it more difficult for CSSP to demonstrate than an adequate level of verification is being carried out. Some staff indicated a concern that the actual versus planned Compliance Verifications rate may not provide a sufficient level of assurance that processing plants are in conformance with program requirements. Again, this demonstrates the challenges in providing a high level of assurance that all products reaching the consumer are safe.

2.2.3 Assessment

Overall, Canadian shellfish growing waters carry relatively low risks for some pathogens such as salmonella and hepatitis A, but not necessarily for all biotoxins or bacterial contaminants. The single greatest risk is associated with contaminated products entering the market and affecting the health of consumers resulting in personal sickness and possibly death. As well, these impacts can result in a loss of both shellfish and fish sales domestically and internationally (landed value of $5 billion, and 130,000 jobs). This risk can have legal implications and can tarnish the image of and confidence in the CSSP and more broadly, the Government of Canada.

Risks and liabilities are addressed largely at the operational level by specific activities such as sampling regularly for pathogen indicator organisms, pathogens and marine biotoxins, patrolling closed areas and enforcing fishery regulations, evaluating testing as per the Quality Management Program at the processing plant level, identifying pollution sources, providing educational outreach sessions, and ensuring that emergency closure protocols are in place when contaminated shellfish are identified. At the program level, the national and regional committees address risks and liabilities. These committees meet several times a year and provide a forum for discussing pressing risks and how they should be addressed.

Regional staff also apply regional-specific mitigation measures, that is, unique mitigation measures for addressing the cultural nuances within their respective regions. For instance, Quebec and the Pacific region apply the precautionary approach to their unclassified areas by applying closed area conditions to these areas.

These methods have been effective at managing risks where the program is delivered. Despite an implicit risk based approach to some program areas (e.g. DFO patrols, CFIA inspection programs, EC sampling programs) there has been no formal program wide integrated risk assessment to help target risk reduction strategies.

2.3 Governance, Management and Coordination

2.3.1 Overview

In accordance with the Memorandum of Understanding (MOU), CFIA, DFO, and EC have shared responsibility for managing the CSSP in three broad regions⁴, the Pacific, Quebec, and Atlantic regions, which encompass a total of six provinces⁵.

At CFIA, responsibility is shared across the Program, Operations, and Science Branches. Within the Program Branch, the Fish, Seafood and Production Division is responsible for the overall management and coordination of the CSSP activities at CFIA.

At DFO, the Director of Resource Management leads the CSSP, within the department's Fisheries and Aquaculture Management (FAM) Sector. At the regional level, the Regional Director of Fisheries and Aquaculture Management leads the program with support from the Resource Management Branch (fisheries management) and the Conservation and Protection Branch (enforcement).

At EC, the National Marine Water Quality Monitoring Office, located in the Science and Technology Branch, is responsible for delivering on the department's responsibilities described in the MOU. Marine Water Quality Offices are located in each region; and led by a Head who is supported by Area Coordinators and scientific and technical staff. Regional Heads report to the National Manager in the National office in Ottawa. Currently, EC is the only CSSP partner coordinating all of its CSSP activities through a single program and organizational unit.

At the program level, the National Interdepartmental Shellfish Committee (NISC) and Director General (DG) Operations Committee oversee the management of the CSSP. The NISC is supported by Regional Interdepartmental Shellfish Committees (RISCs) for the Pacific, Quebec, and Atlantic regions.

An Assistant Deputy Ministers (ADM) Steering Committee is also in place to oversee the CSSP. This committee meets once a year, or more frequently as required.

The MOU is the keystone document providing an overview of the program's purpose, governance structure as well as highlighting the federal partners' roles and responsibilities. Similarly, the CSSP Manual of Operations is the key reference document for program staff involved with CSSP program delivery. The Manual outlines the applicable policies and procedures that apply to the CSSP.

2.3.2 Issues and Challenges

Coordination and Planning
Program managers and senior staff consistently emphasized that the CSSP is not being governed as a single, coherent program and that the program lacks overall corporate level planning, coordination and oversight functions. Federal partners are operating the program in line with their respective departmental or agency mandates and are carrying out a number of activities which are consistent with their responsibilities outlined in the MOU; however, the absence of a cohesive management regime has resulted in a lack of strategic guidance and direction for the program as a whole, and the absence of a coordinated implementation plan. This has created a program that is delivered primarily through the partner departments and agency's regional program structures, focusing on operational issues within the partners' mandates, as opposed to being designed to meet the fully-integrated overarching objective for the program.

National and Regional Governance
The regional and national committees tasked with governing the program are working reasonably well and provide an adequate forum for exchanging ideas and discussing pressing issues. However, the NISC does not have a role, nor is it functioning as a program-wide decision making body that provides policy direction or establishes policies and plans relevant to CSSP delivery. Similarly, the RISCs are working well as a forum for information exchange and discussion amongst the federal partners. However, although some industry and provincial representatives do participate in the meetings, the committees' effectiveness is hindered by inconsistent or inadequate representation from industry and/or provincial authorities. Collectively, these committees have difficulty making timely decisions and reaching a consensus on delivery issues.

There are no dedicated resources in place to ensure that the cross cutting program requirements and issues are addressed from a CSSP, as opposed to departmental or agency, perspective. As a result, there is difficulty in addressing CSSP related issues that go beyond the mandates of the three partners.

In addition, under the current approach to program delivery, there are no shared performance accountability mechanisms in place to ensure that partners are providing the services and level of activity required meet CSSP objectives.

Roles and Responsibilities
The partner department's roles and responsibilities for program delivery at the operational level are described in the MOU; they are clear, well understood, and complement each partner's respective mandates. Although the MOU provides good guidance in terms of defining roles and responsibilities, the MOU is outdated and needs to be updated to reflect changes that have occurred since it was originally signed in 2001. There are a number of areas where the MOU needs to be updated, but in particular, the MOU needs to formalize CFIA's role as the program lead and explicitly explain roles and responsibilities for program coordination, planning and oversight (monitoring and reporting) mechanisms. Roles and responsibilities for cross cutting activities and issues, such as ensuring the CSSP Manual of Operations remains a strong reference tool for program staff conducting CSSP activities, need to be clarified and assigned.

2.3.3 Assessment

Overall, the CSSP is governed by the roles laid out in the MOU, with coordination between departments facilitated through the noted interdepartmental committees. To date, this approach is working relatively well in ensuring the departments are meeting their responsibilities as outlined in the MOU. However, the CSSP lacks a strong centralized governance mechanism that provides integrated planning and coordination support to the CSSP as a whole, and to the involved federal partners for managing the CSSP. Although CFIA has taken the nominal lead for the CSSP, this lead is not fully reflected in the MOU and is currently not well defined or being delivered effectively.

In the absence of a formal centralized governance structure, federal partners are implementing the operational components of the program within their respective mandates, but this is done within departmental and internal program "silos" and is resulting in inconsistent approaches for delivery at the regional levels. Generally speaking, planning and coordination for CFIA and DFO are comprised of input from the regions or internal programs, and does not reflect a strong governance framework for CSSP activities. As mentioned previously, EC is the only CSSP partner coordinating all of its CSSP activities through a single program and organizational unit with an integrated plan for delivery of their CSSP activities.

2.4 Program Delivery

2.4.1 Overview

The MOU is the fundamental document that defines the activities for each of the federal partners. It describes specifically each partner's areas of responsibility and provides details on the major activities to be completed by each partner.

The MOU highlights that CFIA is the lead agency responsible for the handling, processing, importing and exporting of shellfish, and managing the marine biotoxin program. The Agency is also the nominal lead responsible for ensuring that a comprehensive program for food safety is in place and operating throughout all of Canada. The Agency's role with respect to the CSSP is defined in the Fish Inspection Act, the Fish Inspection Regulations, and the Food and Drugs Act.

DFO is responsible for ensuring that bivalve molluscan shellfish are harvested from approved growing areas⁶. The department's legislative authority to implement CSSP activities is embedded in the Fisheries Act and the Management of Contaminated Fisheries Regulations.

EC is responsible for undertaking sanitary and water quality surveys for the purpose of making recommendations for the classification of shellfish growing areas. The department is also responsible for promoting pollution prevention information and remediation of shellfish growing areas. Although the department does not have a regulatory responsibility to conduct CSSP activities, it maintains an administrative responsibility under sections 36-42 of the Fisheries Act which highlights the department's role in water pollution control under the General Provision and Regulations under this Act. EC also has the mandate to monitor and report on environmental quality under Part III of the Canadian Environmental Protection Act.

2.4.2 Issues and Challenges

Regional Variations in Delivery
As previously noted in Section 2.3 of this report, there is no single governance structure for managing the CSSP and the approach for delivering CSSP activities between the federal partners varies. For example, even though EC has a centralized management structure in place, water quality sampling programs are conducted primarily by staff in one region, and primarily by contractors in another. Similarly, DFO and CFIA are operating multiple programs in each of their respective management regions, often designed to accommodate regional differences. While many of the regional differences have evolved/been initiated to better respond to local needs, local availability of the resource and unique operational/geographic conditions, the differences are also resulting in inconsistent delivery within each of these departments. For instance, at CFIA, it is largely contractors and industry who conduct marine biotoxin field sampling in the Pacific region; whereas in the Atlantic region, sampling is conducted exclusively by CSSP staff. From a programmatic view, the dispersed program structure has not allowed for the effective development of national policies, coordinated work plans, or procedures to provide the necessary strategic direction and guidance for delivering CSSP activities in the management areas.

These regional differences in program delivery are not in and of themselves an issue, provided the program differences are in response to differing risk profiles across the regions. In some cases the regional differences were created in response to local availability of the resources and operational / geographical conditions. Delivery within a rigid national approach is not necessary for ensuring the program meets a common health outcome. While some flexibility in application of best practices allows for improvement in program efficiencies and effectiveness, it can also create perceived and/or actual inequities between and within regions. For example practices with respect to industry in-kind contributions, and contract sampling of PSP monitoring vary considerably between the Pacific and Atlantic regions.

Limited Resources are Affecting Activities
The current level of resources is affecting the partners' ability to deliver all activities listed in the MOU. Evidence provided by CSSP program managers and staff at EC highlight that there are limited resources available for pollution prevention outreach activities. At DFO, resources available limit the department's capacity to patrol frequently enough to fully meet USFDA patrol standards. Similarly at CFIA, the Agency does not have sufficient resources to conduct the required Compliance Verification (CV) inspections at federally registered processing facilities.

Lack of Awareness for Recreational and Aboriginal Harvesters
Recreational and Aboriginal harvesters are not provided with sufficient information on shellfish closures and do not have a clear understanding of the health risks associated with consuming contaminated shellfish. The lack of readily available information poses a health risk to non-commercial harvesters, whether it is residents, tourists or local subsistence harvesters, or Aboriginals harvesting for Food, Social, and Ceremonial (FSC) purposes. Similarly, most Canadians do not understand how shellfish are contaminated and underestimate the dangers of eating contaminated shellfish and the ill-effects to human health⁷. As a result, the CSSP cannot assume that Canadians are knowledgeable about shellfish and of the risks associated with eating contaminated shellfish.

Federal-Provincial Relationships
Evidence provided emphasizes that federal and provincial partners are working well but there is concern that federal and provincial shellfish programs are not sufficiently coordinated and greater alignment is needed. CSSP staff commented that provincial regulators often issue aquaculture permits without consulting with CSSP staff and do not consider the Government of Canada's limited resources for implementing CSSP activities. Similarly, CSSP staff feel that the provinces encourage development along shorelines at a rate that forces the closure of open harvest areas by non-point sewage pollution.

Interviews conducted with Provincial regulators highlight that there are regional differences with respect to the level of activities conducted regionally. More particularly, interviewees commented that enforcement, and water quality sampling and surveying are not keeping up with the current demands in the regions. Furthermore, provincial authorities feel that the program focuses heavily on commercial harvesting and less on Aboriginal and recreational uses.

Federal-Industry Relationships
Information obtained from interviews with harvesters, processors, and industry associations indicates that communication and coordination between federal and industry stakeholders is weak and is resulting in a lack of confidence in the CSSP from the private sector. Industry members feel that they are not well represented at the regional level and that generally speaking, their knowledge and opinion with respect to managing the shellfish resource is undervalued. Furthermore, industry members argue that the CSSP lacks the appropriate mechanism to allow for the flexible delivery of the activities needed to address acceptable harvest requirements (i.e required sampling to open harvest areas).

2.4.3 Assessment

Despite limited resources for delivering the program, and the absence of a centralized governance structure responsible for developing a cohesive national work plan, federal partners are carrying out the major operational activities in accordance with the MOU in the areas where the CSSP is delivered. Furthermore, federal partners are concentrating their activities in areas where they believe the greatest health risk and commercial value exist. Sanitary and marine biotoxins sampling programs are strong and are working well to ensure that Canadians shellfish are safe for public consumption. Also, DFO is working well to patrol and enforce shellfish growing areas. Relationships with the respective provinces are generally working well; however there are planning and coordination issues between federal and provincial programs that strain program resources and lead to policy incoherence between the two levels of government (e.g., promotion of the shell fishery and coastal development). Current, mechanisms for engaging industry are inadequate and are creating a negative relationship between the Government of Canada and industry. Surveys indicate that consumers generally have a low level of understanding of the risks associated with consumption of shellfish products.

2.5 Program Costs⁸

2.5.1 Overview

Based on the best available estimates (see noted issues below), the total program expenditures for the CSSP are approximately $14.63 million per year. At the CSSP partner level, CFIA accounts for $7.46 million, approximately 51% of the total program expenditures. Environment Canada and the Department of Fisheries and Oceans maintain comparable expenditures at $3.71 and $3.46 million respectively.

TABLE 1: CSSP Program Costs by Partner and Region (2006-07)

	Pacific	Quebec	Atlantic	HQ	Total
EC	1,089,200	798,000	1,513,000	314,000	3,714,200
DFO	697,000	473,000	2,231,000	55,000	3,456,000
CFIA	2,110,426	1,369,130	3,833,487	150,000	7,463,043
Total Expenses	3,896,626	2,640,130	7,577,487	519,000	14,633,243

In addition to estimating federal resource expenditures on the CSSP, the evaluation also estimated the landed value of the shellfish resource harvested in Canada. DFO Fisheries Management currently tracks landed values from the wild-commercial harvest and the aquaculture industry. Species monitored include: clams, quahogs, geoducks, oysters, scallops and mussels. In 2005, approximately 89,029 tonnes of CSSP-related shellfish were harvested from wild harvest areas. An additional 37,451 tonnes of shellfish were harvested from aquaculture facilities. Combined, the total landed value of these shellfish is approximately $203.8 million⁹. Species harvested from the wild represent roughly 71% of the landed market value, which equals approximately $145.1 million. The remaining $58.7 million is generated from the aquaculture industry.

The fishing industry is an important economic base for Canada's coastal communities, providing opportunities for jobs and economic development over several generations. More than 130,000 Canadians earn their jobs at sea, on inland fisheries, in processing plants, or aquaculture operations. It is a significant business with annual wild harvest and aquaculture production of approximately $2.8 Billion. Maintaining the economic viability of this industry depends on consumer confidence in the safety of fish products, including shellfish.

Table 2: Value of Commercial Landings, By Province ($000)

	Nova Scotia	New Brunswick	PEI	Quebec	NL.	British Columbia	Total Canada
Clams/ quahaug/ geoducks	7,799	2,099	3,087	2,425	15,398	24,742	55,551
Oyster	330	942	6,113	0	0	0	7,386
Scallop	67,024	4,273	629	1,901	8,079	143	82,049
Mussels	0	119	0	15	0	0	134
Total	75,153	7,433	9,829	4,341	23,477	24,885	145,120

 

Canadian Aquaculture Production Statistics ($000)

	NS	NB	PEI	Quebec	NL.	British Columbia	Canada
Clams	0	0	0	0	0	8,378	8,378
Oysters	686	1,950	5,500	0	0	7,959	16,095
Mussels	3,060	550	21,400	980	6,900	278	33,168
Scallops	91	0	0	0	0	988	1,079
Total	3,837	2,500	26,900	980	6,900	17,603	58,720

 

Value of Commercial Landings, By Province ($000) and Canadian Aquaculture Production Statistics ($000)

	NS	NB	PEI	Quebec	NL.	British Columbia	Canada
Grand Total	78,990	9,933	36,729	5,321	30,377	42,488	203,840

2.5.2 Issues and Challenges

Lack of Authoritative Cost Figures
There is no centralized cost-tracking system for the CSSP. As a result, there is not a firm understanding of costs at the program, departmental or regional levels. Currently Environment Canada is the only federal partner tracking its CSSP program expenditures.

Costs Borne By Stakeholders
Departments involved with the CSSP do not have a full understanding of costs borne by non-federal stakeholders. Evidence submitted for the evaluation emphasizes that the industry and provincial authorities accrue significant costs that contribute to the program, and assist with conducting activities that are specified as federal responsibilities under CSSP. Anecdotal evidence highlights that industry accrues costs for water sampling and analysis in newly classified areas and areas under conditional management plans. However, there has been no attempt to quantify these expenditures for the entire program. There also has been no rationalization of the use of stakeholders, and no systemic analysis to evaluate and establish the role and value that stakeholders can provide in assisting with CSSP delivery.

External and Internal Pressures
There are numerous external pressures that are straining or have the potential to put additional pressures on current resource levels. Accommodating a growing domestic industry and the need to meet international requirements established by the EU and USFDA are significant pressures for the CSSP. No effort has been made to better quantify the associated costs for addressing international requirements. Furthermore, CSSP costs are influenced by geographical differences. Activities conducted in isolated areas (e.g. sampling and enforcement in the Northern Coast of the Pacific region) are more expensive and time consuming resulting in increased costs for the program. As a result of these external pressures, federal partners are expected to conduct more activities in existing or expanding growing areas. Concerns were also mentioned about the inconsistent approach to the use of industry stakeholders and their possible role in assisting with CSSP delivery. Evidence gathered during the evaluation emphasizes that CSSP program staff from all three partners feel that the CSSP has reached its capacity and cannot meet all of the existing or emerging demands and pressures.

Internally, federal partners conduct their CSSP activities in conjunction with other programs or priority areas. As previously noted in Section 2.2.2, competing departmental priorities are affecting the resources available and time dedicated to conduct CSSP activities. For example, CFIA lab analysis activities are often completed in parallel with other responsibilities. These lab services are not coded or tracked for the purpose of estimating CSSP expenditures. As a result, partners are experiencing difficulty estimating the true costs for the CSSP program.

Despite the lack of readily available quantitative information, the results of the interview phase of the evaluation highlighted that CSSP activities for some CSSP partners are increasing despite static, or reduced levels of, funding. For example, anecdotal evidence gathered from EC indicates that the department has become more efficient by conducting more sampling with lower levels of funding¹⁰.

2.5.3 Assessment

In the absence of a centralized cost tracking system for the CSSP, there is no firm understanding of costs for the program. However this evaluation was able to estimate that approximately $14.63 million is spent on the CSSP annually. This in turn is offset by the contribution that the CSSP makes to the economic well being of Canadians - a contribution that can be particularly important in small communities in remote regions and in maintaining Canada's robust domestic and international shellfishery. Furthermore, the ramifications of not having a fully developed and consistent shellfish protection program can potentially cost Canada billions of dollars in economic impact. This consequence was well illustrated in the Domoic acid crisis of 1987 that took the lives of three Canadians, resulting in a loss of consumer confidence in fish products and caused an estimated loss of $2 billion (1988 $) on the Canadian economy.

Presently federal partners are delivering the CSSP largely in accordance with the requirements of the Operations Manual, however due to financial constraints, they are unable to implement the full CSSP in all geographic areas of the country. This places some limitation on the extent to which the program is able to provide complete assurance regarding the safety of all harvested products (especially non-commercial). A lack of financial resources to effectively deliver program activities in all areas, increases the possibility of a public health incident due to the consumption of contaminated shellfish which in turn could impact significantly on domestic or international markets.

2.6 Program Alternatives

2.6.1 Overview

Beginning in 1994, significant effort has been made to review alternative approaches to the design and delivery of the CSSP. In July 2004, the CSSP ADMs established a project team (with representatives from all partners) to undertake a CSSP program redesign. The team was tasked to address the growing concern about public safety and federal liabilities associated with implementing the program, in light of decreasing resources and increasing pressures.

The redesign team completed an analysis of the CSSP's legislative mandate, relevant cabinet decisions, the federal government's fiduciary responsibility to determine whether the Government of Canada is meeting the Federal Government's mandate and obligations. A presentation made to the CSSP ADMs in August 2004 outlined various alternatives for CSSP redesigned scope, governance, and program principles. Using the information provided in the August 2004 presentation as a starting point, the redesign team (composed of representatives from CFIA, DFO and EC) undertook several studies and cross-Canada consultations. Discussions with regional representatives from all three departments / agencies were held. The results of this work culminated in the development of a presentation that included several recommendations related to scope, governance, and program principles. This presentation was recommended by the DG's Operations Committee, however it was not presented to the ADM's Steering Committee for approval. This effort was superseded by a decision to conduct the current evaluation.

On March 15, 2007, as part of the evaluation process, a workshop was held for CSSP program managers across the country to consider possible scope for the CSSP, key guiding principles, governance options, and selected activities that could be considered for alternative delivery. The summary report from the workshop is included as Appendix D.

2.6.2 Issues and Challenges

Scope
The results of the March 2007 meeting highlighted that there are diverse views with respect to the future scope of the program. Previous attempts to address program scope have faced significant challenges. A number of key issues concerning the program's responsibilities and limitations remain to be addressed, including the program's ability to address the needs and demands posed by recreational and Aboriginal harvesters. More specifically, there is growing concern amongst program managers that the CSSP does not have the resources to address food safety concerns to Canadians living in remote communities (or provinces/territories) or to fulfill the Government of Canada's fiduciary responsibilities to provide services to Aboriginals for food, social, and ceremonial purposes.

Limitations to Alternative Delivery
The results of the March 2007 workshop indicate that there are several CSSP activities that must be conducted by federal partners because of the partners' policy and regulatory responsibilities to provide services on behalf of the Government of Canada. For example, under the Fisheries Act, the Department of Fisheries and Oceans is responsible for regulating the opening and closing of harvest areas. As a result of this regulatory responsibility, the service cannot be allocated to a third party and must be provided by the Department of Fisheries and Oceans. Therefore, these regulatory responsibilities limit the options for delivery the program (e.g. contracting out, formal agreements with third parties, etc.). Other services that are bound by the partners' regulatory responsibilities include: registering shellfish processing and depuration facilities (export only); licensing importers; licensing commercial shellfish harvesters, etc.¹¹.

Alternatively, a number of services that could be conducted by non-federal parties (i.e. industry, private sector, or provinces) were identified. These services include: harvest area monitoring for marine bio-toxins and non-sewage pathogens; laboratory quality assurance and analysis of shellfish products; patrolling and monitoring activities; and managing records, maps and databases for survey results and reports.

Managing Government Horizontal Initiatives
Government horizontal initiatives are often challenged by common pressures that can affect the ability to deliver activities efficiently, effectively, and consistently. Part of the difficulty stems from the extent to which the individual partners retain or consolidate centralized control of core management functions such as setting priorities, planning, coordinating, and implementing activities. Maintaining a dedicated group of staff to look after the horizontal or cross cutting program activities is necessary for providing these functions and implementing any horizontal initiative. Individual departmental mandates can also create disproportionate demands on the level of senior management engagement, ultimately affecting program governance and delivery. These pressures have affected the current governance and delivery of the CSSP and are further elaborated in Sections 2.3 and 2.4 of this report.

2.6.3 Assessment

The CSSP is a complex program delivered by three federal partners spanning from coast to coast. Options for introducing new governance or delivery alternatives are limited due to the challenges imposed by the program's expansive mandate, the federal partners' regulatory responsibilities, and the common pressures associated with delivering federal horizontal initiatives. Despite these challenges, there are options for introducing stronger governance mechanisms that can provide centralized control of key management functions such as coordination and planning, and management oversight. The results of the March 2007 meeting identified governance options ranging from establishing a joint management program to establishing a Secretariat or separate management board. More information on these options can be further explored in Appendix D, whereas a focused discussion on the program's current governance structure can be viewed in Sections 2.3, and 2.4. The evaluation's proposed recommendation for the program's governance is presented in Section 3.1 of this report.

With respect to alternative delivery of services, there are possibilities that need to be explored including: delivery of field services such as patrolling; and monitoring and support services such as laboratory testing and pollution prevention outreach.

---

¹ The following notional time frames are associated with the recommendations included herein:

• High priority: Short-term: response completed within the next 6 months
• Medium priority: Medium-term: - response completed within the next 12 months

² In July 2004, the CSSP ADMs agreed to establish a project team (with representatives from all partners) to undertake a CSSP program redesign within current resource levels and to deal with the growing concern about public safety and federal liability due to the increasing gap between the demands being placed on the CSSP and available funding.

³ The Atlantic RISC (ARISC) also includes four separate Shellfish Working Groups, one for each of the four Atlantic Provinces.

⁴ Management regions vary between the departments. This report refers to the three regions defined for the CSSP that include the following provinces: 1) Pacific (British Columbia); 2) Quebec; 3) Atlantic (all provinces east of Quebec).

⁵ British Columbia, Quebec, Nova Scotia, New Brunswick, Prince Edward Island, and Newfoundland & Labrador.

⁶ Approved areas is understood to mean open areas or closed areas where there are licensed relay or depuration operations.

⁷ Consumer Awareness and Perceptions of Shellfish Consumption and Recreational Harvesting (December, 2006)

⁸ A detailed cost analysis was completed as part of the evaluation and can be viewed in Appendix C.

⁹ Note that this landed value does not include recreational and Aboriginal food, social and ceremonial harvested shellfish.

¹⁰ A five year summary on Environment Canada's expenditures was provided and can be viewed in Appendix C

¹¹ A full list of activities/services that are bound by federal regulatory requirements can be found in the Alternatives Analysis Workshop report.

Next page: Conclusions