The Quality and Reporting of Surveys

Main Points

What we examined

We examined the presentation of survey results in all 2003–04 departmental performance reports. We also examined two of the most important indicators of the quality of surveys conducted under contract by private research firms for federal departments and agencies in the same period. We focussed on whether the federal government provides leadership for survey quality in a way that enables departments and agencies to produce commissioned surveys of sufficient quality for their intended use. We also looked at whether government-wide leadership contributes to consistently high-quality surveys across government. We examined the specific roles played by the Treasury Board Secretariat and by the Public Opinion Research Directorate in Public Works and Government Services Canada. We looked at Statistics Canada surveys only for comparative purposes and not as part of the audit.

Why it's important

The government can use information gathered through surveys for a variety of purposes, such as understanding the views of Canadians on government priorities and policies, improving the management of departments and agencies, and monitoring their performance.

Information generated by surveys must be of good quality if it is to be credible and useful to parliamentarians, government managers, and Canadians, especially when it concerns the performance of government programs. Deputy ministers and agency heads need to be confident about the quality of any survey data included in their departmental performance reports. This is because they sign formal statements that the reports have been prepared according to certain principles designed to assure readers that, among other things, the information in the reports is accurate and any weaknesses and limitations of the data are disclosed properly. Poor-quality survey results presented in performance reports could give a misleading picture of how well programs are performing. Furthermore, there is a risk that inaccurate data from surveys could be used in government decision making.

While the cost of public opinion surveys commissioned by the federal government may be relatively small (between $11 million and $15 million per year), the cost of programs addressed by these surveys is in the billions of dollars. The growth in the number of surveys in recent years underscores the importance of assuring their quality. It is important for Canadians, especially those participating in federal government surveys, to be confident that the results will be of sufficient quality for their intended use.

What we found

• There is insufficient reporting on the quality and limitations of survey results in the 2003–04 departmental performance reports. Without this information, readers lack the means to judge the reliability of the data.
• Two important indicators of the quality of public opinion surveys—that is, population coverage and response rates—raise issues of quality in the surveys commissioned by departments and agencies in 2003–04. These issues signal potential problems that are of concern to us.
• Individual departments and agencies are responsible for the quality of surveys conducted for them. However, Treasury Board policies assign a range of responsibilities for the quality of federally contracted surveys to the Public Opinion Research Directorate. The Directorate is not adequately fulfilling key aspects of those responsibilities that would contribute to survey quality.

The Department and the Treasury Board Secretariat have responded. Public Works and Government Services Canada and the Treasury Board Secretariat are in general agreement with our recommendations. Their respective responses are included throughout the chapter.

Introduction

The importance of surveys for program management and reporting

2.1 Over the past few decades, the federal government has increasingly asked Canadians to participate in many different surveys. Surveys conducted by the federal government, such as the Census, gather important information about Canadians that is necessary for governments to manage its programs well. The private sector also conducts market research, media, and political polling. The recent growth in the number of surveys may have increased the burden on respondents and contributed to declining response rates. Such trends raise questions about the quality of these surveys.

2.2 The federal government can use information gathered through surveys for a variety of purposes, such as understanding the views of Canadians on government priorities and policies, and improving the management of departments and agencies and monitoring their performance. If the quality of surveys is unknown or questionable—for example, leading to results that may not accurately reflect the views of Canadians—then the government risks managing with misleading information.

2.3 The annual departmental performance reports are the key mechanism through which departments and agencies report information on their performance to Parliament. The reports frequently present information obtained from various surveys, including Statistics Canada surveys, contracted surveys, and departmental in-house surveys. Some of these are intended to determine how well a department or agency is performing. If parliamentarians are to rely on information in the departmental performance reports, it is important that the survey data be of sufficient quality for the purpose of reporting. Departments risk misleading Parliament about the performance of their programs if survey results reported in their performance reports are not accurate.

Standards for survey quality exist

2.4 Domestic and international standards have been developed for conducting surveys and reporting the results to clients. Domestically, the quality standards followed by Statistics Canada are generally recognized as being high. Statistics Canada applies these standards to the surveys funded from its own budget as well as to the cost recovery services it provides to other federal departments. While surveys done by other federal departments and agencies are not subject to Statistics Canada's standards, the quality of the Agency's survey work can serve as a good example for others. The market research industry has also developed standards for survey research.

2.5 Beyond Canada, the International Organization for Standardization recently drafted guidelines for the quality of surveys and for the reporting of surveys. The Office of Management and Budget, in the U.S., also recently asked a committee of survey experts to recommend standards on various issues addressing survey quality that would be eventually applied to all U.S. government departments that collect information from the public. These and other standards can provide a basis for analyzing and determining the overall quality of individual surveys in light of their intended use.

Public opinion surveys commissioned by the federal government

2.6 Public opinion surveys and other forms of public opinion research contracted to the private sector by the federal government increased by 300 percent over a nine-year period to nearly 600 projects in 2003–04. The cost of contracted public opinion research projects in 2003–04, including both quantitative and qualitative research (for example, focus groups), was $25.4 million; between $11 million and $15 million of this amount was spent on 388 quantitative research projects (that is, surveys). Although these expenditures may not be a large part of their total budgets, departments and agencies could be using survey information, along with other types of information, to make important decisions about their programs. While we were unable to estimate the total cost of the programs addressed by these surveys, the information we found suggests that the contracted surveys reported in the 2003–04 departmental performance reports related to programs with budgets ranging from $400,000 to $1.9 billion. We estimated that the number of Canadians contacted for federally commissioned public opinion surveys has potentially reached over one million Canadians annually.

2.7 Public opinion surveys commissioned to private research firms are subject to review by the Public Opinion Research Directorate (PORD), which is now located in Public Works and Government Services Canada (PWGSC). The Directorate currently has a staff of 12 and an annual budget of about one million dollars. In its Public Opinion Research in the Government of Canada 2003–04 Annual Report, the Directorate states that it acts as a "public opinion research centre of expertise" that contributes to ensuring "the highest possible quality of research" for the federal government. It further states that its activities include giving advice, providing methodology guides, and sharing information with the government research community, along with other co-ordinating functions.

2.8 According to the Treasury Board Contracting Policy, public opinion research in the federal government is "the planned gathering... of opinions, attitudes, perceptions, judgments, feelings, ideas, reactions, or views." The information is collected using quantitative or qualitative methods, from persons, businesses, institutions, or other entities.

2.9 Individual departments and agencies are responsible for the quality of public opinion surveys that they commission and pay for. If they demand high-quality surveys in their contracts with firms and do not receive the expected level of quality, then the departments can withhold payment. At the same time, Treasury Board policies also require that the Public Opinion Research Directorate review and provide advice on the methodologies of the proposed surveys before they go to the PWGSC contracting unit that produces the contract on behalf of the department commissioning the survey. The Directorate is also required to provide survey expertise to this contracting unit during the process of determining which private research firms have the capability to conduct surveys. From among the qualifying research firms, departments may then select firms for individual survey projects. After the research is completed, departments are required to send copies of the final research reports to PORD to be deposited with both the Library of Parliament and Library and Archives Canada.

Focus of the audit

2.10 The objectives of this audit were

• to determine whether departments and agencies provide sufficient information in their departmental performance reports about the quality of the data produced by surveys to demonstrate that they are fit for use in reporting performance to Parliament,
• to describe some key elements of the quality of surveys that are conducted for federal departments and agencies by private research suppliers and processed through the Public Opinion Research Directorate, and
• to assess whether the federal government is exercising leadership in ensuring the quality of surveys conducted by departments and agencies.

2.11 We reviewed all 90 departmental performance reports for 2003–04 and found that 64 contained information from a variety of surveys. In these 64 reports, there were 209 references to surveys. We also examined 49 of the 388 quantitative surveys listed in a PORD database of public opinion surveys contracted out to private research firms in 2003–04. The 49 surveys consisted of 16 high-value surveys and a random sample of 33 of the remainder. We reviewed the policy framework governing the planning, contracting, and reporting of surveys and how this framework worked in practice. We also interviewed staff from 10 of the 17 departments and agencies that commissioned the surveys drawn from the PORD database.

2.12 During 2003–04, the Public Opinion Research Directorate was part of Communication Canada, which was disbanded on 31 March 2004. The Directorate then became part of PWGSC.

2.13 For more information on our audit scope and approach, including our sampling method and quality issues, see About the Audit at the end of the chapter.

Observations and Recommendations

Reporting the quality of surveys

Parliamentarians cannot determine the quality of surveys

2.14 The Treasury Board Secretariat Preparation Guide—Departmental Performance Reports (2003–04) aims "to provide parliamentarians and Canadians with high-quality information about the plans and achievements of the Government of Canada." To do this, departmental performance reports (DPRs) should provide "clear, complete, concise, and credible information on departmental financial and non-financial performance." Departments should explain "why the public can have confidence in the methodology and data used to substantiate performance." Reports should "give readers the means to make informed decisions about the reliability of the performance information [provided]." While these broad principles apply to all information in DPRs, they do not provide sufficient guidance on disclosing the quality of survey data in the reports.

2.15 However, specific requirements have been developed elsewhere for the disclosure of information about surveys when survey results are included in accountability documents, such as departmental performance reports. The American Association for Public Opinion Research (AAPOR), which represents the survey research industry, has standards for the minimum disclosure of information about surveys, including their quality. The Office of Management and Budget, in the U.S., has identified standards for "short-form" reporting, such as departmental performance reports. These standards are intended to ensure that consumers of survey data have sufficient information about the survey to judge the quality of the resulting data.

2.16 Given the broad reporting principles of the Treasury Board Secretariat and the availability of other public and private-sector reporting standards for the disclosure of information about survey quality, we expected that basic information about surveys would accompany the presentation of survey data in DPRs. We identified key elements of data quality common to these standards that could reasonably be expected to be found in a departmental performance report. To determine the adequacy of the reporting of the basic quality of surveys to Parliament, we looked at all 90 departmental performance reports submitted to Parliament in 2003–04. In 64 of these reports, there were 209 references to surveys, including those co-ordinated by the Public Opinion Research Directorate, and from Statistics Canada and a variety of other sources. Exhibit 2.1 lists these key elements, explains why they are important, and presents what we found in the 2003–04 DPRs.

2.17 Overall, we found that none of the 209 references contained all of the information listed in Exhibit 2.1. Key indicators of survey quality were often missing, notably response rates, descriptions of sampling frames, and confidence intervals. Because there was so little information about survey methods or about the quality of the resulting data, readers of departmental performance reports lack a sufficient basis for judging whether the data are accurate enough for the intended use of the survey. In this context, the main use of survey results is for departments' accountability to Parliament. Ideally, this information could be disclosed in a footnote or endnote, as follows:

Data reported are from the [survey name] conducted by [firm name] for [department name]. A non-proportional stratified random sample of the target population [population size] was selected using random-digit dialing. The response rate for the telephone survey was [percent], with a final sample size of [size] and a confidence interval [for example, CI=4.0 percent @ 95 percent]. All estimates have been weighted to reflect proper geographic distribution. The sample excluded people under the age of [age]. Data were collected from [date]. The final report is available at the following Web site [address].

2.18 The Preparation Guide—Departmental Performance Reports (2003-04) requires that deputy ministers and agency heads sign a Management Representation Statement in their performance reports. This formal statement states that the reports have been prepared according to certain principles, designed to assure readers that, among other things, the information in the reports is accurate and that any weaknesses and limitations are disclosed properly. Therefore, deputy ministers and agency heads must be confident that the survey results are of sufficient quality to be reported in the performance reports and that the risk that the survey information will mislead Parliament is minimal. However, the Preparation Guide does not currently provide managers with guidance on how to support deputy ministers and agency heads in taking this step.

2.19 Recommendation. The Treasury Board Secretariat's annual Preparation Guide—Departmental Performance Reports should ask departments and agencies to ensure that

• references to survey data are accompanied by a basic description of how the survey was conducted, along with key indicators of data quality and any data limitations;
• this information is readily accessible through footnotes or endnotes; and
• more detailed methodological information is publicly accessible, preferably through an Internet link to the final survey report.

The Treasury Board Secretariat's response. Through its guidelines on the preparation of departmental performance reports, the Treasury Board Secretariat encourages departments to explain in their reports why the reader can have confidence in the methodology and data used to substantiate performance. Among other things, good-quality public performance reports should be balanced, easy to understand, and unburdened by large amounts of technical detail. The Secretariat recognizes that reasonable documentation of survey methodology, presented in an unobtrusive manner, can enhance the quality of a public performance report. The Secretariat will address the reporting of survey methodology in upcoming versions of its guidelines and will continue to work with departments to improve the quality of reporting to Parliament.

Quality of public opinion surveys

The quality of public opinion surveys is a concern

2.20 In addition to how well the quality of surveys is reported in departmental performance reports, we are also concerned about the quality of the surveys themselves. The quality of surveys is important, not only for reporting in DPRs, but for other uses by the departments. We focussed on those public opinion surveys commissioned by departments and agencies and reviewed by the Public Opinion Research Directorate (PORD). (All references to surveys from this point are to public opinion surveys commissioned by departments and agencies and reviewed by the Public Opinion Research Directorate.)

2.21 While the federal government has identified industry standards to be applied to public opinion surveys co-ordinated by PORD, such as the need to pre-test questions, the government has not established benchmarks that would provide specific cut-off measures for minimum quality (for example, surveys with response rates below a certain percentage would be considered of poor quality). By contrast, the U.S. Office of Management and Budget asked a committee of survey experts to develop survey standards for all federal departments collecting information from the public. These proposed standards and guidelines for statistical surveys provide some benchmarks for describing the quality of surveys.

2.22 The Directorate receives final reports on the surveys from the departments commissioning these surveys. We examined 49 survey files from 2003–04 that were drawn from a PORD database. These included 16 high-value and 33 randomly sampled files, of which 45 contained final survey reports. We looked at two of the more important indicators of survey quality—population coverage and response rates—that have become of increasing concern to government, academic, and private sectors. These indicators can signal the risk of potential bias in survey results.

2.23 Population coverage. When a sampling frame excludes a segment of the target population, the result is population under-coverage. If the excluded segment of the population is relatively large and has different views from those who were contacted, the results of the survey could be biased. When the list includes people who are not part of the target population, the result is population over-coverage. For example, a survey that draws conclusions about attitudes of youth has to ensure that all youth are properly included and that adults are excluded. As well, the method by which survey information is collected may have an impact on the population covered. For example, telephone surveys using land line-based random digit dialing would exclude households that do not have a land line-based telephone.

2.24 The federal government has not established benchmarks for population coverage. By contrast, guidelines proposed by the U.S. Office of Management and Budget recommend that sampling frames cover at least 95 percent of the target population and that, when the coverage falls below a benchmark of 85 percent, an evaluation of the potential for bias be done. We expected that, at a minimum, all final survey reports would include a discussion about population coverage and resolve any questions about the potential for bias.

2.25 While the final reports for 9 of the 13 high-value surveys and 17 of the 32 randomly sampled surveys we examined described the method used to contact respondents, few (3 of the high-value surveys and 3 of the randomly sampled surveys) contained a discussion of population coverage—that is, whether parts of the population were systematically excluded or over-represented. Overall, population coverage was rarely discussed in the final reports of the surveys we examined.

2.26 Response rate. Response rates are another key indicator of whether survey results accurately reflect the views of the target population. Although a low response rate does not necessarily indicate a bias in the results, an extremely low response rate should always be a concern, especially where a sound analysis of potential differences between respondents and non-respondents is not possible or has not been done. A 2002 study of response rates conducted by the Professional Marketing Research Society found that response rates have been steadily declining, from an average of 19 percent in 1997 to an average of 13 percent in 2002 for omnibus telephone surveys. A similar trend was found for one-time surveys.

2.27 While low response rates do not mean that survey data are necessarily unrepresentative or biased, the risk of potential bias occurring rises as response rates fall. According to our advisory panel of national and international experts on surveys, a clear statement about the representativeness of sampled respondents should always accompany a report presenting survey findings. The U.S. Office of Management and Budget advisors recommend that an analysis be done when the response rates suggest the potential for bias and that additional analyses determining sample representation be mandatory when such rates fall below their proposed benchmark of 80 percent.

2.28 We expected that all final survey reports would provide the response rate and include an analysis and discussion of the representativeness of the sample. We found that 10 of the 13 high-value surveys and only 14 of the 32 randomly sampled surveys either reported a response rate or provided information to calculate a response rate in their final survey reports.

2.29 Because the risk of potential bias rises as response rates fall, we also examined whether the survey reports contained an analysis of whether respondents were similar to non-respondents at three risk levels: 80, 50, and 20 percent response rates. We found that, of the 24 final survey reports providing information on response rates, only 3 offered the expected analysis of potential bias. All 3 surveys were between the 20 and 50 percent risk level, with response rates of 45, 42, and 31 percent. None of the remaining 21 survey reports included the expected analysis, and 9 of these surveys were at the highest risk level with response rates under 20 percent.

2.30 We then examined the range and average for the response rates. We found that the response rates for the high-value surveys varied from 12 percent to 52 percent, with an average response rate of 29 percent. The response rates for the randomly sampled surveys varied from 8 percent to 66 percent, with an average response rate of 32 percent. While the number of final survey reports with response rates was insufficient for us to generalize our findings to all of the federally commissioned 2003–04 surveys, we are concerned about what may be a more widespread absence of an analysis of the potential for bias related to response rates.

2.31 If a small proportion of people agree to participate in a survey, report readers should question whether their responses represent those who were also contacted but did not participate. To be considered unbiased, a response rate of 10 percent means that the responses of 10 out of every 100 people should represent the range of responses of the 90 others contacted who did not participate in the survey. Even with an analysis of the potential for bias, the quality of surveys with such low response rates is—by any benchmark—in doubt and should be of concern to the government.

2.32 In our view, while the required quality of surveys is determined by their intended use, there must still be a minimum level of quality for any use. The lack of analysis to determine whether respondents represent the target population is also a concern. Both of these issues raise questions about the quality of the surveys.

2.33 Although the federal government has identified industry standards as those to be applied to surveys, these standards do not provide benchmarks to help departments determine the quality of surveys, including for the two indicators we examined. Therefore, we are concerned that departments may not be able to determine whether the surveys they commission are of sufficient quality for their intended use. At present, only Statistics Canada has standards for all major quality elements that it applies to all of its own surveys on a case-by-case basis.

2.34 For two of the key indicators—population coverage and response rates, we could not form a conclusion about the quality of the 2003–04 surveys we examined because the federal government has set no benchmarks. Nonetheless, our findings suggest a lack of consistent quality in commissioned surveys across the government. We are particularly concerned about coverage and response rates and the potential for biased results when issues related to key indicators of survey quality have not been addressed. We are also concerned that 21 of 45 final survey reports did not provide any information on response rates. In our view, the government should also be concerned about these issues.

Leadership for the quality of public opinion surveys

Government-wide leadership for the quality of public opinion surveys is limited

2.35 Departments are responsible for the quality of individual surveys. Deputy ministers and agency heads are responsible for surveys commissioned by their departments and agencies. According to the Communications Policy of the Government of Canada (2002), departments "must ensure the quality and value of the research they commission or produce." However, if the government has an interest in commissioned surveys being of consistent quality across the government, there is an opportunity to contribute to survey quality before and after the contracting stage. This function should be carried out by a department with government-wide leadership for providing departments with expert advice and then reviewing final survey reports.

2.36 A department will require two kinds of expert survey advice: requirements for making the survey consistent with government-wide standards and with any benchmarks for quality, and particular quality requirements for each survey. These requirements could be reflected in the department's contract with the private sector research firm. Once a department has committed to the contract, it is responsible for ensuring that the quality requirements of the contract are satisfied in the final survey report. However, if the contract does not commit the firm to producing a good-quality survey for the department's intended use, then the department cannot be guaranteed that this quality will be produced.

Government-wide leadership for the quality of public opinion surveys has been important

2.37 Historically, the federal government has recognized the importance of having a lead department or agency contribute to consistent quality in all government surveys. In 1974, a Treasury Board directive approved the Guidelines for Requests for Information from more than Ten Respondents. While this directive was intended to reduce response burden by avoiding duplication of surveys, it also emphasized the importance of survey quality through designing and carrying out surveys that conform to statistical standards. Statistics Canada was responsible for systematically reviewing and advising all departments and agencies about their proposed surveys, and their planned methodology. It would then submit its report to the Treasury Board, which would decide whether the project would be funded.

2.38 Under this directive, Statistics Canada found few instances of duplication and received surveys too late to have an impact on their quality. So, during a period of budgetary restraint in the 1980s, Statistics Canada's role in giving advice on contracted public opinion surveys was significantly reduced. However, the Agency currently offers a variety of survey services, ranging from advice on survey design to the collection of data and statistical analysis. This work is done on a cost recovery basis, typically for other federal departments and agencies. Statistics Canada told us that it applies its quality standards to all its work, including cost recovery work.

2.39 Other countries have maintained government-wide leadership on survey quality. For example, since the 1940s, the U.S. Office of Management and Budget (OMB) has been responsible for reducing the burden on citizens and businesses providing information to the government. It reviews survey proposals, methodology plans, response burdens, and the various techniques used. All surveys of more than 10 individuals conducted by the federal government, internally or externally, have to be reviewed and comply with OMB standards before departments are permitted to carry out their surveys.

2.40 Over the past couple of decades, the leadership for federally commissioned surveys in Canada has moved between various departments. We looked at Treasury Board policies that evolved to give the Public Opinion Research Directorate a government-wide leadership role in contributing to the quality of commissioned surveys and how the Directorate interpreted and implemented these policies.

The Treasury Board has defined a government-wide leadership role for the quality of public opinion surveys

2.41 We found that the Treasury Board's Common Services Policy (2002) identifies common service organizations (CSO)s that, among other things, will contribute to more efficient government and "seek benefits from the pooling of specialized expertise." The policy goes on to say, "Certain services provided by CSOs are designated as mandatory....when a government-wide interest or consideration prevails over, or coincides with, the interests of individual departments and agencies." The interest or consideration can include providing access to "centres of expertise and specialization" and responding to the need for "a high level of consistency." For particular services, all applicable departments must use the services of the mandatory CSO. The Public Opinion Research Directorate, currently in PWGSC, is such a mandatory service.

2.42 According to Treasury Board Secretariat officials, the Communications Policy of the Government of Canada and its procedures, together with the Common Services Policy provide the rationale for why the Directorate is a mandatory common service organization. The Communications Policy defines PORD as the "technical and co-ordinating authority for Government of Canada public opinion research." Further, the related Communications Policy procedures state that PORD "will assist institutions by reviewing their research methodologies and clarifying the research objectives; advising on the preparation of statements of work...." and "advise institutions on....generally accepted standards of the market research industry." All departments planning to commission surveys from private research firms must first provide a project description for the proposed survey to the Directorate before contacting the firm. Analysis of the survey proposal provided by the commissioning department gives PORD staff the opportunity to offer required advice on the methodology and related quality requirements of the survey.

2.43 The Common Services Policy and the Communications Policy make clear that the final responsibility for the quality of the survey lies with the commissioning department and that PORD's role is one of providing advice. However, in our view, the policies also make clear that PORD is required to provide advice on issues relating to quality. It thereby acquires a government-wide leadership role through meeting its responsibility to offer persuasive and consistent high-quality advice to the commissioning departments.

The Public Opinion Research Directorate has interpreted its role in a limited way

2.44 We expected that PORD would provide and document advice to departments at key stages of the planning, contracting, and reporting process, which would contribute to consistently high-quality commissioned surveys across the government.

2.45 We found that the Directorate's annual report for the relevant period largely confirmed its government-wide leadership role, as defined in the Treasury Board policies. The Public Opinion Research in the Government of Canada Annual Report 2003–04 states that PORD's mandate is "to provide a public opinion research centre of expertise" and that its research expertise helps government organizations "develop relevant and reliable research to make informed decisions." This report also says that PORD's assistance to departments and agencies helps to "enhance the quality of the research they undertake" and is "uniquely tailored to each assignment."

2.46 By contrast, PORD officials told us that they view the policy framework as defining a more limited role than defined by the policy framework for a department with the government-wide lead for survey quality. First, they informed us that, because individual departments are responsible for the quality of their surveys, the Directorate "has no accountability for ensuring survey quality, and consequently does not formally assess quality." Second, they told us that PORD staff provide advice "as required by the accountable institution" and that their "advice is provided on a case-by-case basis where there are concerns." In our view, PORD is required to review and provide advice on all submissions that it receives, if only to indicate that, based on its analysis, no substantive advice is needed. We found little evidence that PORD systematically reviewed methodologies of departments' proposals. Without a systematic review of methodology, it would be difficult for PORD to know whether advice was warranted.

2.47 PORD staff also informed us that they follow the Treasury Board Communications Policy requirement to play a co-ordinating function by linking departments to contractors, providing support services to departments such as facilitating interdepartmental committees, and providing workshops and research guides.

2.48 However, PORD did not place sufficient emphasis on providing expert advice to departments about the quality of individual surveys to justify the Directorate's role as a mandatory common service organization, as understood from the relevant policies. Therefore, we concluded that the Directorate does not appropriately interpret the applicable government policies regarding its mandatory role, as a common service organization, to advise departments on survey quality.

2.49 Recommendation. Public Works and Government Services Canada should ensure that the Public Opinion Research Directorate, as a mandatory common service organization, place sufficient emphasis on providing expert advice to departments and agencies on the quality of all surveys.

Department's response. While departments are accountable under the policy framework for ensuring survey quality, the Public Opinion Research Directorate (PORD) recognizes the importance of this recommendation and will continue to provide expert advice to departments on survey quality. In May 2004, new contracting tools for national public opinion research were implemented, which included more stringent contractual requirements for the industry to improve research quality. Further quality elements will be added to contracting tools for public opinion research, to be renewed by 2007 and communicated to departments. PORD will also update its existing research guides to better inform departments of survey quality matters. As a further measure, the Directorate will implement a method for systematizing and documenting expert advice given to departments and will continue its practice of double-reviewing project files prior to sending projects for contracting.

The government-wide leadership role for the quality of public opinion surveys is not adequately fulfilled

2.50 We expected that a department with a government-wide leadership role for contributing to consistently high-quality surveys commissioned by departments would have clear responsibilities for

• setting government-wide benchmarks and adapting appropriate standards for the quality of federally commissioned surveys and for the content of final survey reports;
• providing documented expert advice to departments when reviewing the survey methodology and objectives. This advice would be based on applying government-wide standards and benchmarks on a case-by-case basis to each commissioned survey, with the objective of this advice being reflected in the individual contracts between departments and firms; and
• reviewing the final survey reports to determine whether they meet the specific quality requirements included in the advice given earlier, before the final survey report is sent to the Library of Parliament and Library and Archives Canada.

2.51 Appropriate benchmarks for survey quality do not exist. The Treasury Board Communications Policy procedures (2002) outline the responsibilities of the Public Opinion Research Directorate for advising departments on "generally accepted standards of the market research industry." However, as discussed earlier, these standards do not provide specific benchmarks, at least for the two key indicators of survey quality we examined. We expected that the Directorate would have established appropriate benchmarks for government to use as the basis for its advice to departments on how to distinguish between good-quality and poor-quality surveys. Advice of this kind would also contribute to the achievement of consistently high-quality surveys across the government.

2.52 PORD officials told us that they did not have the authority to develop standards for survey quality and are limited to using market research standards. However, the Communications Policy procedures indicate that one of the government's objectives is to ensure that survey research "meets or exceeds recognized standards of the market research industry" and, as noted, assigns responsibilities to the Directorate for advising departments on these industry standards. In our view, this statement requires PORD to identify the relevant market research standards for application to surveys commissioned by departments. It would also permit PORD to adapt other standards to fill gaps in the market research standards, or to develop its own benchmarks, where desirable, to fit the needs of government survey work.

2.53 Recommendation. To assist departments in distinguishing between good-quality and poor-quality surveys, the Public Opinion Research Directorate should adapt the relevant standards of the market research industry and develop suitable benchmarks applicable to surveys commissioned by departments.

Department's response. The Public Opinion Research Directorate (PORD) recognizes the importance of this recommendation and will establish an expert technical advisory panel, which will include participation from Statistics Canada. The panel will assist in the development of suitable benchmarks for inclusion in the contracting tools that will be developed and made available to departments for the procurement of public opinion research. PORD will also continue to apply and adapt the relevant standards of the marketing research industry when reviewing projects and providing advice.

The Public Opinion Research Directorate's advice on survey quality is informal

2.54 In our view, advice given to departments that would contribute to the consistent quality of their commissioned surveys should include three key areas.

2.55 First, the advice should clarify the survey's research objectives and include a review of the research methodology, according to the Communications Policy procedures.

2.56 Second, the advice should be made on the basis of industry standards and benchmarks that are appropriate for government use. We believe that this advice is most effective when it is provided for each survey on a case-by-case basis and is reflected in the contract between the department and the survey research firm. Therefore, we expected that the Directorate would give advice to departments, based on a suitable application of appropriate standards with accompanying benchmarks for survey quality, or, at a minimum, that the Directorate would advise departments about these industry standards, in accordance with requirements of the Communications Policy procedures.

2.57 Third, this advice should be documented, so that the department with the leadership role to provide the advice can demonstrate how effective it has been in reviewing and providing advice on proposed surveys, thereby contributing to consistently high-quality survey research across the government. The Treasury Board Policy on the Management of Government Information states that "federal government institutions should create, use, and preserve information to fulfill their mandates, support program and service delivery, achieve strategic priorities, and meet accountability obligations prescribed by law." Therefore, we expected that any advice given by PORD to departments on their proposed surveys would be documented.

2.58 Most of PORD's advice is not documented. PORD officials said that they spend 40 percent of their time providing advice to departments at various stages during the planning, contracting, and reporting process. We reviewed the 49 survey files for written evidence of advice on methodology and issues that could have an impact on survey quality and found such advice in less than half of the files. Directorate staff told us that most of their advice and discussion of standards was given informally, mainly by telephone, and was not documented. Of the staff we interviewed from the Directorate's client departments and agencies, most told us that PORD did not give them advice on designing, conducting, and reporting surveys.

2.59 Little evidence that advice on research methodology or standards was given systematically. The Communications Policy procedures require the Public Opinion Research Directorate to review research methodologies and advise on generally accepted standards. We expected to find a checklist or other documentation indicating that research methodologies were systematically reviewed and that advice from PORD staff referred to standards or benchmarks. We did not find such a checklist or documentation. Only 5 of the 49 files had any documented reference to the market research standards used to judge a department's statement of work or a research firm's proposal. Thus, PORD's advice on research methodology was neither systematic nor adequately documented.

2.60 Quick review of survey proposals is a concern. PORD staff emphasized that they provide advice in a timely way because the Common Services Policy states that "common service organizations will offer services to client departments in a manner that is most supportive of timely, effective, and economical delivery of programs to the public." We found that, in two thirds of the randomly sampled surveys and over one third of the high-value surveys, the Directorate processed the request in less than one day. Although the Directorate may have given informal advice before the official process had begun, the short period available for a formal review of the survey proposals allowed only a small opportunity to provide sound advice to departments. Statistics Canada officials performed a similar role in the 1970s and 1980s, and they told us that a week was not sufficient to provide comprehensive advice on the quality of a proposed survey. In our view, as a mandatory common service organization, PORD should carefully consider the benefits and risks when trading-off timeliness against providing sound advice.

2.61 PORD's other roles do not contribute sufficiently to survey quality. PORD staff told us that they contribute to survey quality when they help the contracting unit of PWGSC determine if a research firm can produce quality surveys. However, this only establishes that a firm is capable of meeting or exceeding general standards of survey quality in a future survey project. PORD staff also told us that a firm's formal offer to provide its services to a department at some later date includes a requirement to meet or exceed market research standards. However, such a promise could not realistically lay out the future conditions of quality for any specific survey conducted for a department; nor could it anticipate the particular quality requirements of a future survey, based on its intended use. By contrast, Statistics Canada ensures that its surveys are of high quality by applying its standards to all of its own surveys on a case-by-case basis, depending on the intended use. In our view, while PORD's other roles may contribute somewhat to the overall quality of commissioned surveys, they are insufficient to ensure that the quality requirements for specific surveys are met consistently across the government.

2.62 Recommendation. When the Public Opinion Research Directorate advises departments and agencies about surveys, Public Works and Government Services Canada should ensure that

• the advice is based on the systematic application of relevant standards and benchmarks to each survey on a case-by-case basis;
• all advice is documented; and
• departments and agencies are encouraged to reflect this advice in the contract so that concrete expectations of quality will be established.

Department's response. The Public Opinion Research Directorate (PORD) accepts this recommendation. The Directorate will develop a framework and working tools to aid advisory staff in systematically applying standards and benchmarks to the formulation of advice on a case-by-case basis. New practices will be put in place to better document advice given to departments. PORD will continue to provide advice to departments to be incorporated in the contract so that concrete expectations of quality will be established. In addition, further quality elements will be added to our next contracting tools for public opinion research, to be renewed by 2007 to assist departments in obtaining quality elements in the contracting process.

Complete information on public opinion surveys is not available in final reports

2.63 Contracts for surveys typically require some form of a final survey report to be provided by the research firm to the department that commissioned the survey. A survey report normally describes the methodology and the results of the survey. This information is important for readers to properly interpret the survey results. The Communications Policy procedures only require that these final reports be sent to PORD and that the Directorate then deposit them with the Library of Parliament and Library and Archives Canada. The procedures do not require PORD to review the reports.

2.64 However, we note that the Policy on the Management of Government Information requires that all departments "collect, create, receive, and capture information in ways that ensure its relevance, reliability, and completeness." In our view, this policy requirement would permit PORD to review the final reports and provide it with an opportunity to add value to survey and archival quality by assessing the reports to ensure that critical information is supplied. Such a procedure would allow the Directorate to close the loop by assessing the extent to which the advice its officials initially provided to a department is reflected in the completeness of the final research report and in the quality of the survey. PORD's assessments could be used to refine and improve its advisory services for subsequent research activities.

2.65 In the 49 survey files we examined (16 high-value and 33 randomly sampled reports), we found that most of the contracts drawn up by PWGSC's contracting unit contained a common set of elements that required the final reports to describe the quality of the surveys. PORD officials told us that, because departments could change these elements, not all contracts would have this common set of elements. Of the 45 survey files that contained final survey reports, 10 of the 13 high-value reports and 30 of the 32 randomly sampled reports were required to include this common set of elements describing survey quality. However, we found that none of these 40 final survey reports included all the common elements required by the contract.

2.66 We also examined whether the 45 final reports contained five key quality elements we selected from this common set. We found that only 9 of the reports provided all five of the key quality elements identified in Exhibit 2.2.

2.67 Overall, we found that while PORD received the final reports of commissioned surveys, Directorate staff did not review them for completeness and reliability before sending them to the Library of Parliament and Library and Archives Canada. This is consistent with the current Treasury Board Secretariat policy framework for PORD.

2.68 Recommendation. Public Works and Government Services Canada should ensure that the Public Opinion Research Directorate review the final survey reports for completeness and reliability.

Department's response. The Public Opinion Research Directorate (PORD) recognizes the importance of final survey reports being complete and reliable. Prior to submitting final reports to the Library of Parliament and Library and Archives Canada, the Directorate will continue to review final reports for completeness and advise departments of required elements under Government of Canada policies. In keeping with institutional accountability for the quality of research under the Communications Policy of the Government of Canada and the management of expenditures under the Financial Administration Act, PORD will continue to advise and encourage departments to include necessary elements for describing quality and take recommended steps in any instance where standards and benchmarks have not been met. Further quality elements to promote completeness and reliability will be added to the next contracting tools for public opinion research, to be renewed by 2007.

Conclusion

2.69 The growth in the number of federal government surveys underscores the importance of assuring their quality and how they are reported publicly. Canadians should be confident that the information they provide in their responses to surveys will be used by the government to manage programs better and to report on program performance to Parliament.

2.70 Clear guidance has yet to be provided to departments and agencies on reporting the quality of surveys in their departmental performance reports. Consequently, the public and parliamentarians were not adequately informed about the quality of the surveys reported to them. There is a risk that they were misled by poor-quality survey results about government programs.

2.71 Based on our examination of two of the more important indicators of survey quality, we are concerned about the quality of those public opinion surveys commissioned by departments and agencies that we looked at. We are particularly concerned that any potential biases in surveys were not adequately addressed and that survey results could have been misleading. We are also concerned that most of the final survey reports were incomplete.

2.72 Because the organization with government-wide leadership for survey quality—the Public Opinion Research Directorate—did not adapt industry standards for government use or develop benchmarks, it missed an opportunity to provide advice to departments and agencies that would contribute to the consistent high quality of their commissioned surveys. While each department and agency is responsible for the quality of its own surveys, the department with the mandate for a government-wide leadership role contributing to consistently high-quality surveys did not adequately fulfill that role.

About the Audit

Objectives

The objectives of this audit were

• to determine whether departments and agencies provide sufficient information in their departmental performance reports about the quality of the data produced by surveys to demonstrate that they are fit for use in reporting performance to Parliament,
• to describe some key elements of the quality of surveys that are conducted for federal departments and agencies by private research suppliers and processed through the Public Opinion Research Directorate (PORD), and
• to assess whether the federal government is exercising leadership in ensuring the quality of surveys conducted by departments and agencies.

Scope

Our audit examined the sufficiency of the reporting of surveys, from a variety of sources, in departmental performance reports. We also examined the quality and reporting of contracted surveys conducted by private sector firms for federal departments and agencies that were processed through the Public Opinion Research Directorate. Finally, we examined leadership on survey quality.

The scope of the part of our audit that focussed on public opinion surveys commissioned by the federal government excluded the following:

• Statistics Canada's A-based surveys—its cost recovery work in the Special Surveys Division and the Statistical Consultation Group was examined as a general benchmark to provide a basis for comparison with other federal surveys;
• information on how the surveys were used by departments and agencies;
• surveys conducted by agencies under Section III of the Financial Administration Act, as they are not subject to review by PORD;
• surveys conducted solely for internal use or conducted in-house by departments and agencies, and surveys contracted by Consulting and Audit Canada;
• qualitative public opinion research, due to the narrower scope and use of these data, and the distinct nature of applicable quality standards; and
• the management of files, the evaluation of firms for contracting purposes, and the co-ordination of contracts.

Approach

Selection of samples. Using a PORD database that recorded all public opinion research conducted during 2003–04 (593 cases), we identified all projects that involved some form of quantitative research (388). This included projects that have a combined quantitative and qualitative element as well as syndicated studies. One file had a negative budget amount and was excluded from the sampling process. As expected, the distribution was positively skewed, with a minority of surveys having very high values. Using Tukey's Outlier Filter, we defined high-value surveys as having a budget of $136,000 or more. We identified16 surveys as high-value.

The remaining surveys (372) were divided into two levels for non-proportional sampling—surveys of less than $50,000 (263) and greater than $50,000 (108). Estimates for this sample were appropriately weighted.

A total of 57 surveys were sampled. Eight were excluded from the audit because they fell outside the target population. Some of these files were incomplete because they involved the purchase of data from a syndicated survey (see following table).

Results were reported for the high-value items using a census of available surveys, and for the remainder of the population using the non-proportional stratified sample. The results are accurate within plus or minus 12 percent, 18 times out of 20. Extreme findings (less than 10 percent or greater than 90 percent) are accurate within plus or minus 10 percent, 18 times out of 20.

For the 19 cases where the final report included at least a partial record of contact, we estimated the response rates of surveys based on information within the "record of contact" using the AAPOR Response Rate 3 (RR3) method of calculation. This method is less conservative than the method prescribed by the Professional Marketing Research Society—the AAPOR's method will produce higher response rates than the Society's method. In most cases, there was insufficient information in the record of contact to calculate a precise RR3. We estimated an eligibility rate based on the target population and Statistics Canada 2001 Census information. In the 5 cases where a report listed only a response rate, with no information on how it was calculated, the response rate was taken at face value. The remaining 21 cases contained no information on response rate.

Interviews. We conducted interviews with staff of 10 of the 17 departments and agencies that were part of our sample of contracted survey projects. In most cases, those interviewed were public opinion research co-ordinators in 2003–04. We selected the departments to represent various sizes and various amounts of research experience. Each department was asked standardized questions.

Audit team

Assistant Auditor General: Ronnie Campbell
Principal: Barry Leighton
Directors: Catherine Livingstone, Colin Meredith

Doreen Deveen
John McGrath
Anupheap Ngoun
Paul Pilon
Ruth Sullivan
Jacqueline Wickett

For information, please contact Communications at (613) 995-3708 or 
1-888-761-5953 (toll-free).

Definitions:

Sampling frame—Any list, material, or device that identifies and allows access to the elements of the survey population. (Back)

Response rate—The proportion of eligible respondents selected to participate in a survey that actually participated. (Back)