Quality

Quality control
Quality management in statistical agencies

Quality is an essential element at all levels of processing. Statistics Canada's reputation as the best statistical agency in the world is based on the quality of its data. To ensure the quality of a product or service in our survey development activities, both quality assurance and quality control methods are employed.

Quality assurance

Quality assurance refers to all planned activities necessary in providing confidence that a product or service will satisfy its purpose and the users' needs. In the context of survey conducting activities, this can take place at any of the major stages of survey development: planning, design, implementation, processing, evaluation and dissemination.

Examples of planned activities include:

improving a survey frame
changing the sample design
modifying the data collection process
improving follow-up routines
changing the processing procedures
revising the design of the questionnaire

Quality assurance attempts to move quality upstream by anticipating problems before they occur and aims at ensuring quality via the use of prevention and control techniques.

Quality control

Quality control is a regulatory procedure through which we

measure quality;
compare quality with pre-set standards; and
act on the differences.

Some examples of this include controlling the quality of the coding operation, the quality of the survey interviewing, and the quality of the data capture.

The objective of quality control is to achieve a given quality level with minimum cost. Some assurance and control functions are often performed within the survey unit itself, especially in connection with the tasks of data coding, capture and editing. Several of these procedures are automated, some partially automated and others employ purely manual methods.

Outlined below are some of the key differences between quality assurance and quality control:

Quality assurance	Quality control
- anticipates problems before they occur	- responds to observed problems
- uses all available information to generate improvements	- uses ongoing measurements to make decisions on the processes or products
- is not tied to a specific quality standard	- requires a pre-specified quality standard for comparability
- is applicable mostly at the planning stage	- is applicable mostly at the processing stage
- is all-encompassing in its activities	- is a set procedure that is a subset of quality assurance

Quality management in statistical agencies

The quality of the data must be defined and assured in the context of being 'fit for use'. Whether or not data and statistical information are fit for use will depend on the intended function of the data and the fundamental characteristics of quality. It also depends on the users' expectations of what they consider to be useful information.

There is no standard definition among statistical agencies for the term official statistics. There is a generally accepted, but evolving, range of quality issues underlying the concept of 'fitness for use'. These elements of quality need to be considered and balanced in the design and implementation of an agency's statistical program.

So, how does Statistics Canada define quality? The following is a list of the elements of quality:

relevance: The relevance of statistical information reflects the degree to which it meets the real needs of clients. It is concerned with whether the available information sheds light on the issues that are important to users. Assessing relevance is subjective and depends upon the varying needs of users. The Agency's challenge is to weigh and balance the conflicting needs of current and potential users to produce a program that goes as far as possible in satisfying the most important needs within given resource constraints.
accuracy: The accuracy of statistical information is the degree to which the information correctly describes the phenomena it was designed to measure. It is usually characterized in terms of error in statistical estimates and is traditionally decomposed into bias (systematic error) and variance (random error) components. It may also be described in terms of the major sources of error that potentially cause inaccuracy (e.g., coverage, sampling, nonresponse, response).
timeliness: The timeliness of statistical information refers to the delay between the reference point (or the end of the reference period) to which the information pertains, and the date on which the information becomes available. It is typically involved in a trade-off against accuracy. The timeliness of information will influence its relevance.
accessibility: The accessibility of statistical information refers to the ease with which it can be obtained from the Agency. This includes the ease with which the existence of infromation can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. The cost of the information may also be an aspect of accessibility for some users.
interpretability: The interpretability of statistical information reflects the availability of the supplementary information and metadata necessary to interpret and utilize it appropriately. This information normally includes the underlying concepts, variables and classifications used, the methodology of data collection and processing, and indications or measures of the accuracy of the statistical information.
coherence: The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytic framework and over time. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys. Coherence does not neccessarily imply full numerical consistency.

These elements of quality tend to overlap, often in a confounding manner. Just as there is no single measure of accuracy, there is no effective statistical model for bringing together all these characteristics of quality into a single indicator. Also, except in simple or one-dimensional cases, there is no general statistical model for determining whether one particular set of quality characteristics provides higher overall quality than another.

Achieving an acceptable level of quality is the result of addressing, managing and balancing over time the various factors or elements that constitute better quality. Paying attention to the program objectives, the major uses of the data, costs, and conditions and circumstances that affect quality and user expectations is also important in determining an acceptable level of quality. Since the elements of quality have a complex relationship, an action taken to address or modify one aspect of quality tends to affect the other elements. Thus, the balance between these factors may be altered in ways that cannot readily be modeled or adequately quantified in advance. The decision and actions that achieve this balance are based on knowledge, experience, reviews, feedback, consultation and, inevitably, judgment.

Previous section