The Health Canada Policy Toolkit for Public Involvement in Decision Making

Level 2 Technique:
Surveys (also see "Questionnaires")

What Is It?

A survey is a method of primary data collection based on communication with a representative sample of individuals. Surveys are usually descriptive in nature, yet can also be used to provide casual explanations or explore ideas. A survey can be conducted using different information-gathering techniques such as mailout questionnaires, in-person interviews and telephone surveys. There is also an increasing use of technology driven fax, email and Internet-based surveys.

Determining which method is best to use depends partly on factors such as:

• The study's purpose
• The topic
• The target population's characteristics
• The importance of avoiding the use of volunteers for the survey
• The researcher's resources

When Is It Most Useful?

Surveys can fulfil a number of objectives such as identifying a group's characteristics, measuring attitudes and describing behavioural patterns. The advantages of conducting mailout questionnaires, in-person interviews and telephone surveys are that they are:

• Accurate
• Inexpensive
• Efficient
• Quick

Logistics and Limits

Comparing the three survey methods

Mailout Questionnaires

Mail questionnaires are quite structured and the questions should mean the same thing to every respondent; thus, questions and instructions must be easy to read and straightforward.

Special considerations must be made for people who will answer questions on their own.

• Respondents must be literate in the language and terminology used in the study.
• Researchers must ensure that the terminology used in the questions is appropriate.
• Ambiguities and misunderstandings should be expected when designing questions.

Advantages
• Reduction in the error that may result from variability of interviewer's skills
• Greater anonymity
• Mail questionnaires necessary when questions demand a considered rather than an immediate response
• Accessible coverage of geographic area at minimal cost.

 
Disadvantages
• Requires simple questions in the absence of an interviewer
• Provides no opportunity for probing, if mailout type
• No control over who fills out the questionnaire
• Generally elicits a low response rate (between 20% and 40% (without follow-up).

Interviews

Face-to-face contact will most often provide higher response rates, and the opportunity to clarify ambiguities or misunderstandings and to monitor the conditions for completing the questionnaire. Inperson questionnaires and interviews share some advantages and disadvantages because they are both interactive and both rely on self-reported answers.

Advantages
• Versatility and the opportunity to hear feedback from a respondent
• Opportunity to follow up or probe complex answers
• Interviews more conducive to lengthy questionnaires
• Personal interviews increase chance of completed questionnaires.

 
Disadvantages
• Respondents not anonymous, therefore there may be reluctance to provide confidential information
• Bias due to different interviewer capabilities
• Characteristics of interviewer influence responses.

Telephone Interviews

The telephone interview, often called the telephone survey, can be described as a semi-personal method of gathering information. Telephone interviews used to be viewed with skepticism because of the high likelihood of a sampling bias by over-representing the population who could afford telephones. It is now a respected research method and may actually increase the quality of the data because the telephone interview can be supervised.

Advantages
• Convenient, quick and cost-effective with the advent of technology
• Often receives a higher response rate than door-to-door interviews because people may not want to open their doors to strangers
• Increased accuracy because interviewers are monitored.

 
Disadvantages
• Non-response (respondents hanging up before completing the questionnaire)
• Uneasiness about divulging certain information over the phone
• Limited duration of interview necessary to prevent uncompleted interviews.

Cost Implications

Personal interviews are usually more expensive than both telephone interviews and mail questionnaires. Pretesting the questionnaire to a small sample helps to identify errors before they become costly to change. Additional cost considerations include:

Mail questionnaires
Inexpensive:
• Cost of stationery
• Stamps
• Follow-ups

 
Telephone Interviews
More expensive:
• Expensive and efficient technological advances when the timing of data collection is not a factor

 
Personal Interviews
Relatively expensive:
• Travel cost of interviewers

Expectation for Feedback or Follow-up

Regarding mail questionnaires, follow-up letters may be necessary for reminding respondents to return their questionnaires. Furthermore, to increase the rate of survey returns, reference can be made in the covering letter regarding how the results of the survey will be sent to all respondents. As well, it can be advantageous to make results available upon request.

• Results of the survey have to be interpreted and this is often contracted out to a research company which can provide experience and credibility.
• A final report and summaries of the results and interpretations are almost always prepared.
• Communication lines can be established for media and other stakeholders.

Timelines

• Personal interviews provide fast access to data.
• Mail questionnaires take the longest time to gather information.
• All methods require careful consideration of the questionnaire's design.
• Time is also required to input/code the information.

Potential Pitfalls

Special care must be made to avoid general disadvantages of survey errors, such as:

Random sampling errors

• The difference between the result of a sample and the result of a census of the entire population conducted using identical procedures

Systematic errors

• Sample bias: when the sample is not representative of the population
• Non-response error: caused by people who are sampled but do not respond and by those who may differ from the respondents in a significant way
• Mistakes in recording responses.