Health Canada First Nations and Inuit Home and Community Care (FNI HCC) Program - Evaluation Guide

Conducting the Evaluation

This section provides suggestions for conducting surveys, interviews, and focus groups.

Conducting Surveys

If you have decided to use surveys as part of your evaluation, there are a number of ways to collect the data.

There are four main ways to conduct surveys:

1. Telephone
2. In-Person
3. Mail
4. Email/Internet

• Survey Method: Telephone
  When to Use:
  • Target population is dispersed over a wide geographic distance
  • In-person (face to face) interviews are not possible
  Advantages:
  • Convenient for interviewer
  • Respondents might feel more anonymous
  • Can cover a wide geographic area
  • Can clarify questions if respondent has difficulty
  Limitations:
  • Not everyone has a telephone
  • Respondents might be more comfortable discussing in person
  • Respondents must be at home
  
  
• Survey Method: In-Person
  When to Use:
  • Geography and/or time is not an issue
  • If you think you might get more information than over the telephone
  Advantages:
  • Intimacy
  • Can clarify if something is confusing
  Limitations:
  • Not everyone has a telephone
  • Respondents might be more comfortable discussing in person
  • Respondents must be at home
  
  
• Survey Method: Mail
  When to Use:
  • Resources to conduct the surveys over the phone or in-person
  are unavailable
  Advantages:
  • Respondent can do it at his/her convenience
  • Fewer human resources required
  Limitations:
  • Need a way of getting surveys returned
  • Requires a certain level of language comprehension
  • Response rates tend to be low
  
  
• Survey Method: Email / Internet
  When to Use:
  • Participants are geographically dispersed
  • Participants have access and a high level of comfort with email
       or Internet
  Advantages:
  • Surveys can be administered in a timely and cost-effective manner
  Limitations:
  • Target audience needs to be comfortable with the technology
  • No human interaction

What kinds of questions should you ask?

Now that you know how you are going to survey respondents, you need to come up with the questions.

Closed Ended vs. Open Ended Questions

These are two technical terms to distinguish the two main types of survey questions. However, it is worth noting the distinction between the two types.

For closed-ended questions, survey respondents must choose from a pre-determined list of responses. In the example below, respondents must pick one of four responses provided.

Example: How long have you been a volunteer care worker?

___    Less than 1 year
___    1 to 3 years
___    3 to 5 years
___    More than 5 years

As the name implies, open-ended questions allow for an unlimited (open) number of responses. There are no lists or pre-determined answers.

Example: Why do you continue to serve as a volunteer care worker?

There are reasons to use both types of questions but keep in mind that closed-ended questions are easier to administer and analyze. Open-ended questions require coding (classifying similar responses into groups) and are more labour-intensive. As such, the fewer open-ended questions you have the more manageable your survey will be.

Tip!
For every question you come up with ask yourself "How will this information be used? " If it is not critical to the evaluation, consider removing it from the survey. It is easy to get caught up in asking the "nice to know" questions. Focus on the"need to know" issues.

Demographic Questions
These questions relate to a respondent's gender, age, language, occupation, family composition, household income, etc. They are used to see if there are any patterns among the different variables used. It is always a good idea to reinforce that information is confidential. Ensure that the questions are culturally sensitive and test them by obtaining feedback from others.

Length of Surveys
In general, the shorter the better. However, you must ensure that the survey gets all of the information you need for your evaluation.

For each question, ask yourself "What will I do with this information? " If it is more"interesting" than "necessary" consider removing that question.

Test it!
Have a couple of people who are not close to the project complete the survey. You can observe them to see any trouble spots as they go. Alternatively you can give them a written copy and get their feedback after. As well, you can visually see by their answers if there were questions they skipped or did not complete correctly.

Who do you survey?
This relates back to the objectives of the study and the information you are trying to obtain. Some of the different groups that might be the subject of a FNI HCC survey include:

• Clients
• Care providers
• Employees/volunteers
• Community leaders
• General community

How many completed surveys do you need?
There is rarely a magic number and the purpose of this document is not to get into the actual math process involved. A good rule of thumb is that the more you get, the more confidence one can have in the results. Try for as many as you can obtain and manage, while ensuring that you keep within the target audience you want to survey.

Conducting Interviews and Focus Groups

If interviews or focus groups are part of your evaluation, this section provides some suggestions designed to make the process go smoothly and efficiently.

Types of Interviews & Focus Groups

• Method: In-Person Interviews
  When to Use:
  • The topic is not appropriate to discuss in a group
  • Geography is not an issue
  Advantages:
  • Respondents may feel more comfortable than in group environment
  • Personal
  • Can probe to obtain in-depth perceptions and opinions from individual
  • Can observe body language
  Limitations:
  • Take comparatively more time
  • Do not obtain the dynamics associated with groups
  
  
• Method: Telephone Interviews
  When to Use:
  • The topic is not appropriate to discuss in a group
  • Geography IS an issue or coordinating times for personal interviews
    is too inconvenient
  Advantages:
  • Convenient for both interviewer and respondent
  Limitations:
  • Interviewer is "only" hearing respondent's voice (not able to
    observe body language)
  • Less personal
  
  
• Method: Focus Groups (6 to 10 participants)
  When to Use:
  • When the benefits of group dynamics outweigh the need for
    private communication
  Advantages:
  • Opinions and beliefs tend to be more spontaneous in groups;
    lead to other valuable discussions
  • Can talk to more people in shorter period of time
  Limitations:
  • Some people may not be comfortable revealing opinions with strangers or other community members
  • It can be challenging keeping a group on track, getting quieter
    people to speak up or more outgoing ones to moderate the
    amount they speak
  
  
• Method: Mini Groups (Approximately 4 participants)
  When to Use:
  • When you are looking for more in-depth feedback from individuals but
    want some group dynamics
  Advantages:
  • Can be easier to control smaller group
  • Each person has more "air time" for opinions
  Limitations:
  • You may need to do more groups to obtain enough opinions