A Guide for First Nations on Evaluating Health Programs

Introduction

This guide is published by the Health Funding Arrangements Division, Program Policy Transfer Secretariat and Planning Directorate, First Nations and Inuit Health Branch (FNIHB), Health Canada. It presents basic information on evaluating health programs for First Nations communities that are taking control of their health programs under the Department's Health Transfer Initiative.

This guide is not a complete textbook on evaluating health programs in First Nations communities. Instead, it will help First Nations communities to understand what evaluation is and how it works. It also explains the need for evaluations and the part they can play in helping to conduct better community health programs.

Evaluation involves looking closely at programs to find out whether they did what they were supposed to do. It also means looking to see whether the results of the health program activities were worth the time and money communities spent.

Evaluators' reports are a very important part of the evaluation process and these reports are the final product or result of the evaluation work. Most of the information evaluators need to write their reports must come from communities, not from evaluators. The active involvement of communities is important. Therefore, communities need to develop and collect program information well before evaluation begins to facilitate program evaluation.

For more information on transferring control of health programs to First Nations and Inuit communities please refer to the following handbooks:

Transferring Control of Health Programs to First Nations and Inuit Communities: Handbook 1C An Introduction to Three Approaches provides an introduction to transfer of control of health programs and summarizes First Nations and Inuit Health Branch (FNIHB) policies concerning the control of health programs by First Nations and Inuit communities across Canada.

Transferring Control of Health Programs to First Nations and Inuit Communities:Handbook 2C The Health Services Transfer provides information about the health services transfer process and the procedures and policies for planning under the transfer approach.

Transferring Control of Health Programs to First Nations and Inuit Communities: Handbook 3C After the Transfer the New Environment explains what happens after transfer is completed.

Basic Information About Program Evaluation

This chapter answers two questions about program evaluation: What is program evaluation and why is it necessary?

What Is Program Evaluation?

Program evaluation is a way of measuring whether programs are doing what they are supposed to do. Evaluation means collecting information about program activity effectiveness and then comparing expected to actual results. Every program has one or more results that it wants to accomplish. Evaluations help communities to see if their programs are achieving these results.

Why Do Communities Have to Evaluate their Health Programs?

Evaluations help communities to understand how their health programs are working. At any point in the life of programs, communities need to know whether programs are doing what they set out to do. Therefore, the Transfer Framework requires that evaluation be completed every five years. These evaluations address the effectiveness of community health programs and objectives. Also, they identify changes in health status of community members. Communities agree to provide their evaluation reports to the Minister prior to the end of the five-year transfer period to allow for joint discussion and analysis with FNIHB before renewal of transfer agreements.

Evaluations show other communities what works in particular communities. For instance, a prenatal program may have had a very positive impact on pregnancies in one community. Other communities will want to know how to make such a program work for them. Evaluations describe what happens in programs and enables communities to learn from one another's experiences and to adapt successful experiences to their communities.

Evaluations show staff what they are doing. People working on programs need to see their work in a larger picture. Sometimes seeing past day-to-day challenges is difficult. Evaluations can show people whether their work is making a difference. Done properly, evaluations can give communities much information about their programs and lead to program improvements. Consequently, all community members benefit.

Linkage Between Evaluation and Program Management

As we mentioned earlier, evaluation is an essential activity that shows whether health programs are producing good results. Figure 1 below shows how evaluation fits into the Community Health Plan and shows the linkage between evaluation and program management.

Figure 1

Community Health Needs Assessment

Needs assessments are part of the planning stage of programs and identify community health priorities. Needs assessments help communities to understand the following:

• what kinds of health problems communities are experiencing;

• what causes these health problems;

• what resources are available to address these health problems;

• what goals and objectives communities need to write to help solve these health problems;

• which community members have the most urgent needs; and

• how best to meet the needs of community members.

Community Health Plan (CHP)

The CHP is the first step in planning what health programs communities want to provide for their members. When developing the CHP, communities have to begin thinking about evaluation. To illustrate, the CHP should specify the following four items that tie in with evaluation:

• the programs and activities that communities plan to conduct;

• the goals and objectives for each;

• the indicators that communities use in their evaluations to measure how well programs meet their objectives; and

• the day-to-day records and other evaluation information (data) that staff collect when programs are running.

Communities have to plan for evaluation from "day one" in the CHP.

Operating Programs

When communities conduct community health programs, they need to make sure that program staff collect the right kind of information on how well programs are working. Staff members need to collect this information every day. Communities need to identify the resources, people, money, materials and time needed to produce the desired results. If communities do not collect this information daily, they cannot give evaluators the tools they need to judge whether programs are improving community health. In other words - "No information, no evaluation."

Program Evaluation

The following questions form the core of evaluation and apply to all types of community health activities.

Did we do what we said we would do?

The answers to this question describe work done in programs and its relevance in meeting program objectives. Indicators provide the criteria that communities can use to measure program success.

What did we learn about what did and did not work?

The answers to this question give communities reasons for their success in conducting programs. Finding out what worked well in programs encourages participation in evaluation that focuses on success, learning and action. Health care providers, users and other interested parties need actively to participate in all phases of evaluation. Their involvement ensures that evaluations include ways to use the results throughout the life of programs. Participation in evaluation promotes ownership, focuses on community needs and encourages follow-up action.

Did this work make a difference?

The answers to this question measure program successes in changing knowledge, attitudes, skills and behaviour. Program success indicators identify the benefits that communities expect to gain from program work. Also, they provide the criteria against which to measure change throughout the life of programs.

What could we do differently?

Evaluations help communities to learn, and often the best way to learn comes from examining the challenges that programs present.

How do we plan to use evaluation findings for continuous learning?

Community participation in evaluation ensures that evaluation results are used throughout the life of programs.

Seeking answers to these key questions guides the evaluation process throughout the life of programs. By answering the questions, communities learn how to shape current and future programs.

Using the Results of the Evaluation

Communities can use evaluation results to help make decisions about their programs, to show where and how they need to revise community health plans, and to generate new program ideas.

Program Reformulation and Change

Program evaluation gives managers the facts they need to make informed decisions about community programs. Figure 2 below shows that evaluation is an ongoing process throughout the life of programs.

Figure 2

How to Prepare for Program Evaluation

Introduction

If communities have developed CHPs, they have already built the foundation for evaluating their health programs. A complete CHP lists the programs and activities, goals and objectives and the indicators for each program and activity. It also lists the information or data about program results.

Communities may want to use the points in the next few pages as a checklist to make sure that their CHPs are complete and updated. If communities have not yet completed CHPs, the information here can help them to put one together.

Program

Evaluators need to know the purpose of each program. For that reason, community members need to ask the following questions for each of their community health programs:

• Does this program meet a need?

• What is the purpose of this program?

For example, if teenage pregnancy is a concern in communities, include a teenage pregnancy program in CHPs. If it is not a concern, programs dealing with teenage pregnancy are a waste of time and money. Asking whether programs listed in the CHP are meeting community needs is an important step towards evaluating them.

The following definitions pertain to evaluations.

Goal

Goals are broad statements that describe what programs or activities should achieve. For example, the goal of a diabetes program could be " To help community members who have diabetes to lead more comfortable lives through learning about the benefits of better nutrition and regular exercise."

Objective

Objectives state exactly what programs should do. Objectives are identifiable and measurable actions to be completed by a specific time. When objectives are stated in measurable, time-related terms, evaluators can find out if objectives have been met. The following example shows a program objective:

The Maternal and Child Health Program has the following basic objective: "By December 1999, to reduce by 40 percent the number of fatal and non-fatal injuries to children younger than 10 years of age."

Activity

An activity is something communities do to help meet program objectives. Activities support programs and are really mini-programs. Programs may include one or more activities. They are the building blocks that make up programs. The four activities shown below would help to achieve the following objective: "By December 1999, to reduce by 40 percent the number of fatal and non-fatal injuries to children younger than 10 years of age."

• Conduct parent education classes to instruct parents in preventing childhood poisoning, burns and injuries to occupants of motor vehicles.

• Start school and playground inspections to identify and remove threats to child safety.

• Give information and counselling sessions for teachers about teaching safety to school children.

• Write a report on fatal and non-fatal childhood injuries to the band council recommending that council enact local by-laws to prevent pedestrian injuries to children.

Indicator

Evaluation indicators are signs, events or statistics that measure the success of programs or activities in meeting their objectives. Each program should include one or more success indicators. The greater the number of indicators, the more "rulers" available for measuring the effectiveness of programs or activities in improving the health of community members. For example, a success indicator for a prenatal program might be a decrease in the number of complications during or after delivery for both mothers and babies.

Hard and Soft Indicators

Indicators fall into two categories: "hard" indicators and "soft" indicators. Hard indicators are based on numbers - they are quantitative. For example, a decrease in the number of cases of a particular disease is a hard indicator. Soft indicators are not based on numbers - they are qualitative. An example of a soft indicator could be the satisfaction community members express about a program or activity.

Short-term and Long-term Indicators

Short-term indicators are signs that appear within a few weeks or months after programs or activities start and that show progress toward meeting objectives. Long-term indicators are signs that may take many months or years to show progress.

Data

Data is information about community programs and activities collected whenever programs start. Examples are statistics, records, surveys, community meetings, and interviews that supply information about programs. Examples of collected data are client information, immunization information, and user information gathered through community surveys and questionnaires.

Management Information System

A Management Information System (MIS) is computer software that collects, stores and processes program data efficiently. When collecting data, program managers should consider the following points:

• what information to collect;

• who will collect the information and how;

• what program indicators to measure;

• what reports to write;

• how to store data securely;

• how to restrict access to data; and

• how staff can use the data system during working hours.

FNIHB, in partnership with First Nations, has developed a computerized information system called the Health Information System. This system helps First Nations to establish their own program priorities and an independent approach to managing community health issues.

The examples below show how programs, goals, objectives, activities, indicators and data relate for a few typical programs.

Program
Childhood Injury Prevention

• Goal
  Decrease childhood injuries by changing the thinking of community members about childhood injury
  
  
• Objective
  By December 1999, reduce the number of deaths and injuries to children younger than 10 years of age by 40%
  
  
• Activity
  • Parent education classes
  • School and playground inspection
  • Teacher education
    
    
• Indicator
  Reduction in numbers of childhood deaths and injuries
  
  
• Data
  • Number of childhood deaths and injuries
  • Number of people attending parent education classes

Program
Adult Diabetes

• Goal
  Decrease the incidence of complications due to adult diabetes
  
  
• Objective
  By December 1999, reduce by 60% the number of diabetics presenting at the health clinic or hospital due to low or high blood sugar levels
  
  
• Activity
  Provide information and counselling to community members with diabetes about diet, weight control, exercise and self-care
  
  
• Indicator
  Number of adult diabetics needing treatment at home, in the clinic or in a hospital to stabilize blood-sugar levels
  
  
• Data
  • Number of diabetics receiving treatment
  • Number of people receiving counselling

Program
Immunization

• Goal
  Eliminate the incidence of childhood vaccine- preventable diseases through immunization
  
  
• Objective
  By December 1999, fully immunize 90% of the infants and preschool children according to provincial immunization protocol
  
  
• Activity
  • Operate regular immunization clinics
  • During pre- and post-natal visits, promote childhood immunization with mothers and caregivers
  • Encourage schools to require up-to-date immunization before school registration
    
    
• Indicator
  • Well-child clinic
  • Number of contacts, content of discussions, increased knowledge, attitude and practice of mother or caregiver
  • Existence of school policy requiring full immunization
  • Percent increase in immunization levels
    
• Data
  • Number of children vaccinated in community clinics
  • Number of mothers receiving information about immunization
  • Number of children vaccinated at school

This chapter has discussed the building blocks of evaluation - programs, activities, objectives, indicators and data. The information on how objectives, indicators and data for community programs and activities fit into the evaluation process should give communities enough background to begin working on evaluation plans. The next chapter will cover this topic.