Our mission is to help the people of Canada
maintain and improve their health.
Health Canada

National Clearinghouse on Family Violence
Family Violence Prevention Unit
Public Health Agency of Canada
Health Canada (Address Locator: 1909D1)
Ottawa, Ontario K1A 1B4

Telephone: (613) 957-2938
or call our toll-free number:
1-800-267-1291
For TTY users: (613) 952-6396
or call toll-free:
1-800-561-5643
Facsimile: (613) 941-8930
FaxLink: (613) 941-7285
or toll-free: 1-888-267-1233

Published by the authority of the Minister of Health Canada, May 1996

Également disponible en français sous le titre : Guide d'evaluation de projet Une démarcheparticipative

This publication can be made available in or on (computer diskette, large print, audio-cassette, braille) upon request.

Please feel free to photocopy, with acknowledgement to Health Canada, any useful information in this document.
 

Cat. No. H39-355/1996E
ISBN 0-662-24231-9

ACKNOWLEDGEMENTS

Special thanks go to the writers of this Guide, Donna Denham and Joan Gillespie, of Denham Gillespie Associates, who were instrumental in the design and development of this document.
 

CONTENTS
 

Chapter	1.	Introducing the Guide
	1.1	Why evaluate?
	1.2	Getting started
	1.3	A note on terminology
Chapter	2.	Evaluation for Learning
	2.1	Participatory evaluation
	2.2	Putting participatory evaluation into practic
	2.3	Activity: Introducing a participatory approach to evaluation
	2.4	Handout: Principles of a participatory approach to evaluation
Chapter	3.	A Framework for Project Evaluation
	3.1	The five key evaluation questions
	3.2	The five evaluation process steps
	3.3	Tools for using the evaluation framework
	3.4	A framework for project evaluation
Chapter	4.	Defining Project Work
	4.1	Developing project goals and objectives
	4.2	Writing Project Objectives
	4.3	Activity: Writing Effective Project Objectives
	4.4	Role of the outside evaluator
	4.5	Points to remember
Chapter	5.	Developing Success Indicators
	5.1	Purpose of success indicators and their measures
	5.2	The process of developing success indicators and their measures
	5.3	Activity: Defining success indicators
	5.4	Success indicators for project activity types
Chapter	6.	Collecting Evaluation Data
	6.1	Determining information collection needs
	6.2	Information collection tools
	6.3	Sample evaluation tools
Chapter	7.	Analysing and Interpreting Data
	7.1	Analysing project evaluation information
	7.2	Preparing useful evaluation reports
	7.3	Activity: Analysing and interpreting data
Chapter	8	Using Evaluation Results
	8.1	Using evaluation results
Chapter	9	Putting It Together
Appendices
	1.	Definitions of evaluation terms
	2.	Annotated bibliography of evaluation resources
	3.	Framework worksheet for the five key evaluation questions and examples of developing the questions by project activity type
	4.	Framework worksheet for success indicators and examples of developing the indicators by project activity type
	5.	Success indicators of increased public participation and strengthened community groups
	6.	Reaction sheet for evaluation workshop
	7.	Guided telephone interview - Community Resource Handbook for Women with Breast Cancer
	8.	Focus group interview guide - Child, Safety Awareness Program
	9.	Guidelines for keeping a project diary - Child Safety Awareness Program
	10.	Mail-out questionnaire -Advisory Committee, Health and Disabled Women's Project
	11.	Mail-out questionnaire - Health Care Providers, Health and Disabled Women's Project
	12.	We Want Your Feedback

1    INTRODUCING THE GUIDE

1.1     Why evaluate?

Effective project evaluations can

•  account for what has been accomplished through project funding
• promote learning about which health promotion strategies work in communities and which don't
• provide feedback to inform decision-making at all levels: community, regional and national
• contribute to the body of knowledge about health promotion
• assess the cost-effectiveness of different health promotion strategies
• position high quality projects for future funding opportunities
• increase the effectiveness of project and program management
• contribute to policy development.

1.2     Getting started

The Guide to Project Evaluation: A Participatory Approach provides direction for your work in planning and implementing effective project evaluations.

While no single resource can answer all your questions, we hope that the Guide to Project Evaluation.- A Participatory Approach provides you with clear directions. Add to it, adapt it, and customize it to meet your own needs.

1.3     A note on terminology

For many people the language of evaluation is a barrier that prevents them from getting on with the real evaluation work. This guide attempts to avoid this problem by using plain language throughout. Appendix 1 provides a brief overview of definitions of the more common evaluation terms.

To make the guide as practical as possible it includes

• a framework to guide the step-by-step process of developing effective evaluations.
• activities to introduce and plan for project evaluation.
• examples demonstrating the application of the evaluation framework to health promotion projects.
• an annotated bibliography of a selected number of useful evaluation resources (see Appendix 2).

This guide is based on the belief that evaluation can be a useful and positive experience that promotes learning and action. What is learned from project evaluation is as important as what the project produces or creates.

2.1     Participatory evaluation

Health promotion activities enable people to take more active roles in defining their health needs, setting priorities among health goals and influencing and assessing efforts to improve their health. Participatory evaluation work supports these activities because it is a collaborative approach that builds on strengths and that values the contribution of everyone involved. While there are other approaches to evaluation, a participatory approach seems most consistent with the goals of the Public Health Agency of Canada's strategies and programs.
 

Principles of participatory evaluation Participatory evaluation focuses on learning, success and action. The evaluation must be useful to the people who are doing the work that is being evaluated. The evaluation process is ongoing and includes ways to let all participants use the information from the evaluation throughout the project, not just at the end. Recognition of the progression of change - knowledge, attitudes, skills and behaviour - is built into the evaluation. The project sponsors are responsible for defining the specific project evaluation questions, the indicators of success and realistic timeframes. Participatory evaluation makes it possible to recognize shared interests among those doing the work, the people the work is designed to reach, the project funders and other stakeholders.

For a more detailed examination of these principles, refer to the handout on page 6, Principles of a participatory approach to evaluation.

2.2     Putting participatory evaluation into practice

Participatory evaluation calls for collaboration among those who share a common interest in improving health. The collaborative process starts at the beginning of a project and continues throughout the life of the project. This type of evaluation is never a one-time, end-of-project event.

Refer to Chapter 9 of this guide, Putting it Together, for a checklist of common points to consider in each stage of project evaluation.

Collaboration allows those involved in the project to

• work in partnership with community groups to do evaluation
• recognize the experience and expertise of community groups
• recognize the health outcomes of the project
• make evaluation questions and findings relevant to all stakeholders
• increase the acceptability of and support for the evaluation process and outcomes
• produce more meaningful results that can be used by both programs and projects to learn how to improve the work being done and to influence policy and program directions.

For a thorough discussion of the principles and application of participatory evaluation, we highly recommend the following two resources:

• Keeping on Track: An Evaluation Guide for Community Groups, produced by the Women's Research Centre of Vancouver
• The Royal Society of Canada, Study of Participatory Research in Health Promotion, prepared by the Institute of Health Promotion Research, University of British Columbia.

 2.4     Handout: Principles of a participatory approach to evaluation

Participatory evaluation encourages a positive experience with the evaluation of health promotion activities. The key principles of this approach are outlined below. They have been adapted from Keeping on Track, An Evaluation Guide for Community Groups, produced by the Women's Research Centre in Vancouver.

•  Participatory evaluation focuses on learning, success and action.

An important question to ask in evaluation is what we learned about what worked and what did not work. Then we need to ask how can we use these learning's to move to action. The people and groups most directly involved decide what determines success.

• The evaluation is useful to the people who are doing the work that is being evaluated.

The project's goals and objectives - what the project intends to accomplish - must be the standards against which the project work is measured. Evaluators must pay special attention to the project's specific needs and available resources.
 
• The evaluation process is ongoing and includes ways to let all participants use the information from the evaluation throughout the project, not just at the end.

The material produced for the evaluation must be given back to the participants on an ongoing basis in a format that is useful and clearly written in plain language.
 
• Recognition of the progression of change - in knowledge, attitudes, skills and behaviour - is built into the evaluation.

To measure people's success in changing knowledge, attitudes, skills and behaviour, think in advance about the kinds of changes the project strategies and activities can produce. It is important to describe how these changes can be recognized and measured in a way that is possible and practical within the timeframe and resources available to the project.
 
• The project sponsors are responsible for defining the specific project evaluation questions, the indicators of success and realistic timeframes.

Community sponsors of projects must participate in decisions about what questions will be asked and what information will be collected to measure the difference, the work made in a given period.
 
• Participatory evaluation makes it possible to recognize shared interests among those doing the work, the people the work is designed to reach, the project funders and other stakeholders.

The evaluation must include information and input from the people doing the work, the people who the work is designed to help or reach and the project funders.

The evaluation framework presented in this guide meets this challenge. It is composed of two parts:

• five key evaluation questions
• five evaluation process steps

The following two sections discuss the evaluation questions and the process steps. An overview of the evaluation framework is on page 14.

3.1     The five key evaluation questions

The process of developing the answers to the evaluation questions will vary, as each project varies, but the five fundamental questions remain the same.
 

5 key evaluation questions What?1 1. Did we do what we said we would do? Why? 2. What did we learn about what worked and what didn't work? So what? 3. What difference did it make that we did this work? Now what? 4. What could we do differently? Then what? 5. How do we plan to use evaluation findings for continuous learning?

1.     Did we do what we said we would do? "What?" (Description of activities)

The responses to this question describe the work done in the project and the relevance of this work in meeting the project goals and objectives. The project success indicators provide the criteria against which success is measured. They assist the project sponsor to collect the information needed to answer this and subsequent evaluation questions. (Chapter 5 discusses how to develop project success indicators.)

Some of the more specific questions that may need to be answered to describe the project work include the following

•  What activities were undertaken and how did they link to meeting the project goals and objectives?

Examples:
            - Describe the resources that were developed to increase awareness.
            - Describe the training workshops that were conducted for skill development.
            - Describe the new partnerships that were formed to work on accessibility issues.
• What were the major achievements of the project and what resources did they require?
• If the objectives changed during the course of the project, how and why did they change?

Participatory evaluation focuses on success, learning and action. Finding out what worked well in a project and what didn't work well practices this principle. Here are some of the questions that could be included in this discussion:

• What strategies worked well for involving the target population in the project. Why?
• What strategies didn't work well for involving the target population in the project. Why?
• What strategies worked best for broadening the base of community support for the project. Why?
• What strategies didn't work well for broadening the base of community support for the project. Why?
• Which activities and strategies did we change. Why?
• What was learned about the relative cost-effectiveness and efficiency of various project strategies and activities?
• How realistic and relevant were the project goals and objectives?
• In what ways did the project planning process work most effectively?
• What did we learn about working together as a group?

The answers to this question measure a project's success in changing knowledge, attitudes, skills and behaviour. The project success indicators represent the group's assumptions about what changes should be expected from the project work and provide the criteria against which to measure change both during and at the end of the project. (Chapter 5 discusses how to develop success indicators.)

There are two main ways project sponsors can assess impact: by using summarized data related to the success indicators and by asking specific impact questions of people who were involved in the project and who were the target of the project's work.

The following types of questions may be helpful in discussions about this part of the project evaluation:

• What changed as a result of the project?

- knowledge
- attitudes
- skills
- behaviour
• What changed as a result of the project for

- members of the target population?
- community groups?
- service providers?
- caregivers?
- project sponsors and staff
• Were there any unexpected changes resulting from the project work? Describe them.
• In what ways did this project contribute to increased public participation?
• In what ways did this project help to strengthen community groups?
• To what extent did the project reduce barriers to health?
• What evidence is there to attribute any of the above changes to the project? What other factors outside the project might have contributed to the changes?
• Were other initiatives started, alternative services proposed or new funding resources acquired as a result of this project?
• In what ways did this project contribute to better health practices?
• What new partnerships developed from this project? What was the nature of the partnerships and what was their contribution?
• Is the model or approach continuing beyond the initial funding?
• To what extent is this model or approach transferable to other communities?

Evaluation is for learning and often the best learning comes from examining the challenges that projects present. Here are some of the questions that could be included in this discussion:

• What more effective methods for achieving the objectives emerged from the work?
• What additional knowledge development is required to do the work more effectively?
• What additional support from the funders and community sponsoring agencies would have been useful to the project in meeting its goals and objectives?
• Are there more cost-effective ways to achieve the project's objectives?
• Who else could have been involved in the work?
• What could we do to expand the network of people involved in working on this issue?
• Were all the project's needs met?
• Is there a better way of developing realistic project goals and objectives in the initial planning stage?
• How did management and administrative systems change through the project to become more effective?

Participatory evaluation includes ways to use the evaluation results throughout the project as well as at the end. Some questions to consider in developing the evaluation are as follows:

• How were evaluation findings used on an ongoing basis to contribute to the planning and implementation of the project strategies and activities?
• How will project findings be used for future knowledge development?
• How will the final evaluation learnings be documented and distributed?
• Are there alternative ways to present the evaluation results so that more people can make use of the learnings?
• How will the evaluation results be used for new project planning?
• How will the evaluation results be used to influence policy and research priorities?

3.2     The five evaluation process steps

The steps to developing answers for the five key evaluation questions are briefly outlined below, and then are further developed in the next five chapters of the guide.

1.     Define the project work.

To evaluate a project there must be clear, measurable project goals and objectives that outline what the project plans to accomplish. While this may seem self-evident, many evaluations have gone off the track because this initial work has not been done.

Chapter 4, Defining Project Work, provides ideas on how to strengthen the development of clear project goals and objectives.

2.     Develop success indicators and their measures.

The process of defining what constitutes success for a project is another important step in developing evaluations. Project sponsors need to define the success indicators for their projects. The success indicators allow project sponsors to evaluate whether they accomplished what they set out to do and what the impact of their project has been.

Chapter 5, Developing Success Indicators, discusses this process in more detail, gives some examples of specific indicators and describes an activity that could be used to help identify success indicators for projects.

3.     Collect the evaluation data.

After the first two steps have been completed, it is necessary to decide

• what information the project needs to collect
• who has the information
• how the information will be collected.

4.     Analyse and interpret the data.

As the evaluation data is collected, it should be summarized and analysed and key learnings should be identified. This ongoing process will help projects prepare their final evaluation reports.

Chapter 7, Analysing and Interpreting Data, provides some ideas to help with this process.

5.     Use the evaluation results.

Evaluation findings can be used throughout the project to improve the planning and implementing of project activities. By sharing project results with others, each project adds to the body of knowledge about health promotion.

Chapter 8, Using Evaluation Results, provides ideas on how to use evaluation findings during and after the project.

Working through these five steps will provide project sponsors with the information and tools they need to answer the five key evaluation questions. For small projects with limited resources, the process will be simple and straightforward. For large projects with greater resources, the work involved in each step will vary to reflect the complexity of project goals and objectives.

For all projects, project sponsors should:

• set realistic limits on the number of project-specific evaluation questions and on the amount of evaluation information to be collected, as determined by the evaluation resources available to the group
• remember that the quality of information collected, not the quantity, is the most important factor in evaluation.

3.3     Tools for using the evaluation framework

To help in applying the evaluation framework, several different tools have been developed for this guide. Examples provided reflect the most common Health Canada project activity types, which are

• needs assessments
• education and awareness
• resource development
• skills development
• developing innovative models
• reducing barriers to health.
• One-page overview of the Framework for Project Evaluation (see Section 3.4)
• Framework Worksheet for the Five Key Evaluation Questions and Examples of Developing the Questions by Project Activity Type (see Appendix 3)
• The blank worksheet can be used by projects to develop the five evaluation questions. The examples show how the questions can be further developed to reflect the specific evaluation needs of projects.
• Framework Worksheet for Success Indicators and Examples of Developing Indicators by Project Activity Type (see Appendix 4)
• The blank worksheet can be used by projects to develop their own project specific success indicators and their measures. The examples provide ideas for developing success indicators and measures that projects may find useful.

An overview of the framework for project evaluation is presented on the next page. This overview is a useful tool that can be used for

• introducing the framework
• reviewing the evaluation work
A Framework for Project Evaluation

4    DEFINING PROJECT WORK

4.1     Developing project goals and objectives

The project goals and objectives describe what the project wants to accomplish and provide the context in which the five evaluation questions are answered. If the project goals and objectives are not clear, it will be very difficult to answer the first evaluation question, "Did we do what we said we would do?"

Goals are general statements of what a project is trying to do.

Objectives are specific, measurable statements of the desired change(s) that a project intends to accomplish by a given time.

4.2     Writing Project Objectives

Clear project objectives are essential to project work and effective evaluation. Good project objectives set the groundwork for demonstrating the impact of the project. Writing project objectives, however, can be challenging for many groups.

Many people confuse objectives with activities. For example, a project may state that their objective is to create a video explaining how HIV/AIDS is transmitted. Creating a video is an activity. The objective the activity wishes to achieve is an increase in knowledge on how HIV/AIDS is transmitted.

There are two helpful guidelines to use in writing good project objectives: (1) identify the specific changes the project is designed to accomplish, and (2) ensure that these changes are measurable.

To help identify the specific project objectives, it is useful to ask the question:

What are we trying to change?

Projects generally focus on change in the following key areas: €     knowledge (increasing knowledge on a particular issue or subject)
€     attitudes (creating an attitude that favours, a desired behaviour)
€     skills (developing the individual capacity to adopt a given behaviour)
€     behaviour (maintaining or adopting a healthy behaviour) These key areas may be seen as a kind of continuum of change. A change in knowledge can lead to new attitudes. Developing skills can enable people to make positive changes in their behaviour.

Once the areas of change have been identified, it is important to ensure that they are measurable. There are five important elements to consider when creating project objectives that are specific and measurable. These elements are listed below in random order: €    the date by which the change will occur
€    the specific change desired (use action verb)
€    a measure (number or percentage)
€    the target group
€    the location Although their use may vary from one project to another, a good rule of thumb is to write project objectives that include these five elements.

For example, a project with the goal of increasing awareness of factors related to HIV/AIDS transmission among high school students might create a project objective that reads:

By August 1, 1996 (date)/the knowledge of the factors involved in HIV/AIDS transmission (specific change)/will increase by 30% (measure)/among high school students(target)/in Montréal(location).

The following page contains an activity that can help in practicing how to write good project objectives.

4.3     Activity: Writing Effective Project Objectives
 

Topic:	Writing project objectives
Purpose:	To give project sponsors a chance to write effective project objectives for their project.
Time:	1-2 hours
Materials:	flipchart project proposal Guide to Project Evaluation, Chapter 4
Activity:	Have participants refer to the project proposal for their project.   Working in small groups, have participants review the project objectives. Do the project objectives provide enough information to answer the question "What are we trying to change?"   Have participants examine the project objectives to see if they contain the five key elements (refer to Chapter 4 of the Guide):   date specific change desired measure of change target group location   If the objectives contain the five key elements, have participants break down the objectives into the five elements. If the objectives do not contain the five key elements, have participants rewrite the objectives to include these elements.   Bring all participants together to share their results and to discuss their ideas on which objectives are most useful and on how to keep the number of project objectives manageable given the scope and resources of the project.

4.4     Role of the outside evaluator

In small projects with limited resources, the evaluation can usually be done by the project sponsors. Larger projects, having correspondingly larger evaluation requirements, often hire an outside evaluator.

If an outside evaluator is being used, it is essential that project sponsors clarify the evaluator's roles and responsibilities.

Questions to consider when hiring an outside evaluator:

• a list of possible evaluators, including their profiles: what their strengths and weaknesses are, projects they have worked on, any experience working with them previously
• ideas on different roles for outside evaluators, e.g. working with project sponsors to develop the evaluation plan, developing some or all of the data collection tools, analysing the data, writing the summary reports
• sample contracts with outside evaluators
• guidelines on when to use outside evaluators for projects.

5    DEVELOPING SUCCESS INDICATORS

Identifying what success will look like during the developmental phase of a project may seem a little like putting the cart before the horse. Many project sponsors spend a lot of time developing goals and objectives, planning activities and thinking about budgets. The real challenge is to think to the end of the project and name the identifiable changes that they expect to occur as a result of doing the work. These identifiable changes, the success indicators, should be developed as soon as clear project goals and objectives have been established. Therefore, identifying success indicators is the second step in the process of planning high quality project evaluation plans. Project sponsors should identify the success indicators that are most appropriate and best reflect the reality of their own projects.

5.1     Purpose of success indicators and their measures

Success indicators are a group's assumptions about what changes should be expected from doing the project work. These indicators are quantified by specific measures for example, a number, a percentage or a level of satisfaction.

Success indicators and their measures need to link directly to project goals and objectives since they provide the objective and measurable criteria by which groups judge the degree of success they have had in reaching their goals and objectives.

Through their project activities, project sponsors attempt to change the knowledge, attitudes, behaviour or skills of a selected group of people - sometimes referred to as the target group. To measure or evaluate the amount of change, it is useful to know the status of the target group's knowledge, attitudes, behaviour and skills at the beginning of the project. Determining this initial status or starting point is called setting a benchmark. This initial benchmark helps the project determine the amount of change it is trying for in the project. The example below may help to illustrate this process. Other examples of success indicators can be found in Appendix 4 (Examples of Developing Indicators by Project Activity Type) and in Appendix 5 (Success Indicators of Increased Public Participation and Strengthened Community Groups).

Example: Breast Cancer Network Project

• One of the project goals is to promote the development of survivor/directed, self-help groups for women with breast cancer
• A project objective linked to this goal is to raise the awareness of the need for and success of such groups among health care professionals and cancer societies.
• The target group for the project is 200 health care professionals in the project's urban area

The project benchmark was established using the questionnaire completed for the initial needs assessment for the project. The results of the questionnaire indicated that 15% of the 200 health care professionals in the community knew about the advantages of self-help groups and referred breast cancer patients to them.
• The project success indicator is to have 60% of the 200 target group health care professionals know about and refer to self-help groups for breast cancer patients.

5.2     The process of developing success indicators and their measures

Choosing which indicators are the "best" is not an exact science. The process that project sponsors go through to identify their success indicators is as important as the final list of indicators created. Done well, this process can contribute to the building of commitment and excitement for doing an evaluation. It also helps groups develop reasonable expectations of what can be achieved.

Some guidelines for developing success indicators

Success indicators should:

1.     Be results-focused i.e. refer to results or outcomes of the funded activity and not the activity itself

2.     Be challenging but feasible.

3.     Involve a meaningful comparison - a comparison over time, a comparison with other similar activities or (preferably) a comparison against a reasonable
        standard.

4.     Be measurable, using quantitative or qualitative measures. In developing indicators, consideration should be given to data availability and data collection, given
        the resources available.

5.     Refer to a result or outcome that can be reasonably attributed to the project activity.

6.     Be as valid (directly related to the work done and not attributable to other factors) and reliable (able to be replicated) as possible.

7.     Meet the criteria of
            - selectivity i.e. the number of indicators are limited to and focused on the key areas of concern.
            - balance i.e. the indicators refer to a range of project activities and results that together will provide a balanced assessment of project success.
            - usefulness i.e. the potential use of the evaluation information should be taken into account when developing indicators to ensure that they capture the
               relevant information.
 

Benefits of developing good success indicators: clarification of project goals and objectives to make them measurable identification of innovative success indicators that reflect unique community characteristics and needs strengthened strategies and workplans to address some of the identified barriers to success increased commitment to assess impact questions

The activity on the next page has been used with a number of community groups to help them identify success indicators for their projects. Some of the most useful indicators of success have been developed when members of the target population and project sponsors have undertaken this activity together.

5.3    Activity: Defining success indicators
 

Topic:	Defining project success indicators
Purpose:	To give project sponsors a chance to define the indicators of success for their project.
Time:	2-3 hours
Materials:	flipchart handout: an outline of the project objectives and a list of the activities or strategies that are to be undertaken to achieve the objective handout: Success Indicators of Increased Public Participation and Strengthened Community Groups (Appendix 5) Note: Appendix 5 is for use as an example only. The project sponsors need to develop their own success indicators that are relevant to their project.
Activity:	Divide the participants into pairs or groups and assign one of the project objectives and the activities associated with that objective to each group. Ask the groups to list five things for each activity that would indicate that the activities were successful. Examples of possible activities: media campaign, organization of a mother's group, development of brochures, needle exchange program, creation of a 1-800 line. Encourage participants to use all their senses in developing indicators: e.g., What changes do they expect to see? hear? feel? Put quantities on criteria when possible. The participants must be realistic about what they hope can be accomplished. Remind them that useful indicators are measurable, specific, easy to collect information on and ultimately can provide useful information to the group.   Discuss the success indicators with the total group and add new ones. You may want to review the list of indicators for increased public participation and strengthened community groups to see if any of them are appropriate for the project.   With the total group, review the indicators and order them by priority so that only the ones that are most useful and important are selected. The task is to keep the number of success indicators manageable for the project resources.   Once there is agreement on the indicators, the group could begin to look at the kind of information that needs to be collected to document the degree of success.

5.4     Success indicators for project activity types

Although there are many different types of projects funded under Health Canada programs, certain project activity types appear more frequently than others. They have been identified as

• needs assessments
• educational and awareness
• resource development
• skills development
• developing innovative models
• reducing barriers to health

Appendix 5 provides examples of indicators of success for two health promotion program/project impacts - increased public participation and strengthened community groups.

These examples have been included to stimulate thinking and to start the process of developing project-specific success indicators.

6    COLLECTING EVALUATION DATA

Participatory evaluation relies on a systematic and rigorous collection of information from project staff and stakeholders. It draws on both quantitative and qualitative data to measure success and to clarify and make decisions about project characteristics, activities and effects.

6.1     Determining information collection needs
 

Three questions to ask in determining evaluation information needs: 1.  What information is needed? 2.  Who has the information? 3.  How will the information be collected?

1.     What information is needed?

Projects need to collect evaluation information that will provide answers to the five key evaluation questions. The specific type of information to be collected is determined by the work done at the beginning of each project to define the project goals, objectives and success indicators.

2.     Who has the information?

Depending on the nature of the project, the people with information useful to the project evaluation will vary widely. People from whom it may be important to collect information include

Project sponsors decide how best to collect evaluation information based on their project's needs and resources. Designing the information collection tools should be done in collaboration with the people who will be using them. Most community projects don't have the time or the resources to put into extensive recording of data and statistics. The goal is to find ways of collecting information that do not put too much of a burden on the people doing the project work but that still provide the information required to answer the evaluation questions.
 

Characteristics of a good information collection process: useful practical collaborative systematic ongoing accurate ethical

6.2     Information collection tools

There is a wide variety of information collection tools that can be used depending on the project's evaluation needs. Examples of tools that have been used in other projects are listed below.

Written survey questionnaire

• structured questionnaires used to reach large numbers of people
• provides quantitative data (numbers) that can be statistically analysed and qualitative information that can be summarized
• used to survey target population in terms of knowledge, attitudes, beliefs and behaviour.

- When developing the questions for the questionnaire, ensure that they are not worded in ways that lead to biased or misleading
   responses.
- While mass mailing of survey questionnaires has the advantage of reaching large numbers of people, there is no guarantee that people
   will fill out and return it, so the actual response rate may be low.
- Limiting the number of questions may increase the response rate.
- Using smaller but targeted mailings, followed up by a phone call, may increase the response rate.
- Paying attention to respondents' literacy level, language and visual capacity may increase the response rate.
- All survey questionnaires need to be pilot tested to ensure that the questions succeed in getting the information that is required.

Telephone survey

• can ask for the same types of information as the written survey questionnaire
• has the advantage of increasing accessibility and allowing immediate clarification of questions if the respondent is experiencing any difficulties.

- Telephone interviewers may face resistance from people who are tired of answering this type of call or who are
  suspicious because of their experiences with telephone soliciting.
- Ensuring that the respondent is provided with clear information on the credibility of the group doing the survey, the
   purpose of the survey and how the collected information will be used may increase the response rate.
- Finding a convenient time for the respondent to answer the survey questions may increase the response rate.

Reaction sheet

• simple kind of questionnaire that asks questions about people's satisfaction with a particular activity
• easy and fast to administer and summarize
• useful tool for getting an immediate response to new resource materials, workshop models and public education events.

- Avoid using leading questions that prompt positive responses. Instead of asking, "Did you enjoy the workshop?" ask,
  "Did the workshop provide you with enough information to answer your questions about health promotion?"
- Limit the number of questions to increase the response rate.
- Include open-ended questions to obtain qualitative data. Shape these questions carefully to control the amount of
  material received. Examples: "Give three words to describe your reaction to this workshop." "What were the two key   learnings for you from this workshop?" Face-to-face interview

• individual interviews structured around a set of open-ended questions that are developed to guide the interview and to provide consistency in the information collected
• useful method for getting in-depth information on project activities
• provides an opportunity to clarify responses and probe for further information

- This tool can be used with a specific group of people (e.g., project staff to gather their opinions about the strengths and weaknesses
  of the project) or with key informants who are knowledgeable about the project (e.g., frontline service providers about how best to
  conduct outreach within their community).
- It is a good method to use with respondents who have low literacy levels and might be uncomfortable with written data collection
  tools.
- The interviewer needs to be trained not to bias the responses through the use of leading questions.

Telephone interview

• similar process and function as face-to-face interview but conducted by phone
• less expensive to administer than face-to-face interviews.

- Sending the respondent a copy of the interview guide in advance may promote a more thoughtful discussion.
- Interviews, both in person and by phone, are an alternative to focus groups when you want to avoid group influences
  on the responses people give.

Focus group

• group discussion in which 10 to 12 people are brought together in a single session of approximately an hour to generate ideas and suggest strategies
• facilitated using a specific agenda of structured questions, similar to the interview guide, that focuses the discussion in the meeting
• used to obtain in-depth understanding of attitudes, behaviour, impressions and insights (qualitative data) on a variety of issues from a group of people, e.g., project staff or a project advisory committee.

- The facilitator must remain neutral and non-judgmental and have the skills to keep the discussion moving and on track.
- This is a particularly useful method for reflecting on evaluation findings and identifying key learnings. It may also be useful for
   developing preliminary ideas for new programs or for testing messages that will be used in educational and media packages.
- It is not a useful method for developing consensus or making final decisions.

Participant - observation

• involves actual observation rather than asking questions
• used to better understand behaviours, the social context in which they arise and the meanings that individuals attach to them
• observers compile field notes describing what they observe; the analysis focuses on what happened and why.

- This may be the most feasible way to collect data from some hard-to-reach populations (e.g., individuals who frequent public sex
  environments or drug shooting galleries).
- As with all qualitative techniques, the results may not be fully generalizable to the entire study population.

Project diary

• project managers, staff or participants are asked to keep a record of their experiences while working on the project
• provides qualitative evaluation data.

- It is important to provide the participants with clear guidelines on keeping a log book: the type of information you are looking for,
  how it will be used, confidentiality, etc.
- This is a useful method for identifying unintended consequences of a project.
- Some people are very uncomfortable with this method because of the unstructured nature of the writing required.

Program documentation

• analysis of written records (minutes of meetings, telephone logs, intake forms, policy directives, financial records, attendance records)
• can provide information on people's interests, preferences and patterns of usage of services and service locations
• can often, through systematic review, provide important evaluation information, both quantitative and qualitative
• inexpensive source of information.

- This tool is limited in that records document only existing alternatives, they don't show other needs, wants or preferences.
- It is important to identify evaluation information needs at the beginning of a project to ensure that the necessary records are kept
  throughout the project.

Non-traditional methods of documentation

• non-verbal or non-written evaluation tools used to respect diversity and accessibility issues
• examples include cartooning, drawing, poster making, photography, videotaping, audio taping, scrapbooks.

- Qualitative data collected may be difficult to analyse and generalize.
- This is a useful method for getting responses from respondents who are uncomfortable with written tools.

No single evaluation tool can provide all the evaluation information required. A combination of different tools that suit the project needs and available resources has to be developed. Regardless of which tools are selected, they should reflect the following tips to be effective.
 

Tips for designing effective evaluation tools Keep them short and simple. Use plain language with no jargon. For tools requiring written responses - use large print - avoid clutter - leave lots of white space - provide ample room for responses. Ask for key works and key learnings. Develop evaluation tools in collaboration with the people who will use them. Ask only for information that will be used.

6.3     Sample evaluation tools

Sample evaluation tools are provided in the appendices at the end of this guide. They were developed for use in community-based projects and are included here to give project sponsors some ideas, which they can adapt and build on to develop their own project-specific tools. The questions in each of the sample forms have been shaped specifically to provide data to answer the five key evaluation questions.

Sample evaluation forms provided:
 

Appendix 6.	Reaction sheet for evaluation workshop
Appendix 7.	Guided telephone interview - Community Resource Handbook for Women with Breast Cancer
Appendix 8.	Focus group interview guide - Child Safety Awareness Program
Appendix 9.	Guidelines for keeping a project diary - Child Safety Awareness Program
Appendix 10.	Mail-out questionnaire -Advisory Committee, Health and Disabled Women's Project
Appendix 11.	Mail-out questionnaire - Health Care Providers, Health and Disabled Women's Project

7     ANALYSING AND INTERPRETING DATA

The emphasis throughout this guide is on evaluation for learning and action. This section focuses on practical ways that people at the national, regional and community levels can turn evaluation information into usable, accessible summaries and reports that add to the body of knowledge about project success and promote change in attitudes, skills and behaviour. Committing adequate resources at all levels to do the evaluation work is essential if everyone is to benefit from the valuable learnings that can be gained from evaluating health promotion projects.

7.1     Analysing project evaluation information

Analysing evaluation information begins with a review of all the collected data to find the emerging themes or patterns. The five key evaluation questions provide useful categories around which to group information and develop the themes. Look for and record the information that is in the data about how well the project is doing, what is working, what should be done differently and what difference it is making.

Project sponsors may want to record notes on the data on file cards or sheets of paper - one for each question, issue or topic. This makes it possible to see the emerging patterns more easily. Include exact quotations from the interviews and questionnaires. It is essential to stay with what people have said and let the data guide the analysis. Too much detail is better at this stage than not enough. It is always easier to cut down than to add information later.

Once the material has been grouped into themes, it can be analysed to see how the results compare to the changes that were expected as identified by the success indicators. Take the time to reflect on what the analysis reveals. What was learned to answer the "what", "why", "so what", "now what" and "then what" evaluation questions? People who have been involved in the project should be involved in the interpretation of the findings.

Project sponsors or the project evaluator should prepare short summaries of the key learnings from the analysis on a regular basis - for example, every three months or after each project activity. The importance of preparing these brief summaries, which highlight two or three key learnings, cannot be overemphasized. The summaries provide an excellent means of letting the key players in the project know about and begin to use the evaluation findings throughout the project - one of the basic principles of participatory evaluation. By completing summaries of key learnings at regular intervals, the work at the end of the project will be greatly reduced.
 

Summary - Analysing evaluation information Review the collected evaluation material for emerging themes and patterns. Use the key evaluation questions to group the material into themes. Analyse the material by themes, comparing the results to the changes that were expected as identified by the success indicators. Reflect on what the analysis means. Ask other key project players for their interpretations. Prepare short summaries of key learnings under each theme. Prepare summaries of key learnings on an ongoing basis. Submit the summaries to the participants for their feedback and verification of the findings. Develop the final analysis.

Analysis of quantitative data

Quantitative data looks at the incidence and quantity of events. Data gathered through quantitative methods (surveys, questionnaires, administrative records) is numerical and may be analysed by calculating averages, ranges, percentages and proportions. Descriptive statistics simply account for what is happening in numerical terms. For example, when evaluating the use of a needle exchange system, an estimate may be made of the average number of people using the facility each week or the percentage of users returning needles. Bar charts, pie charts, graphs and tables can be effective ways to present the statistical analysis in a clear and concise manner.

Analysis of qualitative data

Qualitative data is information that is primarily expressed in terms of themes, ideas, events, personalities, histories, etc. Data is gathered through methods of observation, interviewing and document analysis. These results cannot be measured exactly, but must be interpreted and organized into themes or categories. The primary purpose of qualitative data is to provide information to the people involved in the project. This standard of usefulness is an important one to keep in mind when analysing qualitative data.

Note: Neither the quantitative nor the qualitative approach to the collection and analysis of data is inherently superior. Each has advantages and disadvantages. For both, it is important to know the context within which they have been used in order to understand the analysis. Whenever possible, project evaluations should include several types of information collection tools. The analysis and summaries of key learnings should draw on information collected from all of them.

7.2     Preparing useful evaluation reports

Once the evaluation information has been analysed, the next challenge is to present the learnings in ways that are both informative and interesting.

The brief summaries of key leamings, described in the preceding section, are often all that is needed to provide information on an interim basis. However, the final project report requires more data. The next section provides some ideas that might be useful for clarifying the expectations about the final report with project sponsors.

Evaluation report outline

Having an outline at the beginning of a project about how the final report will be developed is extremely useful. It helps shape the thinking about what information is needed and how it will be collected, analysed and used.

There are two questions to consider when planning evaluation reports.

• Who is writing the report?

Small projects with very limited resources should have different expectations placed on them than larger projects or projects with funding for an external evaluator.
 
• Who is the report for?

The following sections form the basic structure - the bare bones - of an evaluation report. Personal stories and quotations from the project participants put a human face on the evaluation results and can make the report much more interesting and user-friendly. Groups can adapt and build on the following guidelines to develop evaluation reports that reflect the unique nature of specific projects.

Example of an outline for a project evaluation report
 

Section 1:	Executive Summary This section is for people who are too busy to read the whole report. - One page is best - never more than three. - It comes first but is the last piece written. - It usually looks at what was evaluated and why and lists the major conclusions and recommendations.
Section 2:	Background Information - Getting started This section provides background leading up to the evaluation: - how the project was conceived - why it was needed - the project goals and objectives - who was involved in the work - the project organizational structures.
Section 3:	Description of the Evaluation - How we learned This section describes - the evaluation approach and how it was chosen - evaluation goals and objectives - how the evaluator was selected and managed - how the data collection tools were designed and used - how well the data collection tools worked - any limitations of the methodology - how people were selected to be interviewed, or to receive questionnaires, etc. - who did the interviewing, the number of people interviewed and their situation - how questionnaires were distributed and returned.
Section 4:	Evaluation Results - What we learned One way to organize this section is around the first four evaluation questions: Did we do what we said we would do? - Outline goals and objectives of the project. - Record what happened as a result of the project - e.g., resources developed, training sessions completed, etc. - Describe the changes that occurred in relation to the success indicators.   What did we learn about what worked and what didn't work? - Outline key learnings from the project about making things work. Examples: producing effective resource    materials, structuring productive advisory committees, conducting needs assessments in rural and isolated    communities, building community ownership of health promotion projects, etc. - Identify learnings about what strategies didn't work and why.   What difference did it make that we did this work? (outcomes) - Outline results from the evaluation that show how the project made a difference to consumers, project sponsors    and the wider community. - Identify any changes - of attitudes, knowledge, skills or behaviour - that occurred from the project work, e.g.,    how health practices have improved. - If appropriate, show how the project contributed to increased public participation and strengthened community    groups. - Include personal statements and anecdotal material from project evaluations which illustrate the impact an activity    has had on project participants. Example: "One thing I plan to use right away in my work which I got from the    training is..."   What could we do differently? - List leanings from the projects about different ways to do the work. Examples: improving the cost-effectiveness    of projects, adapting the project model to make it more responsive to volunteers, changing the reporting role for    outside evaluators to improve accountability, etc. - Reflect on cautions and challenges about doing the project work.
Section 5:	Conclusions and Recommendations Final thoughts on what we would like others to know - Conclude with a summary of the work done and how well the goals and objectives were reached. - Include recommendations for further work. - Include recommendations on how the evaluation results can be used.
Section 6:	Appendices - These may include copies of questionnaires or interview schedules, statistical information, program   documents or other reference material important to the evaluation but not important enough to go into the   text. - It is useful to include a bibliography - list of the sources used to compile the evaluation results, other   research studies and articles. A list of who was interviewed or organizations contacted may also be   included.

7.3     Activity: Analysing and Interpreting Data
 

Topic:	Analysing and interpreting data from project evaluations.
Purpose:	To give project sponsors practice in completing the analysis and interpretation of project evaluation results for inclusion in the project evaluation report.
Time:	1-2 hours
Materials:	flipchart raw project evaluation data Guide to Project Evaluation, Chapter 7
Activity:	Have participants review the raw project data  Working in small groups, divide the raw data among groups and have participants analyse the data to find themes (refer to Chapter 7 of the Guide) that relate to the third evaluation question: "What difference did it make that we did this work?"    Have participants prepare short summaries of learnings for each theme   Have participants develop one example of quantitative analysis and one example of qualitative analysis   Bring all participants together to share their results and to discuss their ideas on which information is most useful   Have the total group develop a final analysis plan based on the information presented by the small groups   Have the total group brainstorm on informative and interesting ways to present the evaluation results, including how to organize the evaluation report for maximum effect

8    USING EVALUATION RESULTS

The fifth and final key evaluation question in the framework is, "How do we plan to use evaluation findings for continuous learning?" This is a question that needs to be considered at the very beginning of a project and not only at the end, as is often the case. Having ideas at the start of a project about uses for the evaluation findings helps ensure that the evaluation is conducted and the results reported in a way that meets people's needs. If key stakeholders are involved from the beginning, it increases their support for the process and their likelihood of using the results as they become available.

There are several major ways in which project evaluations can be used to maximize their benefit, A few ideas are listed below.

8.1     Using evaluation results

• Bring together all project staff on a quarterly basis to discuss the evaluation results and look at ways the results can be used to increase performance, improve project administration, enhance planning activities, etc.

• Present the report orally to staff, funders and community members.

• Develop a news release outlining the main leanings from the evaluation and some of its more important conclusions. Send the news releases to key community contacts and evaluation participants.

 
• Involve project participants in developing ways to present the project findings. Build on their stories and personal experiences to give a human face and to create interest in the evaluation results.

 
• Make a presentation on the evaluation results to the local health council or social planning group, highlighting the accomplishments and describing how the results can be used to promote better planning.

 
• Use the evaluation results to shape requests for new or continued funding or for suggesting alternative health practice models.

 
• At the start of each new project proposal process, review evaluations from past projects to discover what learnings are transferable.

 
• Send a letter thanking all project participants for their work on the project and include a summary copy of the key evaluation results.

 
• Develop a short video of project participants discussing what they learned from the project. Use it to promote the project with community groups and with funders.

 
• Build the evaluation results into presentations to local service clubs to show how their funding support could be effectively used.

 
• Commit 15 minutes of time at meetings to information sharing about key leanings from project evaluations.

 
• Extract highlights of project evaluation reports and distribute them regularly.

 
• Develop a workshop to present the project evaluation results at a regional or national conference of health promotion professionals.

 
• Identify other projects that are doing related work and share evaluation reports with them.

 
• Organize a brainstorming session involving staff to come up with creative ideas to document and promote project successes.

 
• Develop a user-friendly yearly summary of key evaluation results from across projects. Include ideas for using the results to strengthen planning and distribute the summary to key stakeholders.

 
• Make presentations to other health care practitioners, using project evaluation results to show how they can benefit from involvement in health promotion work.

 
• Systematically review and summarize all project evaluation results on a twice-yearly basis. Use the evidence-based outcomes to develop and improve health practice models.

This section provides a checklist to use throughout the project evaluation process.

Developing the project evaluation

€ Have you used the principles of participatory evaluation?

€ Have you reviewed the evaluation framework and example worksheet?

€ Have you identified the evaluation resources required to plan and carry out the evaluation?

€ Have you discussed the roles and responsibilities of those involved in the evaluation?

Reviewing the project evaluation plan

€ Are the project goals clear and realistic? Are the project objectives specific and measurable?

€ Are the project goals consistent with the overall goals of the funding program?

€ Is an evaluation framework prepared and included in the plan?

€ Does the evaluation proposal demonstrate a process that will provide information to answer the five key evaluation questions?

€ Does the evaluation proposal demonstrate a participatory process that includes others, eg., target group members?

€ Are the success indicators for the project identified in clear, measurable terms?

€ Is there a practical outline of how evaluation information will be collected and from whom?

€ Does the proposal give ideas on how the evaluation results will be used both throughout the project and at the end?

Assisting and monitoring project evaluation work

€ Is the project sponsor regularly informed of evaluation findings?

€ Have the roles and responsibilities for reporting purposes been negotiated for the project sponsor and the outside evaluator?

€ Does the final evaluation report address all five evaluation questions?

Using project evaluation results

€ Is there a plan in place for identifying different ways to share evaluation information?

€ Are project evaluation results being used to contribute to future project planning?

•  evaluation - a way of measuring if a project is doing what it says it will do.

• goals - general statements of what an organization is trying to do.

• objectives - specific, measurable statements of what an organization wants to accomplish by a given point in time.

• objective approach - an approach which values the perspective, views and opinions of those outside of or distanced from the situation, event, organization, project, etc., as the primary basis for making an assessment or judgment.

• informant - in research and evaluation terminology, the person you inter-view or question is called the "informant".

• impact or outcome evaluation -gathers information related to the anticipated results, or changes in participants, to determine if these did indeed occur. It may also be used to test the effectiveness of a new program relative to the results of an existing form of service. An impact evaluation will tell you about the effects of a project.

•  process or formative evaluation -an ongoing dynamic process where information is added continuously (typically using a qualitative approach), organized systematically and analysed periodically during the evaluation period. A process evaluation will tell you how the project is operating.

 
• quantitative approach - an approach that tries to determine cause and effect relationships in a program. A quantitative approach will use measurements, numbers and statistics to compare program results. The information that is found is considered "hard" data.

This section contains a list of selected resources which you may find useful in your work with participatory evaluations.

• Cultural Competence for Evaluators: A Guide for Alcohol and Other Drug Abuse Prevention Practitioners Working with Ethnic/Racial Communities 

  Mario A. Orlandi (Editor) (1992)

This document contains 10 insightful articles that help evaluators understand the cultural ,factors involved when working in diverse ethnic communities.
 
Available in English from:

Division of Community Prevention and Training, Office for Substance Abuse Prevention,
Alcohol, Drug Abuse, and Mental Health Administration, Public Health Service,
U.S. Department of Health and Human Services, 5600 Fishers Lane, Rockwall 11,
Rockville, Maryland 20857 U.S.A.
 

•  Doing it Right: A Needs Assessment Workbook

Edmonton Social Planning Council (1988)
 

Comprised mainly of activity checklists and work sheets, this workbook should be useful to agencies conducting needs assessments to develop new programs or to evaluate current ones.
  Available in English from:

Edmonton Social Planning Council,
#41, 9912 106 St.,
Edmonton, Alta. T5K 1C5
 

• Making a Difference: Program Evaluation for Health Promotion

 

  Tammy Home (1995)

 
The purpose of this guide is to present basic principles and processes of program evaluation in the context of health promotion. Cost is $21.95 plus shipping and handling.
 
Available in English from:

WellQuest Consulting Ltd.
11521-125 St.
Edmonton, Alta. T5M 0N3
Tel.: (403) 451-6145 Fax: (403) 451-5280

Success Indicators of Increased Public Participation and Strengthened Community Groups

Listed below are additional examples of indicators of success for two health promotion program/project impacts: increased public participation and strengthened community groups.For each impact, sample indicators of success are given. Below the indicators are the types of questions project staff can ask themselves in order to determine these indicators of success.

Remember, setting indicators of success presumes that first you have determined your benchmarks. As described in Chapter 5, section 5. 1, benchmarks are the status of your targetgroup's knowledge, attitudes, behaviour or skills before project work begins.

Indicators of increased public participation:

• The people who experience the health or social issue the project addresses are involved in making decisions about the project.

• The project reaches the consumers it wants to reach.

• Skills and knowledge are transferred from individuals to the community. (e.g., increased sensitivity, interpersonal skills)

• Social support networks are expanded.

• Consumers are involved in every aspect of the project, from planning to evaluation.

• What different groups in our community did we work with? What new groups did we work with? What did we do together?

• In what ways will the work started during the project be continued after the project ends?

• To what extent and in what ways did the image of our group change in the community?

 
•  What experiences did our group have in affecting community decisions to improve the conditions that affect the health of our community? How will our ability to influence decision making be continued?

1.     How useful was this training for you?
 

not useful

so-so

useful

very use

Comments:
 

2.     What 3 words would you use to describe the training?

4.     What suggestions do you have to make the training more useful?
 

5.     What comments would you like to make about the trainers?
 

6. What is one thing that you got from the training that you could use right away in your work?

Community Handbook on Resources for Women with Breast Cancer

• How did you hear about the Community Handbook on Resources for Women withBreast Cancer ? (health professional, friend, advertising poster, other.)

• Which sections of the handbook were most helpful to you?

 
•  How were they helpful?

•  What contributed to the success of the child safety awareness activities in your community?

• What are the barriers you experienced when you implemented child safety awareness programs in your community?
- timing
- money
- human resources
- attitudes
- culture
- politics

We are asking a few people in each of the five pilot communities to keep a project diary during this next year. The diary is your own record of your experience of using the resource materials to do child safety awareness work in your community. There is no right or wrong kind of information to collect. You may decide to note thoughts that you have, ideas that come to you about what you think would work better, anecdotes that actually describe what you did and how you felt about what happened. All of this information will help us put the human side and your unique community experience into the evaluation. Everything we learn from the collective !earnings will be identified and shared with you. Anonymity of individuals will be preserved.

To help us put some structure to the analysis of the material and provide some consistency across all of the communities, we will be using the information in your project diary to answer the following questions:

1.     What difference did each of the following elements of the project make to your community's experience of carrying out child safety awareness activities?
        (a) money (b) project sponsor's presence in the community (c) resource materials (d) program consultant.

2.     What are the major challenges to getting awareness activities going and to keeping them going in your community?

3.     Were there any ways in which the information in the resource material provided help to individuals, coordinating and planning groups and agencies in the
        communities to address the challenges? Describe.

4.     What are you learning about planning and implementing child safety awareness aimed at changing beliefs, attitudes and behaviours?

5.     What suggestions do you have about making the resource materials more accessible and useful?

6.     How do community projects keep volunteers interested, committed and actively involved in child safety awareness work? Who gets involved?

Remember - you just jot down experiences and thoughts you have through the year that relate to your involvement with the child safety awareness project. You do not have to directly answer the above questions. Building collective answers to these questions is our job. Keeping the diary should be enjoyable, not onerous, so do what works for you. Thanks.

The terms of reference for the advisory committee are to:

• hold two meetings a year in Toronto

 
• participate in conference calls

 
•  provide input on project issues as they develop

 
• assist in the development of health brochures and other literature

 
• assist in creating a workplan and strategy to establish working groups in communities outside Toronto who will organize local workshops and will form the framework for DAWN Ontario.

2.     Which task(s) did the advisory committee do well? Why?

3.    What helped you personally to contribute to the work of the committee?

4.     What got in the way of your contributing to the work of the committee?

5.     What impact did being on the committee have on

7.     What suggestions can you give DAWN Ontario for structuring future advisory committees?

8.     What recommendations do you have for making advisory committees in the health care system more accessible to women with disabilities?

1.     How familiar are you with the Health and Disabled Women's Project?
 

Not familiar

So-so

Very familiar

2.     Please identify the type(s) of contact you had with the Health Project.
 

a.	attended a workshop in which Health Project staff participated
b.	reviewed education material developed during the project
c.	attended the symposium at Geneva Park in 1992
d.	requested and received information
e.	responded to the initial questionnaire on accessibility and services in 1991

f. other

3.     As a result of your contact with the DAWN Health Project did you:
 

		No	Yes
a.	increase your own awareness of health issues as experienced by women with disabilities (WWD)?
b.	change any of your own assumptions or practices with WWD?
c.	increase your own interest in providing appropriate care to WWD?
d.	become involved in increasing the accessibility of your services or facilities to better accommodate WWD

Comments :

4. How would you rate the following education tools developed through the project?
 

	Not useful	So-So	Very Useful
a.     The Education Brochures WWD talk about sexuality   You and your doctor   Mothering and WWD   Guide for Health Care Professionals
b.   The Access checklist
c.   Les actes du Colloque

Comments :

5. What kinds of activities would you consider doing in the future to address health care issues of WWD? For example, would you:
 

		No	Yes	Maybe
a.	facilitate a workshop with a disabled woman?
b.	talk to other professionals about the health care needs of WWD?
c.	distribute the education materials mentioned above?
d.	advocate for an accessibility audit to be done on your facility by a barrier-free consultant?
e.	refer a WWD to DAWN Ontario for support and information?

f. other

6. Based on your experience with the Health Care Project, what advice would you give other health care providers about addressing the needs of women with disabilities?

7. DAWN Ontario wants to be a more active partner in the process of defining andaddressing health concerns of disabled women. What suggestions do you have for makingthis happen in your agency/program?

8. What is your title? Briefly describe the kind of work you do.

Thank you for using this evaluation guide! Your comments and suggestions can help us make sure the guide is kept current and useful. If you have a moment, please answer and return the following questionnaire. Respondents are eligible to receive future updates to the guide - please complete Question 5. If you are interested in receiving future updates.

4.     Do you have any other comments or suggestions to Improve the guide?

        Yes  ___ No     ___

        If so, please answer Question 6.

6.     The following information is optional (unless you answered "yes" to Question 5.). These details will help us identify the types of organizations that most
        commonly use the guide and what their specific needs are.

National Program Consultant
Evaluation and Coordination
Health Canada
Population Health Directorate
st Floor, Finance Building
Address Locator 0201C
Ottawa, Ontario K1A 1B4
Fax: (613) 957-1565