Deciding the Type of Information Upon Which Your Evaluation Will Be Based 33
Crime prevention is, typically, only a small part of the solution to larger and more complex social problems. The approach of many First Nations communities is to strategise and plan in
an holistic way which incorporates strategies such as inter-generational participation and broad community responsibility.
With the changing dynamics of policing, community safety and security combined with limited resources, there has been a parallel need to review crime prevention practices. This manual
has been developed to facilitate the evaluation of crime prevention programs in First Nations communities.
With geographic diversity and differing levels of formal policing, a broad range of people will be involved in the delivery and evaluation of these programs. This manual has been
developed as a result of a contribution agreement with the Ministry of the Solicitor General of Canada, Aboriginal Policing Directorate for use by anyone in First Nations communities,
including police, who may be given responsibility for the evaluation of crime prevention initiatives.
While the manual provides a step by step approach to the evaluation of crime prevention programs, the manual must be only one of the tools used to establish, maintain and review
programs. Programs, and the manuals designed for their evaluation, will work most effectively when the community and community agencies are engaged in a co-ordinated approach to community
safety and security.
Who Is This Manual For?
What is Evaluation?
What Should be Evaluated?
Why Should We Evaluate?
When Should We Evaluate?
Who Should Do the Evaluation?
As policing and crime prevention become more community oriented on the one hand and as the community becomes more involved in community safety and security on the other, there is
increasing likelihood that non-police personnel will be actively engaged in the implementation and evaluation of programs.
This manual is intended as a practical workbook for those who have assumed responsibility for the evaluation of a crime prevention program. The evaluators are likely to be program
personnel, Band Council staff or community members. The evaluator may also, of course, be a member of a police agency. It is assumed that the person with the responsibility for the
evaluation is not necessarily well versed in the intricacies of evaluation methodology but wishes to assess the success of crime prevention program operating within their community.
The manual is designed for communities with limited access to the sophisticated hardware, software and personnel often available to communities in urban jurisdictions. Accordingly, the
manual does not address such things as the use of specialised computer software or sophisticated statistical analysis. It is assumed that the evaluator requires a basic understanding of
program review and that a more in-depth knowledge can be acquired through specialist assistance.
The manual is user friendly and focuses on applied evaluation. It is not a theoretical treatise. The terms used are practical and a glossary of terms can be found in the appendix.
Evaluation is a way of checking to see whether a program is doing what it was intended to do. Evaluation is an assessment of whether a program is meeting its objectives and, ideally, to
what extent and why (or why not) the goals have been achieved. How successful has the program been?
Evaluation is merely one stage of the process of addressing a community crime problem. Evaluation of a program is an integral part of the process of program development and
maintenance designed to ensure that the community response to a safety or security issue works in the most effective and efficient manner.
Ideally, the implementation and evaluation of a program are integral parts of a planned process. The principal stages of a crime prevention program are:
- Realisation - the recognition that there is a safety or security problem in a community and that some form of action should be taken.
- Action plan - the formulation of a plan which will lay out a series of actions to address the issue and assign responsibility for the each of the actions to ensure realisation of
the project goal.
- Implementation - Undertaking each of the tasks in the action plan in sequence after securing the necessary funding and ensuring sufficient resources to sustain the program for
the duration of the plan.
- Monitoring - The gathering of quantitative and qualitative data over the duration of the program such that, periodically, an assessment of progress can be undertaken.
- Evaluation - The assessment of information at appropriate points in the program to gauge its success in meeting the program goals. As we will see later, the review can be a
‘process’ or ‘impact’ evaluation. Process evaluation refers to the assessment of each stage of the program to ensure that the action plan is being carried out
appropriately. The Impact evaluation assesses the degree of change which has occurred as a result of the program.
- Feedback - Provides for the adjustment of the program or subsequent offshoots of the program to accommodate the findings of the evaluation. There is little point in introducing
the program or undertaking the evaluation if the findings are not considered as a basis for further action in the community.
Identification - the definition of the problem based upon statistics, interviews, observation or other methods so that the community is provided a focus for action.
Evaluation, therefore, is a planned and important part of a crime prevention program. The nature and form of the evaluation must be anticipated at the program planning stage and should
not be considered an ‘add-on’ as the term of the program comes to an end. However, in reality, many initiatives are introduced with limited planning regarding the mechanics of
an evaluation and, frequently, with little concept of the need for an assessment of the effectiveness of the program.
Crime prevention takes many forms. At one end of the spectrum, there are long term programs which are developed to target one aspect of complex community problems such as youth suicide,
racial violence or child abuse. These programs are often multi-agency funded and supported by federal or provincial policy or legislation. As crime prevention practitioners, you may be
responsible for the implementation of one part of these programs in your community.
At the other end of the scale are the local initiatives which may be individual initiatives on the part of police officers to resolve a neighbourhood, a family or even a personal issue
for a citizen.
These initiatives can all contribute to the prevention of crime. Should one evaluate all these initiatives? All these crime prevention efforts take time and, therefore, money. They
will all have some impact upon the subject and upon the resources of the community. They should all, therefore, be subject to some form of “assessment” to determine whether
the time and effort is worthwhile. Evaluation projects also take time and cost money. The level of sophistication of the evaluation depends upon the amount of investment expended on the
initiative.
A full, complex evaluation to determine whether an officer should counsel a family on a one time basis is obviously not worthwhile. However, even the most minor of initiatives should
be subject to some form of assessment even if this is a “mental tally” by the officer or the supervisor of why this action is being done, the intended benefit of the action,
how will the officer gauge the success, and whether the initiative should be tried on another occasion. This is sort of an ‘intuitive’ assessment. As we shall see later,
these steps are the foundation for an evaluation. These same principles can be applied to any crime prevention program.
So, what should be evaluated? As we have noted, every crime prevention initiative should be evaluated. However, it is obvious that not every initiative should be subject to a
comprehensive evaluation. The amount of energy directed to the evaluation should, roughly, be in direct proportion to,
- the scope of the program (how many people are ‘touched’ by the program)
- the longevity of the program (if a long term investment is anticipated, you should assure that effort is not fruitless)
- the potential community impact of the program (the politics),
- any contractual requirements to evaluate, and
- cost (time, dollars and effort) of the evaluation.
the breadth of effort of the program (how many people are devoted to operating the program)
This manual lays out the principles and steps to be taken in the evaluation of any crime prevention program. The crime prevention practitioner needs to gauge the intended, or likely,
overall impact of the program on resources, time, or the community to determine to what depth the evaluation should be pursued.
The content of the manual then needs to be selectively “geared back” and the methods and effort in the evaluation tailored to the significance of your crime prevention
program. Following the principles outlined, considering each step and deciding whether it is relevant to your program will ensure that whatever the dimensions of the program, you will have
subjected it to a valid review.
Crime prevention practitioners and members of community groups involved in prevention initiatives frequently feel that a program has worked. Intuitively they may know that incidents of
crime are reduced or that the events which gave rise to community concern have been resolved. However, in times of limited resources stakeholders have a right to know whether the
initiative actually has worked, what impact it had upon the problem which had been identified and even if it was the specific program that had the impact or whether it was some apparently
unrelated action. The evaluation also lays the foundation for planning.
Every evaluation is important in that it can help to identify the strengths and weaknesses of a program and help the community or agency to plan strategies. There are several reasons for
demonstrating whether and how the initiatives produced the promised outcomes:
- Most initiatives involve considerable expenditure of money or time. Are these resources well spent or should they be diverted to other strategies?
- Information gleaned from evaluations should be shared with other organisations as a basis for (i) the introduction of the program in another community, (ii) the decision not to
introduce the program in another community or (iii) to adjust the structure of the program to meet local conditions in other locales.
- The community needs to know whether its resources are being used in the most effective way.
- The implementation of a new program may create a sense of security in the community. Is this assumption is well founded?
- People who are involved in programs as victims or offenders need to know that the initiative appropriately addresses their needs.
- To identify or justify the next courses of action in an on-going program.
- There may be many reasons for the success or failure of a program, such as inappropriate training, failure to support the desired behaviours or contradictory policies. These factors
should be identified to ensure that the correct conclusions are drawn and that subsequent actions, such as renewed funding, are directed to the appropriate actions.
- To motivate staff by highlighting successes.
Many programs are funded by an outside resource, contractual requirements often require an evaluation to ensure that such funds are well spent.
We have already noted that evaluation is an integral part of any program. The planning for an evaluation should be undertaken during the planning stage of the crime prevention program.
Evaluation can be time consuming and expensive. If an evaluation does not address the relevant factors then the effort is futile. Likewise, if the program is not evaluated then the time
and effort invested in the initiative also may be wasted.
The planning of the evaluation at the outset of the initiative is particularly important because the evaluation may require certain information which must be built into program
planning. For example, data (or information) might have to be gathered through the use of a special form or, to permit an effective evaluation, minor adjustments may have to be made to
the operational practices or program design. For example, for comparison purposes, different approaches to the crime problem may have to be taken in within the community or in two
different communities or we may have to ask the members of a Victim Services Unit to gather information on the presence of alcohol at domestic incidents or when children are taken into
custody.
Where a process evaluation is undertaken, each stage of the program must be assessed as it is undertaken to ensure that the program is introduced in the correct way according to the
project plan. Therefore, a process evaluation must ideally be ready to introduce as soon as the initiative begins.
The value of a process evaluation is that it allows the evaluator (or the program co-ordinator) to make adjustments to the program to ensure that it is ‘on course’. For
example, staff may have to receive additional training or the types of incidents attended by a Victim Services Unit may have to be expended or reduced. The adjustments which are made
however, should not alter the program process to such an extent that the original objectives are no longer valid.
An impact evaluation assesses whether the program achieved its desired objectives. By definition, it must therefore be conducted sometime after the program has, in a substantive way,
been completed. Sufficient, planned time must have elapsed to allow the program adequate time to take effect. The exact timing of the impact evaluation is dependent upon the nature of
the program.
The Impact evaluation, therefore, presents a few challenges. The methodology must be anticipated based upon a clear understanding of the objectives of the program and the anticipated
outcomes. In addition, the evaluator must assess the potential problems arising from the implementation and the conduct of the program.
Nevertheless, the Impact evaluation is crucial in determining the success of the crime prevention program. We will examine these issues later.
This is intended to be a self evaluation manual. It implies, therefore, that the evaluation is undertaken, or co-ordinated, by a stakeholder from within the community.
The evaluator could, therefore, be,
- individuals working within the program,
- a citizen from the community, or
- someone from another nearby community
a person within the agency who does not work in the program,
Alternatively, the evaluation can be taken with assistance from someone from outside the immediate community but the project should be co-ordinated by a community representative or
group.
Evaluation methodologies have become very complex. Sophisticated statistical analysis is a requirement for publication in many academic publications. Few organisations, however, have
access to either the personnel or the software or hardware to undertake an evaluation of this level.
On occasion, the staff or students from a local college or university may be recruited to undertake an evaluation as part of a course. Involvement in community programs provides
academic institutions with the opportunity to give students access to practical fieldwork.
Likewise, external consultants may be engaged to evaluate programs. Such support is likely to cost money in the form of consulting fees and, in remote communities, would also incur
significant travel costs. If external experts are used as evaluators, project managers should ensure that knowledge transfer is assured to equip local staff or residents with the skills
to undertake future evaluation projects.
Employees of community agencies, such as police officers, may be part of a larger organisation which can provide research support. This is particularly likely if the community is
selected as the site for a pilot program which may be considered for wider use.
Peer evaluation allows for persons from other similar agencies or communities to undertake the evaluation. The peer evaluation may be undertaken on a reciprocal basis with a member of
your community later participating in the evaluation of the programs of the other agency or community.
In most situations, however, it will fall to agency workers or community members to plan, develop, implement and evaluate a crime prevention program. As distance education becomes
more common, many citizens and agency personnel are now well versed in program development and evaluation.
Whoever is ultimately responsible for the evaluation, the involvement of several people as a ‘sounding board’ for ideas and to provide input to the evaluation strategies will
ensure a more comprehensive product. The inclusion of Elders, women, youth or Band Council members on a steering committee will ensure a more comprehensive perspective on the evaluation
process and program.
The involvement of a co-ordinating group will also permit the sharing of responsibilities for various aspects of the evaluation. Overall responsibility must remain with one person who
will orchestrate the activities.
Although, ideally, a member of the community should undertake the evaluation, there are several criteria to consider when selecting a person to undertake the planning and evaluation of
a crime prevention program:
- The person(s) should be as objective as possible. The person should not be directly involved in the program as a victim or offender.
- Where confidentiality of data is important, as far as possible, the person should have nothing in his/her background which may appear to jeopardise the integrity of the data.
- Although difficult in smaller communities, the evaluator should not have friends or relatives who are involved in the program as victim, offender or agency worker. The perceived
confidentiality of data may be compromised and some program participants may be reluctant to share their opinions.
- The person should be seen to be objective rather than have a strong view on the important pro’s and con’s of the program.
- Where possible, the person should have some understanding of program planning, evaluation and statistics. Hopefully, this manual will provide some background in these activities
The person should not have strong vested interest in the outcome of the program. For example, the evaluators continued employment should not be dependent upon the reported success and
continuation of the program.
The peer evaluator must be,
- capable of undertaking a valid review,
- prepared to be open and objective regarding the findings and willing to share the results with the community.
prepared to keep confidential information obtained in the evaluation, and
However, given factors such as geographic isolation and funding limitations, it is more important that the crime prevention programs are initiated and subject to some form of evaluation,
rather than preventive programs not being introduced because of limited opportunity for a sophisticated assessment of effectiveness and efficiency.
If funding has been secured from an outside agency, there will frequently be a requirement that an evaluation component is built into the program. Consequently, sufficient funds should
be available for an evaluation if the program planners have developed the program budget appropriately.
Occasionally, evaluation may be an after thought and funding must be sought after the program has been introduced. Post-implementation planning of evaluations may also arise where it
is realised, after the fact, that a series of actions constitute a program and have led to an outcome which should be assessed.
Federal and provincial government agencies are occasionally willing to fund the set-up and evaluation of innovative programs which can be transferred to other communities. Private
companies are usually more willing to sponsor capital projects such as buildings and, sometimes, high profile crime prevention initiatives rather than operational costs such as crime
prevention evaluation.
The manual provides a structure for developing a proposal for the funding of an evaluation project. The possible costs to consider in developing a budget are listed in the
appendices.
Frequently, it is possible to undertake an evaluation with limited resources. Basic evaluations are often sufficient to determine the relative success of many programs and cost may be
limited to the evaluators’ time and incidentals such as the printing of forms for data gathering and the purchase of stamps if mail surveys are considered. Personal computers now
have the capacity to provide relatively sophisticated support for calculations, tables and report preparation.
In addition, the joint participation in an evaluation with a nearby community which has introduced a similar crime prevention program may help to defray costs. As noted below, it is
therefore important to secure an understanding of other crime prevention programs and evaluations which have been implemented.
Of course, whatever funding you may have acquired should be allotted proportionately to the different stages of the evaluation. If you choose to undertake a method of information
gathering such as a community survey, this will require more funding than, say, a series of interviews with agency staff.
The manual is a ‘how-to’ guide for use by agency personnel who can work through the document, step by step, to assess whether the crime prevention program has achieved its
objectives. The manual can also be used as a structure to develop a proposal to seek funding for an evaluation.
You will note that detail is provided by supplying information, such as, the pro’s and con’s of different methods to gather information. It is presumed that, on many
occasions, you will be undertaking an evaluation with limited outside support. The detail will equip you with the technical knowledge that you need to consider the specifics of the
particular program which you are evaluating and the characteristics of your community and to develop a work plan which is tailored to your needs by using the most appropriate tools.
Ideally the evaluation plan will be built into the program from the outset. In reality, however, the evaluation of a program is often introduced after the fact. The manual attempts
to accommodate the need to sometimes catch-up and to fill-in the gaps after the program has started.
The manual lays out the principles and steps to be taken in the evaluation of any crime prevention program. Not all programs will need this amount of detail. Not all programs will
have all the information noted available to undertake an evaluation. However, this manual will equip you with the skills to tackle almost any evaluation project.
The crime prevention practitioner must gauge the intended or likely overall impact of the program on resources, time, or the community to determine to what depth the evaluation should be
pursued.
The content of the manual then needs to be selectively “geared back” to suit the scope and complexity of the program. Following the principles outlined, considering each
step and deciding whether it is relevant to your program will ensure that whatever the size of the program you will have subjected it to a valid review.
The manual refers to the evaluation of programs. It does not address the design or execution of programs. These are specialised topics in themselves and the reader is directed to other
materials for information regarding these topics.
Throughout the manual we will work with an example of a crime prevention program introduced in Moon River. This is a community of 7500 people. A Steering Committee was formed to
address a community problem. Unfortunately, there was no needs assessment completed and no evaluation was planned as part of the program. You have been asked to undertake an evaluation
as the program comes to the end of its first year of operation.
The following is extracted from Band Council minutes,
“Residents have expressed concern to the Band Council regarding the amount of domestic assaults which occur in the community. A Steering Committee has been formed to develop
a plan to resolve the problem.
The Committee includes Councillors, Elders, community members, police and social service agency.
Police records show that there has been an average of 135 calls regarding domestic incidents each month for the past year..
A Victim Services Unit was formed from community volunteers and attended calls of a domestic nature when called by police.”
At specific points in the manual there are a series of questions which you can pose to yourself to assess where you are in the process. These questions will help you focus on the
principle decisions.
All this might sound overwhelming. Don’t worry. Things will become more clear as you read on and begin to look at the program you are going to evaluate in your community.
The program to be evaluated must be seen in its operational and theoretical context. What is this program meant to be achieving? The background documents and literature upon which
the program was based should be reviewed. Where did the idea for this program come from and why? If possible, conduct a literature search or make a few telephone calls to determine
whether other programs of a similar nature have been introduced and whether these programs have been subject to an evaluation.
Local agency contacts such as (other) police officers or government department personnel may provide information. The Internet and distance education facilities can give access to a
wide variety of national and international literature, and follow-up via telephone may provide hard-copy or e-mail versions of the background materials.
If the program has generated local interest then local media files may provide background information.
Consult the stakeholders in the program. In your discussion, focus on what the program is and was intended to be rather than on what form it now takes. Understanding what
is currently happening will come later.
It is important to view the evaluation (and the original program) in its broad context so that, as an evaluator, you are able to later assess issues which have to be considered in the
evaluation. Consider factors such as the,
Community - Geographic size, population and age breakdown, general crime trends, suicide rates, divorce figures, cultural and historical backgrounds and migration factors.
Economic factors - Band and individual income and sources, employment and unemployment, opportunities for off-reserve income, businesses which generate income that remains in the
community rather than businesses where the money moves out.
Socio-political - The views of the Band Council, the views of the Elders, influence of the provincial and federal governments, impact of other factors such as the media and major
income generators such as casino’s or oil companies.
Sources such as census data, government inquiries, newspaper files and oral traditions may provide background information.
When the information has been gathered,
- Prepare a brief description of the operation of the program. Explain who, what, when, where, is (or was) the intended focus of the program and how it was supposed to operate.
- Describe similar programs in other communities. How have these programs been operated? Were the programs evaluated? How were they evaluated? Were they successful?
Define the purpose of the program. What were the program objectives? What was the original vision for the program when it was introduced? What specific community order or crime problem
was the program intended to address?
Don’t forget, at this time, there is no need to concern yourself with whether, or even how, the program currently works. We are only looking at what it was meant to be doing and
in what context it is operating.
In our example,
The minutes of the meeting do not give us the objectives for the Victim Services Unit. Ideally, the objectives would have been established as the program was set-up. These would
probably have been something like,
‘Improve the way in which the community deals with domestic situation’, or
‘People feel safe in their homes’
In the absence of prepared objectives, talk with members of the Steering Committee and the volunteer workers to try and establish the objectives of the Unit.
Examine census data to get an understanding of demographic data that might impact this issue.
Victim Services Units are certainly present in other communities, draw in some of the background to these programs and the ways in which they may have been evaluated.
Maybe undertake a file search in the local newspapers as this should be a topic of public interest.
The objectives provide a focus for the work. What were the original objectives of the crime prevention program? Understanding these will assist you in determining the objectives of the
evaluation.
Then, following on from this, what are you trying to determine in your evaluation?
An objective states clearly what you want the evaluation project to do. An objective (or objectives, there may be more than one), is a clear statement of the results which you want to
achieve. A program and an evaluation plan must be realistic. Select a manageable number of objectives. Perhaps, three to six is manageable. More objectives than this and the
evaluation will lose focus. (More program objectives than this and the original program will also lose focus).
If there is no clear statement of objectives for the program, and you are unable to discover one in your initial talks with stakeholders, then you must “blue-sky” to develop
possible impacts of the program to provide some focus for your evaluation.
- It must be clear so that all participants in the program or in the evaluation fully understand what the program is intended to do.
- The objectives must be achievable. Be creative and optimistic but be realistic.
- The objectives must be measurable so that the evaluator can collect information to assess the success of the program, whether it has “measured up” to expectations. Later we
will describe the types of data which will become the foundation for the evaluation.
- The objectives must be long range. This provides a focus for program activities and permits sufficient time to have elapsed for the program to have some effect. At the same time, the
duration must not be too long. Resources are being expended and may be more efficiently spent in other programs.
- The duration of the program must be stipulated. A point(s) in time for evaluation must be decided at the outset and a point in the future when you will make decisions regarding the
continuation of the program should be decided in the planning stage. This will provide focus for the participants, ensure stakeholders that an assessment will be made at some time and
prevent the program simply limping on with no evaluation.
- The objectives should be flexible. As you monitor the program (see above) it may be necessary to adjust the program due to unforeseen circumstances. It is pointless to doggedly
pursue the original goals if the circumstances surrounding the crime prevention program have changed. There is nothing more futile than doing well something that doesn’t need to be
done at all!
- An integral part of the objectives is assigning responsibility for action. Participants must know what role they will play in achieving the program or group goal.
The objectives must be acceptable to all participants. The goal must not only address the problem which has been identified but also be ethically acceptable to everyone. For example,
nobody should have vital services withdrawn as an experimental part of a program or evaluation.
In our example,
We are not given information regarding the objectives.
If prepared properly, the objectives might have been,
‘Reduce the number of call-backs to domestic incidents by 40% within one year’. Or,
‘Reduce time spent by officers at the scene of a domestic incident by 25% in one year.’ Or,
“increase the level of community satisfaction with the way domestic incidents are dealt with by 40% in one year’.
You, as evaluator, would have to develop the objective(s) based upon your conversation with the committee members and staff. You can assume that it involved a reduction in incidents of
domestic violence but this would have to be determined with anticipated reductions and a time period to achieve the reduction.
Alternatively, the actual impact can be determined as a result of the evaluation. You may try and set some focus for the evaluation by assuming certain impacts may occur and the true
effect will be determined later when you come to review all the information gathered during your evaluation process.
If the objective shows what you want the program to do, then we must also develop an understanding of what types of information you will need to have to know whether this objective has
been met. The information which we use as a basis for this decision is an indicator.
An indicator is a bit of information (such as an event or a statistic) to measure an effect of a program. The information ‘indicates’ whether the program has had an
impact.
Quantitative indicators - These are essentially numbers or quantities. As we will see below, these are based on data such as the calls attended by the police, charges laid or the
numbers of children taken into care or reports of vandalism.
In our example,
Changes in calls for assistance regarding domestic incidents.
Changes in call backs to addresses previously attended.
Changes in arrest rates. or
Changes in time taken by police and Unit members at the scene of a domestic.
Qualitative indicators - These are based upon the ‘quality’ of the program activities and may be opinions of participants or agency workers. These tend to be
descriptive but when collected or aggregated may show, for example, the percentage of those surveyed who believe that a program works.
In our example,
Decrease or increase in level of satisfaction amongst victims regarding the way domestics have been dealt with.
Decrease or increase in the levels of satisfaction of police and agency staff regarding the way domestic incidents are dealt with, or,
Decrease or increase in the degree to which the general community sees domestics as a community problem.
Short term indicators - These are indicators which are measured over a brief interval and although short term indicators are important and tell us something about progress they
are likely to be inconclusive regarding the overall success of a program.
Long term Indicators - These are indicators which are measured over the longer term and tend to address the long term impact of a program, often over its full duration.
There are three types of evaluation processes that you can use. We mentioned these earlier. Here we will explain a little more.
Needs Assessment - The needs assessment typically occurs at the early stage of the program development process. It will help you to sift through the various crime problems which
your community may face and to focus upon those of the highest priority. Essentially, what are the needs of the community?
The needs assessment should not really be part of an evaluation program. It should have been the basis for the development of the original program designed to address the identified
need. We mention it here because if a needs assessment has been conducted, it will provide some useful information to you as a basis for your work.
Process Evaluation (The means to the end) - The process evaluation allows the evaluator to monitor the implementation and operation of the crime prevention program to assess the
effectiveness of its operation.
The process evaluation may review each or selected components of the program, such as participant selection, training of counsellors, the speed of the police attending calls, to
determine whether each is operated appropriately having consideration of the program objectives.
Impact evaluation (Whether the end was achieved) - The Impact evaluation is usually undertaken at the termination of the program to determine the effect of the program. As the
name implies, the evaluation is conducted to ascertain the impact of the program on the original state of affairs. Did the program achieve its objectives? The Impact evaluation is the
basis for the crucial question regarding whether the resources were well spent, did the community ultimately benefit and whether the program should be continued.
There may be more than one point at which the impact evaluation is undertaken. For example, it may be appropriate when assessing the long term effect of a crime prevention program to
evaluate its impact at the termination of the program and then again one year later.
(You may also consider assessing the impact of one portion of the program. For example, you could develop a brief questionnaire to assess whether the attendees at a meeting found the
session to be useful.)
In our example,
It is too late to do a needs assessment as the program has already been introduced. Although some background information may come out as to whether the problem was correctly identified
and whether the Victim Services Unit was the appropriate solution. This may appear in the conclusions.
You may wish to look at both process and impact evaluations. It will be important to review matters such as, how unit members were selected and trained, what action police take at
the scene prior to calling the unit and even support mechanism for victims. These factors may all influence the outcome, the impact. As far as impact, you will obviously want to know
whether the situation has changed as a result of the Unit.
There are two types of data which can form the foundation for your evaluation.
Quantitative data - This looks at the quantities in the evaluation. It is the counting of results and consists of the statistics and numbers which are obtained through the use
of measurement tools. This is sometimes called ‘hard data’. It is the data which, as we have seen above, supports quantitative indicators.
In our example,
Calls to domestic incidents.
Numbers of reports submitted
Arrests as a result of domestic incident calls
Time taken at the scene of a domestic incident call.
Numbers of injuries to officers or victims at the scene of domestic incidents, or
Incidents of use of force at domestic incidents
Qualitative Data - This looks at the quality of the crime prevention program and is based upon the views and opinions of participants, stakeholders and others associated with the
program. The qualitative approach is more descriptive. It is sometimes called ‘soft data’. It is the data which, as we have seen above, supports qualitative indicators.
In our example,
Opinion of agency workers regarding their perception of how domestic incidents are dealt with.
Victims satisfaction with the service by the unit, or
Community perception of domestic incidents as a problem.
The two approaches are not mutually exclusive. An evaluator does not have to choose one or the other in an evaluation. In fact, some of the best evaluations use a combination of the
two approaches. Conclusions supported by both quantitative and qualitative data are often stronger than those based on one or the other. Also, much of the quantitative data gathered
through interviews is expressed in terms of the quantification of the findings.
In our example,
67% of respondents believe that the service to victims had improved since the introduction of the Unit.
The quantitative and qualitative approaches require different tools for gathering data and different skill in the evaluators.
The quantitative approach is numbers based and is more definitive in the sense that percentages can be compared across programs or communities or regarding the situation before the
program and after the program. (This is called a pre- and post test.)
In our example,
You may find, as a result of your work, that before the Unit was introduced (pre-) 28% of calls regarding domestic incidents resulted in arrest. After one year of the operation of the
Unit, (post-) the figure was down to 15%. You may further discover that this compares to figures of 38% and 22% respectively in the community of Montego River.
The qualitative approach is a more interactive approach in which opinions are asked or documents examined or observations made on activities within the program. The qualitative approach
is a less rigid evaluation design. Although the results of the qualitative approach may ultimately be quantified as a statistical result the data is based upon opinion and feelings.
In our example,
You may find that, ‘Generally, it was apparent in interviews that members of the unit believed that they required more training in mediation skills whereas police officers believed
that the highest priority was to develop a data base with information regarding previous weapons calls to residences to which they are called for a domestic incident.’
There are three types of data which can be used to determine the effectiveness of your program:
Baseline data - This refers to the known data or information which was available before the program began. This provides a basis for comparison as it describes the state of
affairs before any crime prevention action was taken.
However, baseline data is not just the information which was collected in the normal course of operating an agency or a program. Baseline data may be specifically developed for the
purposes of the evaluation to provide a basis for a ‘before’ and ‘after’ test.
In our example,
An average of 135 calls per month for the past year.
A review of police records may also show how long was spent on each call and the number of arrests and maybe whether injuries were sustained by any of those involved.
This will provide some information with which we can compare what was occurring at the end of the program
Progress data - This refers to the information which you are collecting as the program proceeds according to the action plan. Like baseline data, these data may be the
information which is normally collected in the course of operations or the you may have to develop the data gathering tools for the purposes of getting information for the evaluation.
In our example,
A review of police files such as incident reports would demonstrate the number of calls, arrests or other actions each month or each week over the duration of the program. You would
have to check to see whether the unit staff were required to complete reports.
Court reports would show the disposition of cases but remember that offenders take considerable time to pass though the justice system.
Culminating data - This is the information which is collected at the end of the program evaluation term and comprises the closing data. As with baseline data, the information may
simply be that which is collected in the normal function of the agency or program. Alternatively, it may be collected and analysed specifically for the evaluation.
In our example,
Collect the same types of information which were collected for baseline data. During the course of the evaluation, you will have collected other progress (or process) information and
may even have decided that other data should also be collected as part of the operation of the program.
You are interested in comparing what was then with what is now and the process of change.
This progress information would be the ‘connector’ between then and now, between the base line and the culminating data.
There should be considerable culminating data regarding the total calls, arrests and summonses, information which is collected in the normal course of business and, maybe, surveys which
you conducted.
There are several sources for the information which you can use as a basis for your evaluation.
Statistics are merely the numbers you can use as a basis for your evaluation. This is the hard data which we mentioned earlier. Various forms of statistics are compiled by most
government, commercial and private organisations and agencies. In the normal course of business, these operations need such data to monitor organisational and individual performance or to
determine such things as unit cost.
The federal government requires that agencies involved in the administration of criminal justice such as police, courts and probation, collect, compile and forward to Statistics Canada
certain information regarding the operation of the agency. In the case of the police, which will deal with the majority of crime prevention initiatives in the community, the Uniform
Crime Reporting System (UCRS) specifically requires that detailed information concerning reported crime, arrests, charges and other data is compiled and submitted to the federal government
for analysis. These aggregate data are readily available from the police or from Centre for Criminal Justice Statistics of Statistics Canada.
There are other centrally co-ordinated statistical collections which are not a function of Statistics Canada. One such data base is Vi-class which is dedicated to the development of an
information data base concerning violent offenders. This is a highly specialised data base and, generally, only accessible to law enforcement agency staff.
As initiatives such as Problem Oriented Policing become more common, general statistical information regarding crime and disorder in the community will become more accessible.
Other statistical data may have to be collected specifically for the evaluation program. The type of statistics you may have to develop will depend upon the specific issue being
evaluated and the characteristics of the community in which you are working.
In or example,
You may wish to determine the geographical spread of calls based upon statistics kept by police or by the unit. You may also wish to know age distribution of victims and offenders. You
would almost certainly want data such as arrests, charges, offences committed and sex of victims and offenders, (don’t always assume that it is males who are the offenders).
Owing to the wide variety and volume of data collected, it is important that you are specific regarding exactly what information is required to facilitate the evaluation. Sometimes it
may be necessary to go beyond the data collected for the purposes of federal government analysis and use information collected in case files or through a Computer Aided Dispatch (CAD)
system which tracks information regarding the dispatch of police to incidents and the reporting of criminal incidents.
The policing standards in most provinces now require police agencies to maintain a statistical data base and a formal case management system. Providing confidentiality requirements are
met, access to this data will provide background to process and impact evaluations.
Annual reports prepared by criminal justice agencies are a ready source of information. Periodic reports from the federal government will allow cross-community comparisons.
Wherever criminal statistics are used in an evaluation there must be a strong caution. Criminal statistics are more the record of police actions than a record of criminal events which
occur in the community. There are several filters which will influence the accuracy of such data.
- Not every incident is witnessed or witnessed by someone who cares enough to involve the authorities
- Often victims do not know a crime has occurred or simply do not bother to report or may be too intimidated to report.
- Police will not attend every call for assistance. The police often have other priorities and, particularly, in remote areas the police are not available to deal with many incidents
which may receive attention if they occurred in an urban area.
- Even if the police attend an incident, they will not always arrest an offender even where sufficient evidence exists to justify arrest. Many factors beyond simply the facts of the
case are considered by an officer before he/she decides to take a person into custody. Frequently, police will not even complete a report of an incident.
- Frequently enforcement policies influence arrest rates. For example, where a provincial policy addresses specifically “Assault of Women” and requires the police to arrest an
alleged offender, the police are more inclined to arrest than treat the incident in some other way. They are also more likely to arrest a male offender in such a circumstance than a woman
who has assaulted her male partner. Also, some police agencies encourage the use of counselling by officers as a response to limited court facilities.
- Where police statistics are biased, the same bias will be reflected throughout the criminal justice system.
- The culture in a community often means that incidents are dealt with in ways that do not lead to official reporting of an incident.
The so-called “victim-less” crimes are rarely reported as the parties involved have an interest in not revealing the commission of the offence.
In our example,
Information such as calls attended, domestic and others, as time spent by police on domestic incidents may detract from other types of calls.
Victims and offences per 100,000 population are common figures used but have accuracy problems. Stats in your community may be influenced by perceptions of previous police response to
domestic incidents. The existence of a Victim Services Unit may increase the number of calls for assistance as victims may now be more willing to call for assistance.
Check to see whether there are any provincial or police agency policies which require certain actions be taken at domestic incidents. In such cases the discretion of the officer
is reduced or removed.
- Publications available from UCRS will provide you with an understanding of the breadth of statistical information which is available concerning the activities of the agencies involved
in the administration of criminal justice.
- Review the annual reports of the police or other agencies which may have an impact on the operation of the subject program. These reports will provide a more local flavour and they are
likely to be more detailed than you would find in federal government data banks.
- Remember the caution when analysing police and other agency statistics concerning crime. The data is more an account of agency activity than a true reflection of the crime and disorder
which may exist in the community.
- Aggregated data from local police files will show significant detail regarding the geographic distribution of incidents reported and calls attended, the distribution of workload
throughout the 24 hours and through the week. Use this data for comparative pre-and post-assessment of program impact and for cross-jurisdictional comparisons.
- As you review the statistics to be used in the evaluation consider the factors which may influence the accuracy of the data. Allow for this in deciding your method of collection and
other information sources which you may use.
- Look for ‘groupings’ of statistics, such as changes in the rates of vandalism throughout the year, which will allow for the measurement of change through increases or
decreases over the duration of the program.
- In some communities there may be few stats kept on crime or disorder problems. It may be necessary for you to develop a process to collect this information over the course of your
evaluation.
Decide which group of statistics is relevant to the focus of the evaluation. These data may be available in the public reports issued by Statistics Canada, other data can be ordered
through Statistics Canada for a cost. The exact cost depends upon the complexity of the computer run required to generate the data you require.
The examination of files and unpublished reports relevant to the program being evaluated provide valuable background regarding the development and operation of the program. Examined in
the preparatory stages of the evaluation, the review can place the program in its issue context.
In larger urban centres, file information is often maintained in Management Information Systems (MIS). In smaller centres, it is less likely that such systems, with the relatively
easier methods of extracting data, are available.
Whether an automated or manual system is used, the evaluator first needs to know what information is available. The agency manager (or information systems manager in larger
organisations) should be able to explain the range of data and what might be applicable to the particular evaluation project. Remember the information on file is not an evaluation in
itself. The data which is collected is only, collectively, bits of information which can be used in an evaluation.
The evaluator should not only draw client or victim or incident data from files. In gaining an understanding of the process, information such as position descriptions, training
schedules, business plans and budget information will provide background regarding the effectiveness of the support elements of an organisation. For example, a crime prevention program
may be designed and implemented such that the opportunities for success is maximised, however, if personnel are not trained to operate in the different work environment then the program may
falter.
In our example,
Review documents such as agency position descriptions and performance appraisal processes. Agency staff, such as police, may be required to be more reactive than pro-active in their
response to community problems. This is likely to orient them more towards arrest rather than other forms of resolution. Alternatively, police maybe encouraged to counsel victims and
offenders. Both approaches will influence the outcome of the program as the opportunities for action by members of the Unit may be reduced.
The evaluator should formulate some idea of the information which is required, based upon indicators, before approaching the file or information bank holder. General ‘fishing
expeditions’ will not encourage a sense of confidence in the evaluation or the evaluator.
Much of the data which is collected as part of the routine management and operation of an agency is collected for local purposes and may not be submitted for federal compilation.
Information such as addresses of offenders or victims, times of calls attended by police, daily records of school attendance are often retained at the local level. Access to such
information may require a confidentiality agreement.
In our example,
Consider police or Unit reports regarding incidents with information such as where in the community the incidents occurred, whether alcohol was involved, injuries sustained, or children
taken into custody.
- The file search and review should be one of the first steps in gathering background data. The method is relatively non-intrusive, is low cost and is often high yield.
- As files and documents are reviewed, gather information which provides you with a broad understanding of the developmental context and operations of the program which is the subject of
the evaluation.
- Take notes on the issues and note names of those involved in the program (or their agency positions).
- Maintain a list, perhaps on file cards of documents used in the review.
Discussion with stakeholders and program staff will provide information concerning files and documents which may be relevant to the evaluation. Reports such as needs assessments which
may have been prepared at the outset of the program may reveal community profiles, the communities perception of crime issues which should be addressed and the rationale for such
decisions.
Surveys can be distributed to large numbers of people with the benefit that a broad spectrum of views is facilitated and more people are provided the opportunity to offer an opinion on
the program.
Surveys can be conducted prior to and at the end of a program to assess, for example, changes in attitudes or perceptions.
The principle survey methods are telephone and mail-out (including general distribution to employees of an agency).
Telephone surveys - Provide the opportunity for large sampling of views from the relative ease at the end of a telephone. However, many isolated communities may have restricted
access to telephones. In addition, given wide geographical areas this method can be costly. Depending upon the size of the survey, significant cost can be incurred if a private company
is used to conduct the work. The advantage is, however, that the staff are trained to conduct this type of work, call-backs are possible at a range of times without too much inconvenience
to the evaluator and, by arrangement, responses are received tabulated.
Unfortunately, response rates are fairly low. This depends upon the issue covered however. Having dinner disturbed by a telephone survey on crime and policing issues may be more
acceptable than less “interesting” topics that are unlikely to directly affect the respondent.
Mail out surveys - Mail-out surveys suffer from similar problem as telephone surveys. A potential respondent is less likely to respond to a survey which arrives through a mail
box than through a telephone contact. There is often a need for follow-up to encourage to potential respondents and even then response rates are generally low.
There is greater chance of a higher response rate if questionnaires are distributed to, for example, employees of an criminal justice agency as the issues are likely to be more relevant
and follow-up is easier. However, unless questionnaires are confidential, responses may be tailored in anticipation of the answers being seen by other agency personnel.
In all survey preparation, specific questions and the format of questionnaires must be prepared very carefully to elicit response which are relevant without introducing bias in the
respondents answers. Questions cannot ‘lead’ the respondent but must give a clear indication of the types of information which is being sought.
Survey questions can be either open-ended, multiple choice or closed.
An Open ended question invites a narrative opinion from the respondent. Although this gives the respondent more latitude to answer in exactly the way that he or she wishes, the
open-ended question is difficult to analyse. The evaluator must individually read and assess the answer to deduce the specific point that the respondent is trying to make in the
response.
In our example,
“As a volunteer with the Victim Services Unit describe what procedures you follow when you attend a domestic incident”, or
“What do you consider are the main benefits of a Victim Services Unit?”
The multiple choice question requires that the evaluator anticipate the range of possible responses to a question. This range is then entered on the answer sheet for
consideration by the respondent who marks the appropriate response or responses which he or she consider most accurately reflects their view. It must be made clear whether the respondent
has an option of one or more responses to each question and a category such as ‘Not known’ or ‘Other’ (with space for elaboration) should always be permitted.
Included in the multiple choice form of questionnaire is the ‘scale’ response in which respondents are invited to provide a view on a graded scale of, say, 1 to 5, with the
numbers corresponding to progressive levels of preference. Alternatively, the numbers may refer to a series of non-progressive options such as the respondents view on the important crime
issues in the community.
A field test, by asking a few respondents the draft questions, prior to the beginning of the evaluation will frequently allow for refinement of the categories in a multiple choice
questionnaire.
In our example,
“As a member of the Steering Committee with the Victim Services Unit, do you think that the quality of service provided to victims to in domestic incidents now compared with a
year ago is:
Significantly better __(1), Somewhat better _ (2) No different __(3) Somewhat worse (4), Significantly worse (5) Don’t know __(6).”
This might be followed by an open ended question,
“Why do you think this?”
or
“Please indicate what you consider to be the top three skills that you should have when you attend a domestic incident call, Please tick the most important three
skills.
Mediation ( ), Self defence ( ), First Aid ( ), Knowledge of the Criminal Code ( ), Dealing with firearms ( ), Report writing ( ), Using special equipment such as radio or cell phone
( ), Native languages ( ), Knowledge of body language and culture ( ), Don’t know ( ).”
Closed answer format gives the respondent only the opportunity to answer with a restricted set of options, such as, ‘Yes’, ‘No’ or ‘Not
sure’.
In our example,
“Do you think that services to victims of domestic violence have improved since the Victim Services Unit was introduced?
Yes ( ), No ( ), Don’t know ( ).
Accurate sampling is crucial to a valid survey. The potential recipients of the survey must accurately represent the breakdown in the population of the community or the organisation
which is being surveyed. The evaluator must, therefore, know precisely what is the make-up of the total group of victims or offenders or community citizens or employees before accurately
determining the range of, and number of, people who will be surveyed. Statistics Canada census data will provide demographic background on very specific geographical areas.
In our example,
‘Domestic incidents’ is a very specific topic for a community survey so you may prefer to undertake a survey of agency workers who are directly involved in dealing with the
issues of domestic incidents. Alternatively, you could expand the survey to address community problems in general and the possible expansion of the Unit or you may decide that you would
get more valuable information and use your time more productively by using other information gathering methods.
All questionnaires should be field tested prior to the full survey. For example, pre-testing through role playing with agency staff may reduce the error rate. The development of good
surveys are quite difficult and time consuming. Through your background research you can often find examples of surveys conducted in other communities.
The Pros:
Information is aggregated or grouped for analysis, so no-one is identified individually. The anonymity may encourage more people to participate.
Conducted using a large number of respondents, general views on issues are captured rather than the opinion of a few individuals which may reflect a bias.
Easy to administer and provides input to the evaluation by large groups of people.
Closed questions allow for easy tabulation of responses
The Cons:
Telephone or mail-out surveys are expensive and generally return or answer rates are low.
They are difficult to develop as the questions must be exact to focus specifically upon the issue(s) which you want to examine.
In many communities, telephones may not be available.
In many communities, telephone or writing on a survey may not be the preferred form of communication.
The sample, or people who are asked to respond, must be representative, that is, the people who are given an opportunity to respond must reflect the total population, such as, potential
users of the program or tax payers or police officers. The sample should reflect only the bias which naturally appears in the target group, such as males and females, age, income levels,
agency ranks, location of residence or victims versus non-victims. A random sample, that is distributing the survey over a wide population in the hope of including all these biases, is an
effective way of gathering information.
- Decide what information you want to know, what are the issues to be addressed bearing in mind the evaluation objective.
- Determine a list of topics which can become the focus of your questions.
- For each topic, develop a series of questions which focus on the nub of the topic. Be aware that too many questions will put people off, not enough will not provide the information
that you are seeking.
- Decide what form of question is most appropriate for each topic. Is it appropriate to ask a question which can be answered by ‘yes’ or ‘no’ or is a more
detailed answer which allows the respondent to explain a feeling or a perception more appropriate?
- If you decide upon a question which allows selection from a multiple choice, have you considered all possible responses and then reduced these to a reasonable number of options?
Somewhere between five and ten is a reasonable number. Allow for ‘Don’t knows’ and ‘Not applicable’
- Review your questions.
- Is each question clear regarding what type of information you are trying to receive?
- Will the wording of the question influence the nature or tone of the response?
- Does each question pose just one question and not more?
- Test the questions on a colleague.
- Does he or she understand them?
- Are there any that are repetitive?
- Are there other responses in the multiple choice which you had not anticipated?
- How much time does it take to complete the survey?
- Place the questions in a logical order. If a topic follows on from, or develops on, another question make these consecutive. Number the questions.
- Prepare an introduction which briefly explains the purpose of the survey and the goal of ensuring confidentiality of information. This will be required whatever method you decide for
administration.
- Decide upon the method of distribution.
- By mail? Have you enclosed stamped self addressed envelopes to maximise the possibility of returns?
- Telephone? Respondents are less patient sitting on the phone than in person.
- Dropped off and picked up?
- Will the questionnaires be anonymous? Do you need to check up on returns? Have you assigned code numbers?
- Maintain a master list of persons to whom the questionnaire is sent. Always place a contact number on the survey form. Respondents may have questions regarding the survey content or
its administration.
Determine the sample group who will receive the survey. This will determine the nature and content of questions. You may need to collect basic demographic data to ensure that your
sample is representative. Question wording should allow for educational, language and cultural differences.
Focus groups provide a forum for open discussion of selected topics as a basis for an evaluation.
In our example,
You may consider holding a focus group with members of the Unit on topics such as ‘Dealing with victims’ or ‘Liaison with the police’. The presence of police
officers may make the session more productive or inhibit the volunteers. You may consider a focus group with staff from shelters in your community and adjoining communities.
The Pros and The Cons of focus groups
The Pros:
Such discussion permits a dynamic exchange of ideas. Views expressed by one person can stimulate an opinion from another.
Structured and facilitated appropriately, the focus group can explore several issues in an in-depth way in a relatively short time.
Although larger focus groups are not advisable, large numbers of people can be provided input through the use of an electronic touchpad. Each participant is provided with a numbered
touchpad on which they enter the number which most accurately reflects their view on a particular structured question and answer. Feedback is immediate, the process is dynamic but tends to
cost several hundred dollars for equipment rental and in remote communities additional travel costs.
The Cons:
The groups must consist of a manageable number of people so a co-ordinator can facilitate the discussion. Six or ten and possibly 15 people may be suitable but two hundred is not.
The participants must be drawn from persons who have some interest or opinion on the issues under discussion. If the person is totally untouched or unconcerned about the issues then
they are unlikely to have any opinion.
An assertive person may be able to dominate the discussion.
The facilitator and the recorder must be adept at their tasks to ensure that discussion does not get ‘off-track’ and that all relevant information is recorded and
analysed.
Focus groups are not appropriate when working in cultures, with persons or, perhaps, age groups which do not expect open, public expression of ideas and feelings. Elders may not want to
share their views with others from the band or have their views challenged.
Focus groups are only effective when the topic is one with which are comfortable discussing in an open forum.
Participants must be able to communicate in a common language.
Developing....
- Determine a list of topics which are suitable for group discussion. There should be between three and seven. The number of issues actually addressed depends upon the amount of
discussion in the group given the time available.
- Identify people who will be suitable participants. As we have seen above, the group should be a manageable number which will allow all to participate in the discussion. Those selected
must also have an interest in the topics which are to be discussed. They may be participants in the program, victims, members of the community, Elders, police officers or program staff.
The mix of person should reflect the various perspectives on the issues but allow for equal participation.
- Locate a suitable space for holding the focus group. Participants should not have to worry about attending to work, family or band matters. Ensure that flip charts are available or that
paper can be affixed to the wall. Provide felt tip markers.
- Estimate the duration of the meeting. A half day might be appropriate, a full day may be acceptable but more than that and participants will be reluctant to take time away from work
and will be unable to concentrate.
- Identify someone who is willing to take notes but who will not participate.
- Notify all participants and consider distributing background material prior to the meeting if this will facilitate discussion.
- and Running.....
- Introduce the session, the objectives and the ‘rules of conduct’ (which might include everyone’s right to be heard) and the requirements for confidentiality.
Introduce the topics to be discussed and provide adequate background without introducing bias to the discussion.
- Moderate the discussion by keeping the group on topic, encouraging participation, being supportive, stimulating discussion where appropriate but, remember, the objective is to hear the
views of the participants not to let others be exposed to your perspective.
- Use flip charts to summarise discussions. These can be used later as input to meeting notes.
- Some participants may not be able to read in the selected written language of the focus group. Assist these people by, for example, explaining the notes on the flip chart.
- At the close of the focus group session ask the participants if there are any issues which have been missed. Thank the participants for their contribution.
- Explain the next steps in the evaluation and what will become of the material which has been collected during the focus group session.
Research the topic prior to the session. You will need to introduce the issue for discussion and often prompt and move the discussion on with examples of the matters being
discussed.
Interviews provide an opportunity for face to face questioning of a program participant, agency worker, community member or a representatives of other programs or agencies which may be
impacted or have a view on the program being evaluated.
There are two types of interview, structured and unstructured.
The structured interview requires the preparation of a series of questions or specific issues to be covered by the interviewer. The interview is focused and proceeds through the
series of topics with the responses noted in detail.
In our example,
“How were you selected to be a member of the Victim Services Unit?”
“Do you think that there should be any other skills looked for in the selection process?”
“What training did you receive?”
“How long was the training?”
“Who conducted the training?
“Is there any additional training which you believe you should have received?”
The unstructured interviewallows more flexibility from both the interviewer and the interviewee. Although the general topics to be addressed are prepared ahead of time, the
direction of the discussion is more free flow and dictated by the collective desire of the interviewer and the interviewee to examine the general issues which are the subject of the
evaluation.
In our example,
“Tell me about your experiences of working with the Victim Services Unit”
The Pros:
Allow detailed descriptions of an event, feelings or perspectives.
Face to face contact develops rapport and, possibly, more open communication.
Response rate to questions is far higher than alternatives as the potential respondent will find it hard to refuse to participate in a one on one interview.
The interviewer can use probing questions to explore at greater depth an issue raised by a respondent.
The body language of the respondent can be observed and the interview adjusted.
The Cons:
Time consuming. Interviews may take between one and three hours. Writing up of notes is likely to take a considerable time.
It is difficult to take detailed notes during an interview and most respondents will be reluctant to speak into a tape recorder.
The body language of the interviewer can be observed and the response adjusted.
- Consider the people who work in, or are impacted in some way by, the program being evaluated. Decide how these people can provide in-depth information which may be relevant to the
evaluation.
- Turn each topic area into a question or series of questions. Develop follow-up questions which may be used as prompts.
- Test the questions on a colleague who understands the need for confidentially. Amend the format to remove duplicate questions and to ensure clarity.
- Make appointments with interviewees, briefly informing them of the context for the interview and provide an estimate of the time to be taken by the interview. If an interview takes
more than one or two hours the respondent is likely to lose interest.
- When meeting with the subject, describe the project and explain the confidential nature of the interview. Provide a brief overview of the topics to be covered and a time estimate.
Keep to the time limitation.
- Allow the respondent to speak freely guided by the interview topics which you have established. Use the prompt questions to delve deeper or to clarify something the interviewee has
said.
- Take notes. Do not hold up the flow of the interview by laboriously writing everything which is said but make your notes sufficiently detailed that you can transcribe them later. Most
people will be inhibited by the use of a tape recorder during the interview.
- Do not debate the issues with the respondent. You are there to listen to the contacts perspective on the topics.
- Remain aware of the respondents reaction to your questions.
- Does he or she understand the question or the language you are using?
- Does body language demonstrate that the person is uncomfortable with the tone of the discussion?
- At the close of the interview, thank the respondent for his/her participation. Offer a contact number should there be a need to communicate with you.
Bearing in mind the objectives of the evaluation, prepare a list of issue areas on which the subject has special expertise or can provide a special perspective.
Observations are important to an evaluation. The evaluator is provided the opportunity to view the program in operation. Observation provides a practical context for the
evaluation.
In our example,
Consider going on a ride-along with the Victim Services Unit or with the police or sitting in court to observe cases arising from domestic incidents.
The Pros:
The over-riding benefit of observations is that the evaluator is provided a practical perspective on the program which often cannot be achieved through other methods of information
gathering.
The Cons:
In the presence of an observer the subject, such as the agency worker or the program participant, is likely to act in a different way than when not being observed. This is particularly
so when the evaluator becomes a participant observer and is involved, even in a peripheral or passive way, in the program. The findings from the observation will, therefore, be a
distortion of what normally occurs.
The subject of the observations is likely to ask question regarding the evaluation. It is difficult for the evaluator to respond without alerting the subject to the issues which are of
importance to the evaluation. The subjects behaviour may change to address these specific matters.
There are occasionally questions of vicarious liability involved in accompanying some agency staff, particularly police, in their work routines. This may make observational work with
select groups difficult.
Observational work is time consuming. A level of trust must be developed between the observer and the observed before actions are ‘normalised’. Even when trust has been
developed, the observations generally focus on the actions of one person or a small group of persons rather than the entire program.
- Carefully select the tasks to be observed. Observational work is time consuming and intrusive.
- Be prepared to sign a waiver absolving the agency of any liability in the case of injury.
- Inform the worker that you are not assessing his/her performance but trying to understand how the program or process works. Ensure the worker that any observations on activities will
not be attributable to the individual worker.
- Observe only. Avoid participating as it is the worker’s actions that you are there to observe. Avoid making comments which may suggest what the worker should be doing or
any comment which may antagonise the offender/victim/witness or citizen.
- Do not actively take notes. This will be very off-putting to the worker. Try to remember what is said and done and make notes later.
- Ask approval before viewing files or entering sensitive areas. You are a visitor in the workers space and routine.
Make an appointment with the supervisor or manager to introduce the objectives of the evaluation and the reason for the request to observe.
As we have seen, there are a variety of methods or research tools to assist you in gathering information. It is unlikely, except in major evaluation projects, that you will use all
types of data gathering.
So, in deciding which approach to take, consider the following.
- What are the skills of those who are working on the evaluation? Some people may be quite comfortable in front of a group running a focus group. Others may be more comfortable with one
on one interviews.
- What information already exists as a normal part of operations? If data gathering is necessary, what inconvenience and resistance would this receive from the workers?
- What funding do you have? Some approaches cost more money than others.
- What is the cultural context of the evaluation. Do the majority of citizens have telephones? Is everybody in the community comfortable communicating in the same way and use the same
dialects and can equally participate in a survey?
What are the skills and preferences of the subjects? Some people may be unwilling to give an opinion on issues in a public way, others may be very nervous with a one on one
interview.
Where possible, use more than one information source for the data which is used in the evaluation. This will help to demonstrate that the conclusion of the evaluation report are well
founded.
We have now reviewed the types of information which you may find useful in undertaking your evaluation. It is unlikely that you will use all these types. Only in the most
comprehensive (and well funded) evaluation would you employ all these types of information gathering tools. You must consider the size of the program you are evaluating and the issues
which it is designed to address to determine which approach or approaches is required to obtain the information which you need.
An important point to remember is to consider the culture and characteristics of the community and the preferences of those who may be asked to provide information. The information
gathering tools must be tailored to these needs. Some communities may not have access to telephones, may use primarily oral forms of communication and some people may be reluctant to
share their views in public.
You have taken a lot of time and effort to get an understanding of what are your options for conducting your evaluation. You now have sufficient information regarding approaches to
assess where you are in the evaluation planning.
At this point in your planning:
- It has been decided that there is a community concern which must be addressed.
- Either the program planning accommodated an evaluation or it has been necessary to review the program and begin the evaluation planning after the introduction of the program.
- You have reviewed the original proposal and program action plan.
- You have conducted a search of information sources such as libraries, internet, contacts in local agencies to obtain information concerning the issues with the program is designed to
address.
- You have reviewed available literature regarding the operation of similar programs in other communities.
- You have reviewed information regarding the evaluation of similar programs.
- You have spoken with the major stakeholders to gain an understanding of how the particular program was intended to operate, and,
- You have reviewed the methodologies for undertaking evaluations and understand the options, benefits and disadvantages of the various methods.
A program has been developed and implemented.
You are now ready to develop a evaluation plan....................
Preparing A Work Plan For The Evaluation and Undertaking the Work
Having gathered the relevant material regarding your options you, or if you are not working alone, the co-ordinating group, should meet to develop a specific action plan.
The plan template on the following page will provide some guidance for structuring the evaluation approach.
This is also a logical place to consider the budget which you require to undertake the evaluation. The plan will either assist you in preparing a proposal or help you in deciding which
aspects of the plan will have to be sacrificed to meet budget restrictions. If the assigned budget is low, you will have to decide whether the absence of funding and the corresponding
reduction in potential sources for information will impact the validity of the evaluation to the point that continuation is futile.
Evaluation Plan:
Program title:
Brief description of the program based upon;
What( problem is the program intended to address)?
What (is the objective of the program)?
What (are the indicators)?
Who (is the program directed at, the subjects)?
Who (is expected to benefit)?
When (is the program intended to bare results)?
Where (is the program supposed to have impact)?
How (is the program intended to have impact)?
What is the form of the evaluation?
(Needs Assessment)
Process evaluation
Impact evaluation
What type(s) of data should ideally be used?
|
Source
|
Qualitative
|
Quantitative
|
Detail specific
information required
|
Currently available - where?
|
|
Statistics
|
|
|
|
|
|
File Review
|
|
|
|
|
|
Surveys
|
|
|
|
|
|
Focus groups
|
|
|
|
|
|
Personal interviews
|
|
|
|
|
|
Observations
|
|
|
|
|
What additional information is needed?
|
Possible information
|
Detail specific
information required
|
Prospective source
for information..
|
|
Statistics
|
|
|
|
File Review
|
|
|
|
Surveys
|
|
|
|
Focus groups
|
|
|
|
Personal interviews
|
|
|
|
Observations
|
|
|
Who is responsible for finalising the development of the data gathering tools?
What is the time schedule for finalising the data gathering?
Who is responsible for each type of data gathering and analysis?
Who is responsible for preparing the draft and final reports?
What is the schedule for undertaking the evaluation and has a project tracking sheet been developed ?
Who will be receiving copies of the report?
Who is the principal contact for inquiries regarding the evaluation?
The plan should become the overall guide for the evaluation. The plan can become the front sheet for the co-ordinators file in which the many components of the evaluation will be
collected.
Each of the tasks should be assigned to a member of the co-ordinating group unless the responsibility is to be assumed by the sole evaluator.
Where, for example, a survey must be developed or a data gathering form has to be designed for statistics, this task should be assigned to a lead person who may engage others to
assist.
In complex, multi-task evaluations it may assist the co-ordination if the work plan is put onto a schedule plan. There are several sophisticated project tracking tools available such
as, Gannt or PERT charts. Many of these project tracking tools are available for computers but most basic evaluation plans can be managed manually.
The important things to track are the tasks, with the person responsible (on the left side of a chart) and the time scale (along the top of the chart). Beside each task draw a line
under the appropriate time period to show the start, duration and end of the task.
The success of the evaluation is partly dependent upon the perception of the evaluating staff by the subjects of the evaluation and other stakeholders. Those undertaking the work must
not only have a workable, valid plan, they must approach their work in such a away that they encourage the trust and support of those who are asked to participate in the evaluation.
An ethical approach to the work is important. All respondents must be aware of the nature of the evaluation in which they are to become involved. They must also understand for what
purpose the information is being gathered.
The confidentiality of information is a foundation for the evaluation. This is particularly important in small communities.
Some cultures respond in different ways to various approaches. Some approaches which require open, uninhibited discussion may not be suitable for some participants. People respond
differently to gestures and body language depending upon their cultural background. There are fundamental differences between aboriginal and non-aboriginal cultures and between aboriginal
cultures. Be respectful of people’s heritage.
If you are a visitor to an organisation for the purposes of undertaking the evaluation, be as unobtrusive as possible and do not disrupt routine.
It may be necessary for the co-ordinating group and even for the individual evaluator to develop terms of reference which establish guidelines for the conduct of the work.
- All participants are informed of the confidential nature of the work.
- The confidentiality of any information gathered is honoured and is seen by everyone involved to be honoured.
- The approach of the evaluator must be seen to be objective, comments concerning the program should not reveal opinion which may appear to bias the findings.
- Negotiation of access to data or arrangements regarding interviews or observations of job tasks must be carried out in good faith. Commitments and promises must be honoured.
- Any information which is included in any interim or final report is revealed in aggregated form, that is the response of any one individual is not revealed.
Records, notes from interviews or other documentation are kept in a safe and secure location.
The evaluator or the co-ordinating group must closely monitor the evaluation process. Adjustments may have to be considered if one aspect of the evaluation fails to achieve the success
that had been hoped. Rigidity in the work plan may impede the accomplishment of the objective of the evaluation.
The co-ordinator or co-ordinating group play an important role in ensuring that the work is progressing as planned. Periodically, meetings should be held to,
- discuss the progress of data gathering and whether any adjustments have to be made to the schedule or methods,
- discuss the data gathered to determine potential interim findings.
to ascertain that all prospective subjects or contacts for the evaluation are assisting as hoped or whether facilitation is required, and,
By now you will have collected piles of information from your different sources. You may have interview notes, and numbers drawn from police records, notes from focus groups and,
perhaps, even survey results. Confusing and overwhelming!
You must now find meaning in the data which you have gathered. Depending upon the size of the evaluation project you are likely to have collected large amounts of information.
Summaries which describe aspects of the program, charts to illustrate processes or statistics will help in the analysis of the information gathered.
Keeping in mind the objectives of the evaluation, collect and consider the groups of information which you have collected. Sometimes even writing the evaluation objective(s) on a large
piece of paper on the wall will assist the process of focusing on the objective while sifting through the material.
The information gathered may have to reorganised “out of” the form in which it was collected. Material gathered in interviews will be combined with information collected
from files, statistics may tie-in to the results of the survey or your observations.
To assist this analytical process it may help to see the complete program in terms of input, process and outputs.
Inputs - Refers to the participants in the program, the persons or issues which are meant to be addressed, the skills, equipment and structures which are providing support for the
program.
Process - Refers, as the name implies, to the activities which occur in the process of operating the program. This information relates to the delivery of the program and
considered factors such as the number of clients, the number of calls, how files are maintained, the treatment of victims by agency workers and other activities which move the program
through its intended course.
Outputs - Outputs focuses upon the impact of the program and should include issues such as whether the number of target incidents declined, whether the participants are satisfied
with the treatment, whether the staff were trained and, at a general level, whether the community ‘feels’ safer.
In our example,
Just to look at one aspect of the whole domestic incident issue, what occurs at the scene,
Information gathered from interviews and policy review, to determine,
training programs,
official status given to volunteers and the public perception of that
the way offenders/victims respond,
the communities reaction to domestic violence as an issue,
the availability of alcohol in the community,
employment figures and, therefore, the presence of partners in the home,
preference given to arrest rather than mediation in the police modus operandi,
and others,
all influence what actually occurs at the scene. Consider the cross use of information sources and very broad possibilities of what might influence outcomes.
Getting started
- Those people co-ordinating the evaluation should meet. The evaluation will benefit from a wide range of views of those involved in data collection. The evaluator working alone may
still benefit from discussing the information with another person. This confidante should not be a subject of the evaluation and must understand the requirements for
confidentiality.
- Arrange the information in groups of data such as the input, process and outputs suggested above.
- Clarify the information, assess its relevance to the objective of the evaluation.
- Review the information available and ask whether there is sufficient data to answer the evaluation questions.
- Look for clusters of information or patterns, trends or themes in the data.
- Look for statistical information which may indicate changes in the frequency of occurrences as a result of the crime prevention program.
- Use a flip chart to note themes and list relevant information and information sources under each theme.
- See if there are any conclusions which can be drawn from the information. Start with very basic questions such as
- Has this program worked?,
- Why or why not?
- What pieces of information lead to this conclusion?
- Does any data conflict with this or these conclusion(s)?
- Is/are the conclusion(s) correct or is there some other or additional conclusion which can be drawn?
- List the reasons for your conclusions and follow the thread back through your data and the outputs, process and inputs to understand the various reasons for the success or failure of
the program. Look for a logical chain of evidence to account for your belief.
- If the information is the basis for a major conclusion, establish that it is reliable and supported by other data possibly from another information source.
- Again, review the information available and ask whether there is sufficient data to answer the evaluation questions.
- Consider whether it is necessary to return to the data gathering portion of the evaluation and seek clarification of any issues, do you need more interviews?
- Ask whether there are there any other reasons for the conclusion which you have drawn. Keep your mind and options open! Even consider possibilities outside the original breadth of
the evaluation.
- Determine whether there is any inconsistency or any bits of information which contradict your conclusions and should the conclusion be changed, explain this apparent contradiction.
- Review the information which you have not actively used in drawing you conclusions. Determine whether there are any additional conclusions which could be drawn from this material or
whether the information supports any other conclusion.
- List your conclusion and discuss ways to address these in the form of actions to be taken. These will be your recommendations. The conclusions are only a starting point for
action.
- Summarise the findings with clear explanations for the conclusions.
Set aside preconceived ideas about the success or failure of the program being evaluated.
In our example,
Again looking at what occurs at the scene of a domestic disturbance, stats. analysis may show that the difference between the baseline data and the culminating data indicates that,
since the introduction of the Victim Services Unit, the number of arrests for assaults arising out of domestic incidents have declined by 14%. This is interesting but when we look closer,
we note that the arrest of females as offenders has actually increased. Why might this have happened?
A review of the files of the local police detachment showed a policy which required officers to arrest the male in the household if there was any accusation of an assault having
occurred, no matter who had been assaulted. Interviews showed that many officers arrested the male even when the evidence was not strong. Some arrests of males occurred even when the
female had committed the assault. Officers said that they believed the policy required them to do this. They further believed that the removal of the male would resolve the immediate
problem.
A review of the Unit training shows that unit members were not aware of the policy directed to the police. When they attended the scene of a domestic dispute they were, therefore, not
bound by the same perceived requirement and counselled or advised the parties as they deemed necessary. If the police arrived later, they were often provided a different perspective on the
incident by the Unit member. As a consequence, first, there was less likely of an arrest as the Unit members were not able to arrest and focused on resolution. Second, if there was an
arrest, it was less likely that it was the male that would be arrested, and, third, perhaps, more likely that the female would be detained.
One of your recommendations may relate to clarification of the policy.
The report informs the stakeholders of the results of the evaluation. The report is important because it will be used as a basis for major decisions regarding the effectiveness of the
staff involved, the success of the program in meeting the original objectives, the adjustments which may be required to refocus the program and even continuation of the program.
A draft report should be prepared for discussion with major stakeholders. This serves several purposes,
- It allows the evaluator to collect his or her thoughts and to organise them in a logical fashion prior to the final version.
- Discussing the draft with the major stakeholders may permit the inclusion of an appendix in the report which outlines remedial steps which have been taken to address matters raised in
the evaluation.
It permits discussion of the findings with the major stakeholders so that they can offer an opinion on the principal findings. The stakeholders may provide a different perspective on
the findings which may point to a failing in the research approach.
Summary - This will provide a brief program description, synopsis of the objective of the evaluation, the principal findings and the sources upon which the consultations are
based. The summary or executive summary provides an easily accessible description for those who have neither the time nor the need to review the full report.
Introduction - Describes the goals of the program, the purpose of the evaluation, the duration of the work and the staff involved.
Methodology - Describes the process for conducting the evaluation including your sources of information. This will not refer to the people who supplied information but the
general information gathering approaches which you used.
Principal findings - An evaluation report must include description and judgement. Describe what was found in the evaluation, what conclusions were drawn from the study and the
reason for these conclusions.
Recommendations - Provide suggestions for ways in which the program can be changed to make it more effective. This should also be a proposal for actions which should be taken to
address the original community crime issues. The recommendations make the report a living document which creates expectations for action by those responsible for, or involved in, the
program.
Prioritise the recommendations and suggest the action required to resolve the issue and the person or position responsible for the actions. Remember, the report is not to highlight
blame but to provide direction to ensure that the community crime problem is resolved.
Conclusion - The summary rounds off the evaluation project by briefly connecting the main themes of the evaluation and the resulting report and tying these to the conclusions.
Bibliography - Credit should be given where the evaluator has referred to previously published material. The bibliography also demonstrates to the reader the breadth of
background work undertaken in the course of the evaluation. Style manuals are available to provide the format for listing reference works, however, the most important thing is that
documents, including files and unpublished reports are listed.
Where confidentiality is not an issue, consider recognising those who contributed to the evaluation by providing information or input.
Following the release of the final report, the evaluator and the stakeholders or agency workers may wish to meet and discuss the findings. The feedback session will allow staff who have
been involved or certainly are acquainted with the conduct of the evaluation to receive feedback from the evaluator.
The feedback session should:
- Be based upon an informed understanding of the contents of the report. Consider distributing a synopsis of the findings prior to the meeting.
- Be facilitated to prevent the session becoming an assault on the evaluator if the report is critical
- Not assign blame to anyone who may have inhibited the effectiveness of the program
- Be constructive and positive by explaining weakness but also offer solutions through recommendations.
- Allow staff or stakeholders to respond to the findings of the report by focusing on actions for improvement rather justifying past behaviours
Be structured rather than a free for all.
Recognising the degree of confidentiality and sensitivity of the Evaluation Report, there are several prospective recipients of copies of the report findings.
Who receives a copy is dependent upon the amount of local interest in the program being evaluated. This interest may be broadened by the nature of issues being addressed in the
findings. For example, if the evaluation highlights controversial issues such as ‘staff inadequacies’ then the findings become more sensitive and public and media interest in
the report may increase. Be prepared to support your conclusions in a public forum.
An evaluation of a pilot crime prevention program may generate significant interest outside the local area and may result in requests for copies of the final report.
- The Program Manager for Program Staff.
- The Governance authority such as the Board of Directors or the Police Services Board.
- The Band Council, the municipal, provincial or federal agency with prime responsibility for the action of the agency.
- The local media.
- Local libraries or academic institutions.
- Presentation at a conference so that other communities may benefit from the experience of your community.
- Publication in a magazine or journal or discussion of the findings on local radio.
The funding agency.
However, remember the more copies of a report that are required the greater the cost. You may consider only distributing an executive summary to certain people. Permission may be
required from the holder of the copyright prior to distribution of the final report of the evaluation.
The preparation and presentation of the final report does not necessarily mark the end of the project. Follow-up is required in several areas.
Follow through of recommendations - If responsibility for recommendations has been assigned, it may be the evaluator’s task to change roles and ensure that recommendations
are addressed.
Media contacts - The media may request access to the evaluator to discuss the findings. The media can also be incorporated in generating interest and action as a result of the
report. You should consider an item in the community or Band Council newsletter.
Thanks - Depending upon the breadth of the evaluation, many people may have participated in the process by supplying data, allowing the evaluator to observe their work routines,
participating in focus groups or providing insight to the evaluation process. Although it may be impossible to contact everyone who participated in the evaluation project, a simple note
of thanks with a mention of the outcome of the work may leave participants with a positive disposition towards participating in future evaluation projects. A community feast or social may
be appropriate if a large portion of the community has contributed to an evaluation.
Self-evaluation - For many, the co-ordination of an evaluation project is a new experience. The evaluator is required to learn different skills and to become acquainted with
terms and practices which are new. The evaluation project is often stressful and time consuming. Since the continuation of a program, along with jobs, is often dependent upon the findings
of an evaluation the task is also one which is associated with significant responsibility.
The process of program development, program operation evaluation and feedback is really a circular process. The recommendations will feed into agency or community decisions regarding
which crime problems to tackle and how to develop a program.
Well done! You have completed the evaluation. Congratulations. Your work has probably made a significant contribution to the community and, possibly, will also have an
impact in other communities.
At the end of the project, you should allow time to reflect upon the process.
Consider:
Salaries: If not volunteer or regular agency staff, researchers salaries, honorarium, consultants fees, clerical or administrative salaries.
Accommodation: Rental costs for office or focus groups
Supplies: Office materials including, paper, printer cartridges, writing tools, flip chart paper, file space, storage
Office equipment: Fax, computer, telephone,
Other equipment rental: computer, touchpad technology for focus groups.
Operational: telephone particularly long distance, e-mail, Internet access fees, postage/courier for documents from/to other communities, computer time for data analysis.
Travel: Visits to other communities, travel around your community, travel by others such as peer evaluators or consultants.
Printing: Surveys, correspondence, draft and final reports.
Tasks to do
|
Person
responsible
|
Scheduled finish date
|
Done
4
|
|
Organise co-ordinating committee (or evaluator working alone)
|
|
|
|
|
Was an evaluation part of the original program plan? What was planned?
|
|
|
|
|
Define the program Objectives.
|
|
|
|
|
Define the Evaluation Objectives (same?).
|
|
|
|
|
Establish the type of evaluation you need?
Process? to find out about the operation of the program.
Impact? To understand the final effects of the program upon the community problem
(Remember, a Needs Assessment is usually something that is done before a program is developed to determine what type of program is required)
|
|
|
|
|
Determine who should do the Evaluation.
Evaluator only
Program manager
Program staff
Community member
External consultant
Peer from another community
|
|
|
|
|
Determine whether other resources needed.
People
Funding
|
|
|
|
|
Determine possible sources and apply.
|
|
|
|
|
Has the project plan form been prepared?
|
|
|
|
|
Identify the most appropriate types of information gathering, taking account of the program issues, personality of those involved and the culture of the community.
Statistics
File Review
Survey
Focus Group
Personal Interview
Observations
|
|
|
|
|
Detail information sources, the agency and contact person + check if approval is required
|
|
|
|
|
Assign responsibility for each inf. gathering task taking into account the skills of the people who are working on the evaluation.
|
|
|
|
|
Assign timelines for each stage of the evaluation and enter on the project schedule.
|
|
|
|
|
Undertake research, gather all information.
|
|
|
|
|
Meet to analyse all information.
|
|
|
|
|
List conclusions with all the relevant support information.
|
|
|
|
|
Prepare draft report and discuss with the principal stakeholders.
|
|
|
|
|
Prepare final report with recommendations.
|
|
|
|
|
Distribute copies of final report.
|
|
|
|
|
Conduct feedback sessions with staff.
|
|
|
|
|
Send thanks to participants.
|
|
|
|
Analyse -To examine information to show its essence or meaning.
Baseline data - Information that is already known. A point from which to measure, a standard. The state of affairs before the program began.
Before-and-After Comparison Measures variables like ‘crime rate’ or ‘levels of fear of crime’ before the program implementation and compares
those figures to the figures collected once the program has been established or completed at any given point. Also called ‘pre-and post-’ comparison.
Closed Questions -Questions which are answered with a ‘yes’ or ‘no’ or some other question with a restricted range of options as answers.
Culminating data - The information (statistics or other forms of data) which are collected at the termination of a program and, compared with baseline data, used as a basis for
determining the impact.
External Evaluation - An evaluation conducted by someone who is not considered a community member or program participant, usually considered an outside expert evaluator.
Evaluation - To examine and judge based upon performance, results and /or impacts. Evaluation allows you to find out how valuable something is but more so it provides
you with meaning. It's empowering in that it equips you with information that can allow you to make sound judgements and decisions. Evaluation is an important tool for managing service
delivery.
Evaluation Approach - Outlines how you plan to carry out an evaluation.
Feedback - Information that is provided back to the group of people such as agency workers whose agency has been evaluated. Feedback provides information regarding the
assessment of what is going on and what they have been doing.
Focus Group - A small group meeting intended to facilitate the discussion of specific points relevant to the evaluation by people with an active interest in the evaluation
issues.
Formative Evaluation -Used to make improvements or changes to a program while it is in progress. Questions asked are: What is working? What needs to be improved? How can it be
improved?
Needs Assessment - The preliminary stage of setting up a program to develop and understanding of the breadth or structure and form of a required program based upon such tools as
an environmental review, and problem analysis.
Impact Evaluation - The assessment of the effect of a program, the degree to which the program has changed the circumstances which is was set out to address. The Impact
evaluation may also go broader into other unanticipated effects.
Indicator - A suggestion, like an event or statistic, used to measure the success of a program in meeting its objective.
Literature Search - The review newspaper articles, periodicals, academic journals, reports secured from other agencies and Internet materials relevant to the crime prevention
program operation or evaluation to secure background information in preparation for the development of your plan.
Objective - An objective is a goal you set which you try to achieve. It is a statement of what you want your program to do.
Open Questions - Those questions that can be answered with description and opinion.
Planned vs. Actual Performance - A project is evaluated against its objectives. Therefore, it compares what it has set out to do with what it actually has done.
Process data- The information which is collected over the duration of a program either as a normal part of business or as a consequence of a data gathering tool for the purposes
of the evaluation.
Process Evaluation - Involves self-examination or assessment of the organisation’s structure, how it runs, who is involved, board structure and operational issues.
Qualitative Data Basic attributes or features of a program usually expressed by opinion regarding the quality of the program. Looking at the meaning and structure of a program
("soft "data).
Quantitative Data The measurement of a program in tangible term, using numbers/statistics ("hard" data).
Self-evaluation - Evaluation conducted by individuals working in the program; a participant in the program; people who belong in community but not necessarily working in the
program; a stakeholder in the program.
Summative Evaluation - An assessment used to judge the usefulness or worth of a program and is conducted near the end of a program. This can be done at a community level where a
single project is evaluated or it can be done at a regional or national level whereby the impact of all projects are compared such as the Aboriginal Head Start evaluation.
Terms of Reference - The Rules of Conduct, the evaluation parameters, a road map for the evaluation that will be conducted which outline the obligations of all participants in the
evaluation. They should reflect evaluation objectives and priorities and should indicate the kinds of general questions the evaluation should cover such as a evaluation objectives, time
frame for evaluation, resources to be committed, list of deliverables.
How To Use The Manual.
What is the purpose in undertaking this evaluation?
Given the progress of the program, is it the appropriate time to undertake an evaluation?
Do I have the resources as far as skills and funding?
If I am doing the evaluation alone, can I organise an advisory committee from the community?
Getting Ready.
Undertaking the Background Research
Have I fully described this program as it is intended to operate in this community?
Have I defined in a concise way the objective(s) or goal(s) of the program?
Have I described the important community factors which may influence the operation of this program?
Do I understand if, and how, the program operates in other communities?
Setting Evaluation Objectives:
Have I described the objectives of the program or, if these were not specified in the original program outline, have I developed objectives for the evaluation?
Have I determined what types of information will be used to indicate whether, and how, the program has worked?
Deciding the Form of the Evaluation Program
Given the nature of the program and the stage at which I am planning the evaluation, which approach or approaches will best allow for an examination of the effects of the program?
Deciding the Type of Information Upon Which Your Evaluation Will Be Based
Considering the nature of this crime prevention program and the sorts of information which might be available, which approach or combination of approaches will provide the best
foundation for an evaluation?
What sorts of information are likely to be available in this community?
Given the type of people in my community and/or the staff working in the program, how am I likely to get the most valuable information?
Where Do We Find The Information That We Need?
Statistics research
Given the objectives of this program what sort of stats information do I need?
Do any of the community agencies, such as police or the Band Council, compile stats. which may be useful in this evaluation?
If the stats which are kept are not entirely suitable, is it possible to develop a process to collect stats. information for the purpose of this evaluation?
File Reviews
Given the objectives of this program what sort of file information do I need?
Do any of the community agencies, such as police or the Band Council, keep file information which may be useful in this evaluation?
If the files which are kept are not entirely suitable, are the required files available in any other location?
Do I need any sort of security clearance to have access to this material?
Surveys
Given the issues being addressed in this program and considering the costs of a survey, will a survey provide information which is important to this evaluation?
Considering the characteristics of this community, is a survey of any sort practical?
If so, should this be telephone, mail-out or distributed in some other way?
Will the questions be open, closed or multiple choice?
Focus groups
Given the nature of the issues addressed by this program would a focus group be an acceptable way to examine peoples views?
Given the characteristics of this community including matters such as the culture, would a focus group provide an open exchange of ideas?
If so, what is the best make-up of the group to encourage an open exchange?
Who do I have working on the evaluation who has the skills necessary to conduct a focus group?
Personal Interviews
Given the nature of the issues addressed by this program would personal interviews be an acceptable way to examine peoples views?
Given the characteristics of this community including matters such as the culture and the personality of possible respondents, would a personal interview provide information valuable to
the evaluation?
If so, who should be interviewed to secure the information I need?
Who do I have working on the evaluation who has the skills necessary to conduct interviews?
Observations-
Given the types of issues addressed by this program and the specific nature of this method of information gathering , will observational work contribute information valuable to the
evaluation?
If so, which places in the process of agencies dealing with these issues would be most appropriate to undertake observational work?
Do I have the skills to do observational work?
Will I be able to get approval to do the work?
What skills do I have or the people working with me and what method of information gathering will work best?
What information was used in other communities where a similar evaluation was undertaken?
What types information will best address the objectives which were set for the program and which I have set for the evaluation?
Considering the indicators that I have decided are ideal to evaluate the program, where can I find that information?
Does the information already exist?
If that specific information is not readily available, is it feasible to develop the methods to gather the information, do I have the skills or the money to successfully complete this
task?
Is it reasonable to expect agency workers to collect this information in addition to the other data that they collect as a routine function of their job?
Given the particular nature of this community what types of information gathering will yield best results?
If it is not feasible to collect the information, how can I change the evaluation plan so that the end result of the evaluation is still valid?
Planning Your Work and Working Your Plan.
Have I prepared a work plan, and decided
which types of information I will need,
where I can get that information,
who is responsible for developing the information gathering tools,
who will conduct the work, and
............with people and time lines scheduled?
Informed everyone involved of the requirements for confidentiality and cultural sensitivity?
Begun gathering the information which will be the basis for the evaluation?
Making Sense Of The Information Gathered
Have I met with my co-ordinating group and sorted the information to themes or issues and listed the particular information or evidence which supports these findings?
Am I satisfied that my conclusions are supported by the data?
Depending upon the particular information gathering tools that I used, is each conclusion supported by various sources such as interviews, surveys and file review?
If there are inconsistencies, can I explain why?
Are my conclusions clear and precise and logical?
Have I addressed the evaluation objective and focused on the indicators?
Are my recommendations do-able and is responsibility assigned?
Preparing The Evaluation Report
Have I prepared a draft report which is based upon the findings of the evaluation and the discussions at the group information analysis and structured the report according to the
suggested format?
Have I discussed the draft report with major stakeholders and considered their feedback?
Have I decided upon the distribution list for the final report and made sufficient copies ?
Follow-up
Did I conduct this evaluation in a fair and objective way?
Did I treat all the participants fairly and equally?
How can I improve from this experience when I undertake the next evaluation?
How can my work on this project reap the greatest benefit to my community and to crime prevention?
............Now take action to improve the program and your community!
Department of Indian and Northern Affairs, First Nations Child and Family Services, Self-Evaluation Guide, undated.
Federal-Provincial-Territorial Working Group on Community Safety and Crime prevention, Models of Practice, undated.
Government of Canada, Treasury Board, Guide on Program Evaluation Function, 1981
Guyette, S., Community Based Research: A Handbook for Native Canadians, University of California, 1983
Holt, J., A Handbook About Project Self-Evaluation for First nations and Innuit Communities, Department of Health and Welfare, 1993
Industry Canada, Focusing on results: A Guide to Performance Measurement, Discussion paper, 1995.
McCaskill, D. And Williams, A. Community Based Evaluation Workbook, Aboriginal Healing and Wellness Strategy, 1997
Ministry of Health and Welfare, A Handbook for First Nations on Evaluating Health Programs, undated.
Oppal, Justice W., Inquiry on Policing in British Columbia, Ministry of the Attorney General, BC, 1994 (for examples of surveys).
Prairie Research Associates, Building a Safer Canada: A Community-based Crime Prevention Program Manual, Ministry of Justice, Ottawa, 1996
and, for statistics, publications such as,
Howell, David, Fundamental Statistics for Behavioural Sciences, Dewberry Press, Belmont, California, 1995.
