|
IN THIS ISSUE:
Message From the Senior Director
Before we head into a much-deserved holiday break, I wanted to take this
opportunity to share a few projects and events of interest to the community.
It has been a busy six months for the group at the CEE. Starting in January,
with the launch of revised guidance for RMAFs, we’ve held 14 information and
training sessions for our colleagues at TBS as well as throughout all
departments and agencies involving over 800 managers evaluators and analysts.
Our website now contains all material that was covered for these
sessions, as well as a guide that outlines a strategic approach to developing
RMAFs.
(please see: www.tbs-sct.gc.ca/eval)
At the end of April, a Heads of Evaluation meeting was held at the Canada and
the World Pavillion. Three deputy ministers and ADMs joined us for the day to
discuss the positioning of evaluation and better ways of engaging DMs in
evaluation. Highlights of this meeting are included in this newsletter.
And finally, our Senior Advisory Committee is finalizing three research papers
that will further support the future direction for evaluation within the federal
government. These research papers focus on the role of evaluation in decision
making, furthering the professionalization of the evaluation function, and
approaches to DM engagement. These reports will soon be available on our
website.
Over the summer months, our group will remain busy on several projects that
we hope to roll out to the community in the Fall. These projects cover issues
such as RMAF implementation successes, the impact of evaluations on decision
making, developing tools for value-for-money reporting, and developing
frameworks for meta-evaluation research.
This will be the final newsletter before the Fall season. Please let us know
what you think by contacting me at (613) 952-7447 or Hunt.Terry@tbs-sct.gc.ca.
Have a safe and relaxing summer holiday!
Terry Hunt ,
Senior Director,
Centre of Excellence for Evaluation
An interdisciplinary team of researchers at the University of Ottawa have
received SSHRC funding for a 3-year study of evaluation capacity building in
Canada. The team will work cooperatively with Treasury Board’s Centre of
Excellence for Evaluation and the Canadian Evaluation Society to explore not
just the capacity to do evaluation, but the capacity to use it.
The research team is comprised of Brad Cousins (Faculty of Education), Swee
Goh (School of Management) and Tim Aubry (School of Psychology) in collaboration
with Bob Lahey (retired: former Director of CEE) and Steve Montague (Performance
Management Network). Catherine Elliott (Faculty of Education) will serve as
graduate assistant on the project.
A range of research strategies are planned over the next 3 years, the first
being a concept mapping study in the federal government sector designed to
deepen understanding about the conditions under which evaluation is likely to be
used. To follow will be questionnaire survey and case study initiatives in
government and non-governmental evaluation contexts. Look for more information
about the concept mapping study in the next issue of the Newsletter. For more
information, contact Brad Cousins at bcousins@uottawa.ca 613-562-5800 ext. 4036.
The Treasury Board of Canada Secretariat’s Evaluation website appears to
have struck a power chord with the Federal Evaluation Community. The website’s
structure and content have been steadily improving over the past few years. As a
result, the Evaluation website is experiencing significant positive growth in
the areas of site usage and traffic.
The Evaluation site is one of the many TBS service websites, and is
maintained and administered by the CEE and RBMD webmasters. It was established
to provide information and resources to support the Federal Evaluation Community
and the implementation of the TBS Policy on Evaluation.
Roughly a year ago, in Summer 2004, the task of refining and enhancing the
site was undertaken by the CEE and RBMD webmasters. This was the second such
website refinement exercise in two years. A new information architecture was
developed to allow for a more intuitive user experience. Our online information
and resources were catalogued, reviewed, refined and then reorganized in a more
“logical” breakdown of sections and subject pages. The emphasis of this
website upgrading was ease of access for the user to locate the desired
information contained in the site. Quick access links from the home page were
added in the shape of the What’s New listings (a chronological list of the
most recent additions to the site) and the Quick Links box, which features links
to the most frequently accessed documents, such as the Evaluation Policy.
|
|
January 1 to June 23
2004
|
January 1 to June 23
2005
|
Difference
|
Visits *
|
61,084
|
106,885
|
+74.98%
|
|
Average visits per day
|
349
|
614
|
+75.93%
|
|
Average length of visit (min.)
|
17:40
|
15:38
|
-11.51%
|
|
Median visit lengths
|
3:42
|
5:45
|
+55.41
|
|
International visits
|
56.05%
|
70.93%
|
+26.55%
|
|
Successful hits for entire site
|
629,989
|
1,171,330
|
+85.93%
|
|
Average hits per day
|
3,599
|
6,731
|
+87.02%
|
|
* Visits - Number of visits to this page. A visit is a series of
actions that begins when a visitor views the first page from the server,
and ends when the visitor leaves the site or remains idle beyond the
idle-time limit. The default idle-time limit is thirty minutes.
|
In a comparative look at the first halves of the past two years the numbers
demonstrate that traffic and website usage overall have increased significantly.
Along with the increases in overall traffic and site
usage, there were some other interesting numbers. One such statistic is the
increase in the number of international visitors to the website. This may be
due, in part, to the outreach activities that the CEE has undertaken with
exchanges between international governments on the subject of Evaluation.
Another is the average length of the visits to the site. At first glance it may
look like a negative statistic, however with increases in traffic, this number
might also suggest that website users are finding what they are looking for in a
shorter time frame.
If anything, the numbers do indicate overall growth for TBS’ Evaluation
website, and that previous efforts to improve the site have produced desirable
results. As we approach the summer months, we are once again looking at the
Evaluation site to further enhance the website’s structure, content, and user
satisfaction. For example, one of the common criticisms about the site is that
unless a link is sent out, it is difficult to access if the user is reliant on
search engines or the TBS website. While we have made efforts to promote the
Evaluation’s URL (web address) by including it on most of our materials, this
summer’s exercise will aim to find ways to improve ease of access to the site
itself.
You, the Evaluation Community, are the primary client users of the Evaluation
site, and as a result, the primary drivers of where our efforts to improve the
site are focused. To this end, we’d like once again to invite your comments
and suggestions for improvement.
Please send your comments to Ross Henwood at Henwood.Ross@tbs-sct.gc.ca, or
to 952-1546. With your help, and our continued efforts, we can ensure that the
TBS Evaluation service site continues to be a number one hit with the Evaluation
crowd.
Report for Fiscal Year 2004-05
Centre of Excellence for Evaluation - Treasury Board of Canada Secretariat
2005
Produced by the Centre of Excellence for Evaluation, this report examines the
health of the evaluation
function in the Government of Canada during fiscal
year 2004-05.
Topics covered include:
- An Overview of the Federal Evaluation
Function;
- Evaluation Infrastructure;
- Human and Financial Resources;
- Evaluation Coverage;
- Evaluation Quality;
- Management and Accountability Framework
Review;
- Evaluation Community Needs Assessment;
- CEE 2004-05 Activities to Support the
Evaluation Community.
This is an excellent resource for heads of evaluation and senior officials
who are interested in supporting sound, evidence-based decision making by
advancing the evaluation function. It serves as an excellent reference tool to
compare how your organization is doing in relation to the federal evaluation
function as a whole or to other departments and agencies. For example, there is
information on how units are funded, what services they provide, essential
infrastructure, in-house versus contracting resources, coverage and scope of
evaluation activities. There is also a discussion on how information collected
by the CEE for monitoring purposes is used by the TBS/CEE to support central
agency decision-making.
Copies of this document can be downloaded from the CES website at:
www.tbs-sct.gc.ca/eval/dev/health-santé/hefgc-sfegc_e.asp
A New CEE Guide to Demonstrate Results Reporting of Low Risk Programs
The Centre of Excellence for Evaluation has developed a Results Reporting
Capacity Check (RRCC) to assist departments and agencies in quickly and
systematically assessing the capacity of low-risk programs and initiatives to
manage, achieve and demonstrate results for Canadians. Conducted by the
department’s Evaluation Unit, the RRCC offers an independent and objective
view of a program or initiative’s strategic, operational and results capacity.
The guide provides detailed instruction on how to conduct an RRCC
including key questions, documents to review, and an assessment template to
guide the capacity analysis.
Results of the RRCC provide a level of confidence to departmental executives
and central agencies that a program or initiative has the capacity – as shown
by its performance – to demonstrate and account for results. Positive RRCC
results provide confidence to senior management that performance information is
readily available and or an evaluation could be easily executed. Poor RRCC
results will result in the need for a formal program evaluation.
Departments and agencies will be happy to hear that positive results of an RRCC
can be used to support a request for an exemption to conduct an evaluation for
their “low-risk” grants and contribution programs pursuant to the Policy on
Transfer Payments.
For more detailed information on the RRCC and how it can be used visit:
www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rrcc-dcrr/rrcc-dcrr_e.asp
Contact :
Victoria Carlan, Senior Analyst, CEE
tel.: (613) 941 7193
email: carlan.victoria@tbs-sct.gc.ca
.
The Role of Evaluation in Evidence-based Executive Decision Making
The Centre of Excellence for Evaluation (CEE) sponsored a full-day meeting of
the Heads of Evaluation on April 27th at the Canada and the World Pavilion.
The theme of this meeting was ‘The Role of Evaluation in Evidence-based
Executive Decision Making’.
The day’s agenda included presentations relating to the needs of Deputy
Ministers, evaluation requirements for expenditure review, and the evolving role
of internal audit and its complementary relationship with evaluation.
The agenda was designed to be interactive with presentations being followed
by equal time for questions and commentary from participating Heads of
Evaluation.
Highlights of the Day’s Events:
Keynote Address:
Samy Watson ,
Deputy Minister for Environment Canada
‘What do Deputy Ministers Need?”
Mr. Watson described evaluation as a central function in the demonstration of
accountability within the current government environment of resource
reallocation.
He carried with him a May 1981 TBS publication entitled, ‘Guide on the
Program Evaluation Function’ which he referred to in discussion of the
continued and present need to demonstrate cost-effectiveness and efficiency.
He indicated that deputies should be involved in results-based management
accountability frameworks (RMAFs) and that the evaluation offices should not
work in isolation but supply the expertise in helping address questions of what
it is a department is trying to do. Evaluation should be seen as value
added and not as a threat and should provide clear advice (eliminating the
jargon).
Munir Sheikh ,
Deputy Secretary to the Cabinet, PCO
‘Expenditure Review and the Role of Evaluation’
Mr. Sheikh addressed the expenditure review process and the ranking of
priority of government programs in regard to value for money. He further
explained that such ranking could not be done without the use of evaluation.
He expressed concern however that all program evaluations could not be done
in a timely fashion with only a few programs being evaluated annually. He
indicated that at such a rate it would take many years to evaluate the full
offering of programs available and suggested that all programs need to be looked
at in a preliminary way, within the course of a fiscal year, in determining
value for money.
A revamped expenditure management system was seen to be a requirement in the
review of good value for money along with preliminary evaluations of all
programs with a focus on priorities and efficiencies in the delivery of core
programs.
New Faces of the CEE
The CEE Welcomes Annique Yovetich
A native of London Ontario, Annique has been living
and studying as an Ottawan for the last couple years, and has come to love it
here in the Nation’s Capital. One of four children, she has accepted a student
position and will join the CEE family for the summer, before returning to
Carleton University for her fourth year of study. She is studying History,
Political Science with a minor in French. Annique will be working with Josée
Boisvert and Rebecca Leslie on the ERIC review exercise.
In the Twelfth Annual Report to the Prime Minister on the Public Service
of Canada (ending March 31, 2005), Alex Himelfarb, Clerk of the Privy
Council and Secretary to the Cabinet, highlights a Public Service where program
and expenditure review must ‘become an integral part of how we work’.
( www.pco-bcp.gc.ca/default.asp?Language=E&Page=Report )
He further outlines the Expenditure Review process in a culture of
transformation and resource reallocation where there will be continuous
reassessment of value for money in resource allocation
decision-making processes.
In addition, the first of four established Corporate Priorities for the
Public Service of Canada (for 2005-06) is Management for Results , which
ensures program effectiveness and value of money through the demonstration of
measurable progress against performance indicators.
These statements remind us of the importance of making evaluation relevant in
terms of the expenditure review process: ensuring that the 7 policy
questions of the Expenditure Review Committee are addressed in evaluation
processes, as well as demonstrating that a Department’s programs are geared to
generating results that matter to Canadian and are providing maximum value from
the resources allocated to them.
|
