Valuing the Evaluation Function: Problems and Perspectives

Valuing the Evaluation Function:
Problems and Perspectives

Danielle Labbé
Centre of Excellence for Evaluation
June 1, 2006

Background

Problems

Perspectives

For many, evaluation is similar to internal audit. However, they are two different things.

• The purpose of internal audit is to examine how we do things – it identifies strengths and weaknesses in the management control framework.
   
• Evaluation indicates whether we are doing the right things – and if we are proceeding in a cost-effective manner.
   
• Evaluation pertains to the effectiveness and value of programs, whereas audit covers financial attest, compliance, insurance and risk.

Audit addresses the manner in which programs are delivered, while evaluation indicates whether programs are obtaining the desired value. 

The federal government has a solid infrastructure for evaluation and for internal and external audit.

• All of the main departments and agencies
  - have internal audit and evaluation sections
  - publish the findings of their audits and evaluations.
  
• The Treasury Board Secretariat
  - established the Centre of Excellence for Evaluation in 2001 in order to
        - serve as a focal point for leadership of the evaluation function in the federal Public Service
        - launch initiatives to meet the challenges shared by the community
        - support capacity building, practice improvement, and strengthening of the evaluation community.
  
• The Office of the Auditor General
  - conducts performance audits of departments and agencies.

In 2001, the TB introduced a four-year investment strategy that strengthened the evaluation infrastructure.

2002*

2005**

TBS’s investment has also led to an increase in evaluation products …

… but this trend did not continue when the investment ended in 2004-2005, thus raising concerns.  

If enacted, the Federal Accountability Act will entail even more careful review of program performance …

• Auditor General to investigate how monies awarded under funding agreements with the federal government are used by third parties.
  
• Parliamentary Budget Office to provide analysis to Parliament regarding the nation’s finances and trends in the national economy.
  
• Departments to evaluate, at least once every five years, the relevance and effectiveness of their grants and contributions programs.

… thus taxing evaluation capacity.

Background

Problems

Perspectives

There are some real challenges for the evaluation function.

• Problems with quality, timely delivery and strategic focus complicate the use of evaluation to support decision-making:
  - often focused on small programs and not strategic
  - can take too long to complete and produce results that are difficult to understand
  - can be biased if funded by program managers
  
• Government-wide capacity issues have made evaluation difficult to deliver:
  - lack of properly trained evaluators
  - skills of those leading the evaluation function not consistent
  - definition of evaluation product has not changed in 20 years
  
• The new Internal Audit Policy forces evaluation to rethink its relationship with audit.

Although they often explore the question of resource optimization, convincing evaluations of a program’s value are rare.

• Under the current policy, evaluations must take into account:
  • Relevance
  • Success
  • Cost-effectiveness
    
• Most evaluations focus on program improvement (72%)
  
• Few evaluations attempt to determine whether the program is cost-effective (26%)

SOURCE: * EKOS Study on quality of federal evaluation, 2004

Evaluations are not conducted at a high enough level to properly support decision-making with regard policies and expenditures.

• Half of all evaluations cover programs under $4 million.
  
• 12% cover programs of less than $500 000.*
  
• Deputy ministers want to know about departmental policy and program performance at the highest level.**
  
• Since 1996 the OAG has been recommending evaluation of larger programs of interest to parliamentarians.***

Size of programs evaluated
(Millions $)

Evaluations are difficult to understand and take too long to perform.

• Quality increased by 18% in the three years following the CEE’s creation.*
  
• However, 23% of evaluations are still inadequate.*
  
• Deputy ministers feel that evaluation reports tend to be too long, theoretical and difficult to understand.**
  
• Evaluations are not conducted in time to respond to decision-making needs.**

Number of months evaluation takes 
(Does not include time required for publication approval) ***

The scope of evaluations is limited, thus highlighting the need to adopt new, more appropriate approaches to evaluating the value of programs.

• Only 23% (9) of departments indicated they intended to evaluate all programs over a five-year period.**
  
• These departments are responsible for most transfer payment programs. 
  
• This poor coverage is also related to capacity problems.

Percentage of departmental program spending in a year*

Deputy ministers and deputy heads, TBS and heads of evaluation are concerned about capacity.

• Deputy ministers indicate that the contribution of evaluation is diminished by:
  - a lack of resources and competent evaluators
  - current focus on internal audit 
  - weak linkages with policy and expenditure management.*
  
• For heads of evaluation, access to competent evaluators is a major obstacle.** 
  
• TBS’s need for evaluations on the effectiveness of horizontal initiatives and programs is increasing, but the evaluation function has difficulty delivering the goods.***
  
• Heads of evaluation see TBS’s lack of coordination of horizontal evaluations as a major obstacle.***

MAF findings, 2005-2006 
(36 departments and large agencies)

SOURCES: *Breen Report, 2005; **Report on CEE consultations, 2004; ***2005-2006, TBS Internal Audit and Evaluation, independent CEE study, 2005

Most small agencies do not have sufficient capacity.

• Office of the Commissioner of Official Languages 
• Canadian Nuclear Safety Commission
• Financial Consumer Agency of Canada
• National Energy Board

Small agencies*

• 17% (5) completed evaluations in 2003-2004 
  
• 20% completed evaluations in 2004-2005

DEPUTY MINISTERS doubt the reliability and neutrality of evaluation given the use of consultants and funding through programs.*

Internal vs. Contract Resources**

Major Sources of Evaluation Funding**

The function also needs to be repositioned within the government structure.

• Deputy heads are concerned about inadequate linkages between evaluation and policy and expenditure management.
  
• The governance requirements of the new audit policy raise questions with respect to the position and the governance mechanisms most suitable for evaluation.
  
• Evaluation’s position must enhance access to the deputy minister and ADM community to enable it to play its strategic role in developing and evaluating a department’s results structure. 
  
• There is a lack of connection between evaluation and similar types of performance-related activities, including performance evaluation, reports to Parliament, MRSS and PAA, and strategic planning.

The introduction of the new Internal Audit Policy…

Governance Issues

The majority of departments have 
combined audit and evaluation committees

• Until now, audit and evaluation were governed by the same deputy-led committee.
  
• Under the Internal Audit Policy, audit committees and audit heads did not need to be involved in decision making. 
  
• Departments are seeking clarification as to the role and governance of evaluation.

…encourages departments to contemplate the best way of using and governing evaluation.

Deputy heads are concerned about the integration of evaluation within their organizations …

• There is a variety of reporting relationships across departments.
  
• The majority of departments have combined audit and evaluation, along with combined audit and evaluation committees.
  
• Overall, the ADMs’ commitment to managing and advocating on behalf of the function is low.

Evaluation units in the departments report to various officials

… and the lack of connection with decision making on policies and expenditures.

Predominant Current Approach

• 77% of departments and large agencies use the current model – OAG questioned whether co-location is consistent with ensuring each function’s distinctiveness.
  
• The audit policy requires that this function be more independent, whereas evaluation is a management function that directs policies and spending. 
  
• Combined audit and evaluation function does not reflect new Internal Audit Policy:
  - no Chief Internal Audit Executive
  - no external audit committee.
  
• Lack of alignment of evaluation with performance measurement and strategic planning.
  
• Lack of influence of evaluation on expenditure management and policy decisions.

Background

Problems

Perspectives

Canadian evaluators want changes, but they will be facing some substantial implementation challenges.

• Demographics – High turnover rates will need to be addressed, with a government-wide recruitment target
  
• Workload – Will be difficult to manage without a realistic plan
  
• Skills – Will take time to rebuild training programs and partner with universities
  
• Jurisdictional – Continually reinforcing the distinction between internal audit and evaluation
  
• Relevance – Good quality, timely products need to be used to inform allocation and reallocation decisions

Step #1 is to renew our policy so that evaluation is ...

• better recognized as a core function of public management
  
• an integral part of the decision-making process
  
• related to budgeting and expenditure management
  
• a neutral and independent function in line with the government’s management priorities
  
• the source of a relevant and quality product.

Step #2 is to zone in on implementation.

• Make sure the function is properly resourced
  
• Build capacity by re-establishing training and recruitment programs and partnering with universities on certification.
  
• Guarantee objectivity – it is up to evaluators to fund evaluations
  
• Redefine the “product”  to promote relevance and better coverage:
  - Strategic policy evaluation
  - Impact assessment
  - Evaluation of resource optimization
  - Evaluation of implementation.
  
• Treasury Board Secretariat to assess quality and set out clear competencies for heads of evaluation and evaluators. 
  
• Present a government-wide evaluation plan.

Strengthening the scope depends on the restructuring of evaluation products to reduce usage times and enhance their relevance.

Strategic Policy Evaluation

• Prospective in nature, review of high-level programs in PAA, focusing on relevance, lessons learned, options and potential consequences. Rapid evaluation that combines policy analysis and evaluation methods at a higher level to identify potential long-term consequences and options.

Impact Assessment

• In-depth review of program effectiveness and impacts (both intended and unintended) and alternatives.

• Review of program relevance and performance (cost-effectiveness and overall effectiveness). Tool used would lead to program improvement plan or more thorough evaluation.

• Review of program operation. Focuses on implementation process and on management issues rather than results. Issues include accountability, governance and delivery mechanisms. 

• Supports the delivery of client-centred programs through the Common Measurement Tool and related tools and approaches for monitoring client satisfaction.

ANNEX: Reference Documents

• Government of Canada 
  http://canada.gc.ca/main_e.html
  
• Treasury Board of Canada Secretariat
  http://www.tbs-sct.gc.ca/index_e.asp
  
• The Health of the Evaluation Function in the Government of Canada 2004-2005
  http://www.tbs-sct.gc.ca/eval/dev/health-sant%E9/hefgc-sfegc_e.asp
  
• Report on CEE consultations, Peter Hadwen, 2005 
  http://www.tbs-sct.gc.ca/eval/pubs/repcon-rapcon/repcon-rapcon01_e.asp
  
• Decision-Making in Government: the Role of Program Evaluation, Peter Aucoin, 2005 
  http://www.tbs-sct.gc.ca/eval/tools_outils/Aucoin/Aucoin_e.asp
  
• Professional Ethics and Standards for the Evaluation Community in the Government of Canada, Ken Kernaghan, 2006
  http://www.tbs-sct.gc.ca/eval/dev/career/ethics-%E9thiques/ethics-%E9thiques_e.asp
  
• Review of the Quality of Evaluations Across Departments and Agencies, 2004
  http://www.tbs-sct.gc.ca/eval/pubs/rev-exam_e.asp
  
• Improving the Professionalism of Evaluation, T.K. Gussman, 2005, 
  http://www.tbs-sct.gc.ca/eval/dev/Professionalism/profession_e.asp