The successful monitoring of Treasury Board (TB) policies is critical in
today's environment of rapid change and challenges in addressing policy compliance
and effectiveness. It is essential that effective policy monitoring
practices be in place at the Treasury Board Secretariat (TBS) to ensure the
success of TB policies. This review was requested by the TBS Internal Audit
and Evaluation Committee to provide TBS management with an independent study of
leading practices in the area of policy monitoring. The review was designed
to assess the current status of policy monitoring within TBS and provide practical
examples of leading practices. The recommendations made in this report will
provide guidance for the Policy Review and Reporting Project as it proceeds to
streamline TB policies and reporting requirements.
Leading Practices Research
The review has researched leading practices for policy monitoring as documented
in literature. Our research indicates that the critical success factors for
implementing effective monitoring and reporting include the following:[1]
- Effective working relationships
with departments to ensure provision of information to the central agency;
- Strong intelligence gathering
skills on policy outcomes and processes;
- Capacity to understand, process,
and utilize information based on robust internal analytical processes;
- Internal information-sharing
linking strategic priorities and policy development to monitoring information;
- Clear understanding of the central
agency's objectives and the ability to promote these to departments; and
- Ongoing evaluation of the
information from departments and assessing its consistency with information
prepared and processed by the central agency.
Proposed TBS Framework for Policy Monitoring (the "Framework")
The review also studied policy monitoring practices at TBS by reviewing
seventeen TB policies. Based on our findings, leading practices have been
presented in a framework to provide a tool for TBS to effectively analyze policy
outcomes and monitoring activities in a consistent and systematic manner.
There are four levels of focus in the Framework:
1. Level of Risk – Risk is the overriding factor that
impacts all elements of policy monitoring. Policies that have greater risk
need to be emphasized to ensure they have effective monitoring procedures in
place. The continued effectiveness of these procedures should be evaluated
on an on-going basis.
2. Policy Objectives and Outcomes – Policies are
designed to support the achievement of an organization's objectives and the
management of risks. The following elements are fundamental requirements for
an effective policy to achieve its desired outcomes:
- Policy objectives are clear and concise;
- Outcomes are well defined and measurable, with performance targets and
indicators[2]; and
- Roles and responsibilities are well defined and communicated to departments.
3. Process Enablers – Policy monitoring "process
enablers" are mechanisms that assist TBS and departments with information
sharing and reporting in order to achieve TBS policy objectives. Policy
monitoring is effective where process enablers are systematically applied to
develop and implement policy and monitor outcomes.
The Framework defines seven process enablers. The one that appears to be
the most effectively applied is TBS and departmental working groups and
committees. The process enabler that appears to be the least effectively
applied at TBS is committed funding, due to insufficient resources.
4. Monitoring and Reporting Activities – Monitoring and
reporting requires multiple approaches and activities for information gathering
and follow-up. Information assists TBS and departments in setting priorities
and compiling outcome gap analysis. Structured reporting relationships are
required between TBS and departments to report and analyze outcomes on a timely
and systematic basis.
The Framework defines ten policy monitoring and reporting activities. The
one that appears to be the most effectively applied is TBS' ongoing advice to
departments. The activities that have been under-utilized or are the least
effectively applied are the use of key performance indicators, ongoing reporting,
analysis and follow-up by TBS, and dependence on internal audit and evaluation
reports.
The Framework has been presented to and discussed with the representatives of
the TBS policy centres, and other TBS staff we interviewed in the study. The
Framework was well received and there was a general consensus that adoption of the
Framework will lead to improved policy monitoring.
Summary of Findings Related to Selected TB Policies
Each element of the Framework was compared to the results from the sample of
policies studied. We grouped the sample of policies by level of risk, and
charted the level of monitoring identified in comparison to a desired level.
From this exercise, we made three observations.
- Generally, TB policies identified as high risk
need to enhance their level of monitoring and reporting activities;
- Generally, policies supporting medium risk
areas have adequate levels of monitoring; and
- Generally policies seen as low risk areas are
not receiving the minimum threshold levels of monitoring.
Areas for Improvement
Our study has revealed that certain elements described in the Framework have
been adopted within TBS. We also observed that policy areas generally
understand the level of risk, a fundamental component of the Framework.
However there is an inconsistent application of the monitoring elements throughout
TBS and certain practices are more effective as TBS monitoring tools.
The report recommends that TBS should adopt the Policy Monitoring Framework as
a tool for a systematic approach to defining policy monitoring and reporting for
policies that are developed in the future. A portfolio of the Framework's
process enablers and policy monitoring and reporting activities should be selected
and tailored to each policy area based on the level of risk.
Recommendations are also made for improvement in the use of nine of the
elements of the Framework. Of these, TBS should initially focus on the
following four areas.
1. Key Performance Indicators
We recommend that TBS continue to establish key performance indicators (KPIs)
as a key requirement for policy development and monitoring. KPIs should be
practical and cost-effective to measure.
2. TBS Committed Funding
We recommend a more effective use of TBS resources in conducting monitoring of
policy outcomes. Steps would include:
- Monitoring high-risk policy areas – Given the large number
of TBS policies, an initial approach to adopt the Framework would be to focus
resources on high-risk policy areas. In most cases, structured
monitoring may occur at the policy framework level rather than at the
individual policy level.
- Key elements to monitor – Given the current TBS resource
constraints, key performance indicators, policy outcomes, follow-up of
departmental action items and implementation are the key elements to focus on.
3. Ongoing Reporting and Analysis
We recommend that TBS adopt a structured reporting approach to policy
monitoring including:
- Reporting by policy outcomes (policy implementation, policy results);
- Pre-determined reporting schedule leveraging existing departmental reporting
tools and processes;
- Improved analytical processes and dissemination;
TBS providing more meaningful and value added feedback to departments
comprising government wide trends and benchmarks as well as departmental
comparisons.
4. Actionable Items
We recommend that consideration be given to improving mechanisms for
information sharing within TBS and to convert information into actionable items
for correction including:
- Departmental intelligence gathering from multiple TBS in-house sources
through the TBS business planning team mechanism;
- Structured department and TBS reporting of action items against policy
outcomes;
- Structured escalation of
deficiencies in action items to the TBS executive level; and
- Ongoing reporting between the
Secretary and departmental deputy heads, using such mechanisms as the Management
Accountability Framework.
Management response
Due to the nature of the report, management has chosen to provide an overall
response to the findings and recommendations. See the management response.
|