|
|
|
Treasury Board of Canada Secretariat
Review of Policy Monitoring
October 28, 2004
TBS Internal Audit and Evaluation Division
Ernst & Young, LLP
Annex A - List of Policies Included in Review
Annex B - List of Departments Interviewed
The successful monitoring of Treasury Board (TB) policies is critical in today's environment of rapid change and challenges in addressing policy compliance and effectiveness. It is essential that effective policy monitoring practices be in place at the Treasury Board Secretariat (TBS) to ensure the success of TB policies. This review was requested by the TBS Internal Audit and Evaluation Committee to provide TBS management with an independent study of leading practices in the area of policy monitoring. The review was designed to assess the current status of policy monitoring within TBS and provide practical examples of leading practices. The recommendations made in this report will provide guidance for the Policy Review and Reporting Project as it proceeds to streamline TB policies and reporting requirements.
Leading Practices Research
The review has researched leading practices for policy monitoring as documented in literature. Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following:[1]
Proposed TBS Framework for Policy Monitoring (the "Framework")
The review also studied policy monitoring practices at TBS by reviewing seventeen TB policies. Based on our findings, leading practices have been presented in a framework to provide a tool for TBS to effectively analyze policy outcomes and monitoring activities in a consistent and systematic manner.
There are four levels of focus in the Framework:
1. Level of Risk – Risk is the overriding factor that impacts all elements of policy monitoring. Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place. The continued effectiveness of these procedures should be evaluated on an on-going basis.
2. Policy Objectives and Outcomes – Policies are designed to support the achievement of an organization's objectives and the management of risks. The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:
3. Process Enablers – Policy monitoring "process enablers" are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS policy objectives. Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes.
The Framework defines seven process enablers. The one that appears to be the most effectively applied is TBS and departmental working groups and committees. The process enabler that appears to be the least effectively applied at TBS is committed funding, due to insufficient resources.
4. Monitoring and Reporting Activities – Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up. Information assists TBS and departments in setting priorities and compiling outcome gap analysis. Structured reporting relationships are required between TBS and departments to report and analyze outcomes on a timely and systematic basis.
The Framework defines ten policy monitoring and reporting activities. The one that appears to be the most effectively applied is TBS' ongoing advice to departments. The activities that have been under-utilized or are the least effectively applied are the use of key performance indicators, ongoing reporting, analysis and follow-up by TBS, and dependence on internal audit and evaluation reports.
The Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study. The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.
Summary of Findings Related to Selected TB Policies
Each element of the Framework was compared to the results from the sample of policies studied. We grouped the sample of policies by level of risk, and charted the level of monitoring identified in comparison to a desired level. From this exercise, we made three observations.
Areas for Improvement
Our study has revealed that certain elements described in the Framework have been adopted within TBS. We also observed that policy areas generally understand the level of risk, a fundamental component of the Framework. However there is an inconsistent application of the monitoring elements throughout TBS and certain practices are more effective as TBS monitoring tools.
The report recommends that TBS should adopt the Policy Monitoring Framework as a tool for a systematic approach to defining policy monitoring and reporting for policies that are developed in the future. A portfolio of the Framework's process enablers and policy monitoring and reporting activities should be selected and tailored to each policy area based on the level of risk.
Recommendations are also made for improvement in the use of nine of the elements of the Framework. Of these, TBS should initially focus on the following four areas.
1. Key Performance Indicators
We recommend that TBS continue to establish key performance indicators (KPIs) as a key requirement for policy development and monitoring. KPIs should be practical and cost-effective to measure.
2. TBS Committed Funding
We recommend a more effective use of TBS resources in conducting monitoring of policy outcomes. Steps would include:
3. Ongoing Reporting and Analysis
We recommend that TBS adopt a structured reporting approach to policy monitoring including:
TBS providing more meaningful and value added feedback to departments comprising government wide trends and benchmarks as well as departmental comparisons.
4. Actionable Items
We recommend that consideration be given to improving mechanisms for information sharing within TBS and to convert information into actionable items for correction including:
Management response
Due to the nature of the report, management has chosen to provide an overall response to the findings and recommendations. See the management response.
The successful monitoring of Treasury Board (TB) policies is critical in today's environment of rapid change and challenges in addressing policy compliance and effectiveness. It is essential that effective policy monitoring practices be in place at the Treasury Board Secretariat (TBS) to ensure the success of TB policies. This review was requested by the TBS Internal Audit and Evaluation Committee to provide TBS management with an independent study of leading practices in the area of policy monitoring.
As the administrative arm of the Treasury Board (TB), the Treasury Board Secretariat (TBS) has a dual mandate: to support the Treasury Board as a committee of ministers, and to fulfill the statutory responsibilities of a central government agency.
Effective December 12, 2003, the functions and mandate of the Treasury Board Secretariat have been amended. TBS is now tasked with providing advice and support to Treasury Board Ministers in their role of ensuring value for money as well as providing oversight of the financial management functions in departments and agencies. The Secretariat also plays a key role in supporting the activities of the Expenditure Review Committee of Cabinet.
The Secretariat makes recommendations and provides advice to the Treasury Board on policies, directives, regulations and program expenditure proposals with respect to the management of the government's resources. Its responsibilities for the general management of the government affect initiatives, issues and activities that cut across all policy sectors managed by federal departments and organizational entities (as reported in the Main Estimates). The Secretariat is also responsible for the comptrollership function of government.
Under the broad authority of sections 5 to 13 of the Financial Administration Act, TBS supports the Treasury Board in its role as the general manager and employer of the Public Service.
TBS has recognized the need to update the policy suite to bring it in line with Results for Canadians: A Management Framework for the Government of Canada. In September 2002, it launched a Policy and Reporting Review Project. The Project issued a report in September 2003.
TBS' 2003 Departmental Performance Report indicated the Policy and Reporting Review Project will streamline TB policies and reporting requirements and lead to a smaller and more strategic set of policies and reporting requirements aligned with the Management Accountability Framework.
A component of the Policy and Reporting Review Project addresses TBS and departmental roles and responsibilities for effective policy monitoring. This Policy Monitoring Review complements and expands on the practice of policy monitoring as it relates to TBS.
Policy monitoring can be defined as "the assessment of the functioning of policy, and the results achieved. It addresses, as appropriate, issues of both policy application and compliance."[3]
Many TB policies have a requirement that departments themselves monitor implementation within their organizations with monitoring of the implementation of the policy across all departments conducted by TBS. There is also a TB "Policy on Active Monitoring", introduced in 2001, which requires monitoring across government. However, we understand this policy is not meeting the needs for policy monitoring.
It was stated in the June 2003 Policy Council Briefing[4] that "For results-based management to work, the policy suite must provide clear and appropriate direction on what actions and results TBS is holding Deputies and their officials accountable for. In particular, it must set out respective responsibilities of departments and TBS for the monitoring and reporting of policy results. Information gathered pursuant to reporting and monitoring activity should be used to inform decision-making at departmental, Secretariat and governmental levels".
The Policy Monitoring Review was designed to assess the current status of policy monitoring within TBS and provide practical examples of leading practices. The review is intended to provide assistance to management and is not an audit. The recommendations made in this report will provide guidance for the Policy Review and Reporting Project as it proceeds to streamline TB policies and reporting requirements.
The review focused on analysing the current activities undertaken by the TBS for monitoring TB policies to identify effective practices for:
The planning for this review, the selection of the TB policies considered in the review, and many of the interviews conducted were completed in the fall of 2003, prior to the announcements on December 12, that altered the functions and mandate of TBS.
Approach
The approach taken to conduct the review included the following steps:
Based on the above steps, practical recommendations were developed to guide the policy review process to ensure that revised policies facilitate monitoring activities.
Methodology
The review team selected 17 policies from the inventory of over 300 TB policies. The sample of policies was chosen, with input from the business line leaders, from the three TBS business lines and included a range from low to high-risk policies and a cross-section of older and newer policies. The policies included in the sample are listed in Annex A.
For each of the seventeen policies in the sample, we interviewed one or more TBS policy centre representatives. We discussed the major issues facing the policies, the policy monitoring and reporting activities and areas for improvement in policy monitoring. Based on our discussions, we developed an assessment of the level of risk associated with a particular policy and confirmed this with the policy representative.
In addition to interviewing TBS policy centre staff, interviews were conducted with departments to gain departmental perspectives on policy monitoring and reporting. The departments in the sample included large and small organizations and are listed in Annex B.
We also interviewed representatives of departmental internal audit groups, one of the TBS program sectors, the TBS Centre of Excellence for Audit and the TBS Centre of Excellence for Evaluation.
The following three documents provided key information for the study:
Introduction
The review was designed to provide senior management with an assessment of leading practices in the area of policy monitoring that will act as a support tool to TBS' Policy and Reporting Review Project.
Our research into external leading practices included "thought leadership" in the areas of corporate governance, performance monitoring and reporting, and management control frameworks. We consulted the work of the Canadian Institute of Chartered Accountants (Study of Control Criteria), Conference Board of Canada, the Committee of Sponsoring Organizations of the Treadway Commission and author Dr. John Carver.
Practices from New Zealand's Pacific People's Monitoring Framework provided valuable insight into monitoring frameworks.
Further, the review team has identified effective practices within TBS as a result of studying the sample of policies from each TBS business line.
The identified leading practices have been consolidated and are presented in the following sections of the report. They are introduced in our proposed model - TBS Framework for Policy Monitoring (Section 3.1). Each of the key monitoring elements of the framework is then defined. The definitions are followed by our findings from the sample of TBS policies studied (Section 3.2). In Section 4.0, we present our analysis of TBS current monitoring practices against the leading practices of the framework. The areas for improvement identified in the comparison are set out for consideration.
Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following:[5]
Based on this information, we developed the framework in Figure 1 to provide a tool for TBS to develop an effective approach to policy monitoring that can be implemented in a consistent and systematic manner. It highlights the importance of obtaining information about policy performance from departments and to effectively analyze policy outcomes.
The four levels of focus of the TBS Framework for Policy Monitoring (the Framework) as illustrated in Figure 1 are: Level of Risk; Policy Objectives; Process Enablers; and Monitoring and Reporting Practices.
The following explanations describe the four levels of the framework.
1. Level of Risk
An organization needs to identify its significant internal and external risks on an ongoing basis so that it can react to or initiate changes in an appropriate and timely manner. Monitoring needs to be structured according to the risk of not meeting the desired policy outcomes.[6]
Risk is the overriding factor that impacts all elements of policy monitoring. Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place. The continued effectiveness of these procedures should be evaluated on an on-going basis.
2. Policy Objectives and Outcomes
Policies are designed to support the achievement of an organization's objectives and the management of risks. The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:
3. Policy Monitoring Process Enablers
Policy monitoring "process enablers" are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS and departmental policy objectives.
Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes[8]. Process enablers seek to:
4. Monitoring and Reporting Practices
Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up. Three policy performance areas requiring monitoring include:
TBS Framework for Policy Monitoring
This study identified seven leading "Process Enablers" and ten leading "Monitoring and Reporting" practices that have been incorporated into our proposed TBS Framework for Policy Monitoring illustrated in Figure 2. The following sections further define the "Process Enablers" and "Monitoring and Reporting" elements of the Framework.
The Framework's seven leading process enablers and their desired characteristics are:
1. TBS and Departmental Executive Leadership/Sponsorship – TBS and departmental executives' expectations and accountabilities for achieving policy objectives and monitoring performance are well understood and aligned. TBS and departmental leadership teams are committed to and focused on policy improvement. TBS and departmental leadership is well informed, and follows up on the results of actions taken.
2. Effective Implementation – Policy development and implementation is effective where policy outcomes are well defined, policy schedule and milestone targets are established and these are agreed upon and followed-up by executives. Support tools, such as training material, guidelines, checklists, and lessons learned accelerate the rollout of the policy.
3. Committed Funding – Adequate resources for monitoring are committed to provide skilled staff and tools for information gathering, key performance indicator monitoring and action item follow-up. Monitoring activities are designed as an integral part of the organization (A-Base budgets) taking into consideration the risks to achieving objectives and the interrelationship with other monitoring elements.
4. Enabling Legislation – The existence of prescriptive legislation (e.g., Employment Equity Act) with well-defined requirements for annual reports to Parliament, audits, and roles of departmental officials is an effective enabler. The legislation provides additional authority that focuses on compliance to government priorities.
5. Oversight by External Bodies – The existence of external bodies (e.g., Commissioner of Official Languages) empowered by legislation to conduct independent audits, and follow-up of complaints is an effective enabler. External bodies provide the independent focus that can bring valuable feedback to TBS and departments.
6. Working Groups/Committees – Formalized working groups, chaired and co-chaired by primary stakeholders, are effective policy enablers. These formal groups have well-defined deliverables for policy design, implementation and follow-up. For optimal effectiveness, executive leadership teams from TBS and departments sponsor policy initiatives from design through to implementation and policy support.
7. Integration of Policy Monitoring with Key TBS Financial Processes – Transaction monitoring by TBS (e.g., Privacy Impact Assessments) is a detailed approach to monitoring specific policy requirements. It is effective in identifying cases of non-compliance or issues identified during a submission review. In order to be effective, sufficient TBS resources must be in place to handle the volume of submissions.
These seven process enablers represent a basket of effective practices to strengthen the achievement of the policy objectives. In developing each policy, an optimal combination of these practices should be selected. The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.
The Framework's ten leading policy monitoring and reporting practices and their desired characteristics are:
1. Key Performance Indicators Reporting (KPIs) – Organizations use a mixture of quantitative and qualitative performance targets to measure progress towards achieving the defined policy objectives. These targets are measurable and aligned with the objectives.[9] Externally derived benchmarks are also used to help ensure targets are aligned with external best practices. Where performance criteria have pre-established targets and are analyzed against policy outcomes, then the performance indicators provide early warning signals as to whether the targets have not been met. It is critical to the success of KPIs that departmental KPI reporting is practical and supported by existing reporting systems.
2. Executive Level Reporting – Executive level reporting and communicating of progress against policy outcomes enables TBS and departments to complete their priority setting and decision-making activities. The enhancements that are being made to management reporting between TBS and departments through various initiatives, such as the Management Accountability Framework, place greater focus on outcomes and performance indicators. The terms of executiveperformance management agreements also endorse achievement of policy outcomes.
3. Strong Relationships with Departments – Developing and maintaining good working relationships between TBS and departments to ensure provision of information is necessary for ongoing monitoring.[10] TBS plays an effective advisory role with the departments where it contributes the following:
4. Ongoing Reporting and Analysis – Departments and TBS each have timely and reliable information to evaluate compliance to policy objectives and to measure against expectations, with performance primarily monitored by the departments. Leading departments have the capacity to understand, process, and utilize information based on robust internal analytical processes. Central agencies also monitor the information provided by departments and assess its consistency with information prepared and processed by the central agency.
5. Audit and Evaluation Monitoring and Reporting – Monitoring is best conducted using multiple approaches including ongoing reporting and analysis. Independent audits and evaluations conducted by internal audit and evaluation and by external bodies provide a fresh focus on the effectiveness of the policy as well as the effectiveness of monitoring.[11] However, audit and evaluation engagements are structured to the specific needs of the departments, as determined through risk assessments and consultation with senior management. They are not driven by individual policy requirements. Therefore, these activities are, at best, viewed as supplementary assessments of the adequacy of the monitoring and reporting processes in place and not as a primary tool for providing feedback on performance in meeting the policy objectives.
6. Advice to Departments – A good understanding of policy objectives and priorities and the ability to promote these to departments is a critical success factor for effective policy development, deployment and monitoring.[12] The central agency positions itself as the "trusted advisor" or subject matter expert working collaboratively and in partnership with departments. The central agency takes a proactive role in identifying and addressing issues in addition to reacting to problems as they arise.
7. Intelligence Gathering – Monitoring the external environment provides valuable information on the state of the internal environment. Changes in external conditions (such as changes in regulations, societal trends, and program and political changes) can have a significant effect on departments' and TBS' ability to meet their objectives.
Intelligence gathering is facilitated by various tools and techniques such as:
8. Knowledge Sharing Tools and Processes – Sharing best practices and lessons learned from stakeholder departments and TBS promotes effective policy implementation. Information obtained and shared provides a range of approaches to addressing policy concerns and the achievement of policy outcomes. Building a culture of knowledge capture and sharing further enhances the TBS' leadership and "trusted advisor" role with departments.
9. Communication – Open communication both creates and depends on trust. A high level of trust encourages people to ensure that everything of importance is known by others. Communication processes support common values and achievement of objectives. For monitoring to be effective, there are communication processes capable of supporting two-way, open communication of timely, relevant and reliable information. These include:
10. Actionable Items - A well established reporting and follow-up protocol between departments and TBS is in place to implement corrective action for policy outcome gaps.
These ten policy monitoring and reporting activities represent a basket of effective practices to strengthen the feedback mechanisms for assessing the achievement of policy outcomes. In developing each policy, an optimal combination of these practices should be selected. The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.
The Policy Monitoring Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study. The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.
Recommendation
We recommend that TBS adopt the Policy Monitoring Framework as a tool for taking a systematic approach to defining the policy monitoring approach for policies that are developed in the future.
We also recommend that TBS adopt a portfolio of the Framework's process enablers and policy monitoring and reporting practices that is tailored to each policy area and level of risk to improve policy monitoring.
Seventeen TB policies were selected for review with key TBS and departmental representatives responsible for the selected policy areas. The purpose was not to critique or comment on specific policy issues. Rather, the objective was to identify and discuss general trends and issues in policy monitoring including:
Based on our analysis, we have grouped the seventeen policies by their level of risk and our assessment of the level of monitoring activity and presented the results in Figure 3. The level of risk is presented on the horizontal axis (Low/Medium/High) and the level of monitoring is presented on the vertical axis (minimum/moderate/enhanced). Also presented in Figure 3, is the "Desired State", which is the recommended level of monitoring associated with each level of risk. Effective policy monitoring requires a balance between TBS monitoring activities and the level of risk associated with the policy area.
Following are three observations with respect to the level of monitoring being conducted from the policies studied in the review.
1. Strategic Impact – Enhanced Level of Monitoring – Generally TBS policies identified as high risk areas need to enhance their level of monitoring and reporting activities to address their strategic significance. These policies in a number of cases have less monitoring than policy areas with medium risk.
2. Operational Excellence – Moderate Level of Monitoring – Policies supporting medium risk areas generally have adequate levels of monitoring. From our study, it would appear that some policy areas with medium risk are over monitored while other medium risk policies are not sufficiently monitored.
3. Operational Stability – Minimum Level of Monitoring – Policies that are seen as low risk areas require a minimum level of TBS monitoring to ensure stability is maintained. Generally, low risk policies were not receiving minimum threshold levels of monitoring.
Throughout the study, we observed that certain elements described in the Framework have been adopted within TBS. We found that certain practices are more effective than others as TBS monitoring tools. We also observed that policy areas generally understood the level of risk, a fundamental component of the Framework. However there is an inconsistent application of the monitoring elements throughout TBS.
After completing our interviews with the representatives of the TBS policy centres and departments, we summarized our understanding of the effective monitoring practices, areas for consideration, level of risk, and constraints to monitoring for each of the policies in the sample. We met again with the TBS policy centre representatives to confirm our understanding, to present the Framework and to discuss it in relation to the policy.
Next we completed an analysis by assessing the effectiveness of each of the Framework elements for each of the 13 policies in the sample that were identified as medium to high risk. We combined the assessments of effectiveness for these individual policies and generated a relative effectiveness rating for each of the Framework elements. The overall rating of effectiveness for any element is influenced by the number of policies in which the element is present and by the assessed effectiveness of the element in each of the policies.
We have illustrated the results of this analysis in the bar chart in Figure 4. Each bar represents one of the leading practices defined previously in the report by Framework element and is measured against our assessment of a low, medium or high effectiveness rating.
There are a number of elements of the Framework that appear to be effectively used by TBS. For example, as illustrated by the bar chart in Figure 4, "Working Groups and Committees" and "Advice to Departments" have been assessed as effective practices. There are some other elements in the Framework, such as "Intelligence Gathering" and "Integration with Key TBS Financial Processes", which we consider less important for TBS to address than others. We will not discuss the elements in these two groups further in the report; however, summary observations, including TBS effective practices, are presented in a table in Section 4.2.10.
We observed that some of the other elements included in the Framework, such as "Committed Funding" are not consistently applied and provide opportunities for improvement. We consider other elements of the Framework, such as "Key Performance Indicators" to be very important and require special attention. In the next section, we set out our observations and discuss our recommendations for improvement for the eight Framework elements that fall into these two groups.
The Framework has defined nineteen elements for effective policy monitoring. We have identified nine of these elements with areas for improvement, including eight that were selected from the chart in Figure 4, plus level of risk. The nine elements are:
These are described in the following sections. Recommendations are included in the areas for consideration.
Observation
There is an absence of systematic and formalized ranking and monitoring of the level of risk associated with the policies.
Historically the number of policies has continued to grow in number without any significant elimination or consolidation of less relevant or outdated policies. As a result, TBS has directed each of its business lines to review the number and content of policies and improve upon their effectiveness.
Areas for Consideration
Risk management is a one of the four levels of focus in the Framework for Policy Monitoring. This is consistent with the TBS Management Accountability Framework and modern management practices.
We recommend that TBS establish a standard set of criteria for evaluating risks associated with each policy area. Criteria would include the impact on departments and departmental risk areas. Additional steps for consideration include:
Observation
Generally the policy objectives and monitoring requirements supporting the TBS policies we studied are broad statements without clarity of outcomes or success criteria. New policies being developed are focusing attention on improved clarity around outcomes, key performance indicators and roles and responsibilities. For example, the policy on Continuous Learning adopted a different policy format from traditional TBS policies by presenting departmental commitments, expectations, timeframes and providing a detailed description of roles and responsibilities.
Areas for Consideration
As part of TBS' policy renewal effort, we suggest creating improved policy objectives and outcomes that reflect the following:
Observation
The establishment and monitoring of key performance indicators that support policy outcomes was the most significant gap from the sample of policies studied. Improvements in data and information capture infrastructure and tracking of performance indicators has been an area of increased focus for TBS policies under development (e.g., Employment Equity uses performance indicators to measure performance).
Areas for Consideration
We recommend that TBS continue with the establishment of key performance indicators (KPIs) as a key requirement for policy development and monitoring. Steps for consideration include:
Observation
A common issue raised by TBS interviewees was that limited resources adversely impacts TBS' ability to conduct policy monitoring and reporting. Furthermore, TBS has faced staff retention issues and its ability to attract top talent in policy areas has been a challenge.
In the case of most policies studied, A-Base funding provides resources to the policy centres within TBS. Although we did not study the level of resources assigned by policy, it is apparent that the amount of resources and consequently the approach for monitoring varies considerably within TBS. In some cases the variation in monitoring can be explained by the level of risk associated with the policy. However, in others, the relationship between risk and resources was not apparent.
In the case of two policy areas (Modern Controllership - Integrated Risk Management and Federal Contaminated Sites Accelerated Action Plan- Contaminated Sites Management Policy) implementation is supported by project funding.
Areas for Consideration
We recommend that TBS assign resources to monitoring activities by adopting the following strategies:
Observation
TBS continues to evaluate its reporting requirements with departments through various initiatives including the Management Accountability Framework.
Reporting tends to be statistical with limited follow-up by TBS. Reporting practices varied considerably within the sample of policies studied. These include the following:
Qualitative analysis was minimal and was generally attributed to lack of TBS resources.
Areas for Consideration
We recommend adoption of additional ways to enhance monitoring and reporting including the following:
Observation
Our study identified a number of effective practices for policy implementation which include the following:
Areas for Consideration
In addition to the effective TBS implementation tools referred to above, we recommend TBS consider more consistent adoption of the following practices to assist in monitoring policy implementation:
Observation
From the policies studied, we observed several practices demonstrating executive leadership from TBS and key stakeholder departments that supported policy direction, development, and monitoring. The Clerk of the Privy Council also provides leadership in some areas in terms of establishing government-wide management priorities. The leading practices we observed include:
Areas for Consideration
For high-risk policy areas, we recommend that TBS develop an escalation protocol and procedures within TBS and with departments to report performance exceptions to TBS executives. This will require establishment of ongoing reporting and analysis of policy outcome gaps.
Observation
The study revealed that TBS is challenged by resource constraints impacting active follow-up and monitoring of policy outcomes and performance. Follow-up and reporting does not appear to be consistent and systematic.
Areas for Consideration
We recommend that consideration be given to improve the mechanisms for information sharing within TBS and to convert information into actionable items for correction including:
Observation
A majority of the policies studied make reference to reliance on the conduct of departmental internal audits as the key, and in many cases, sole identified monitoring mechanism. However, these specific responsibilities are not consistent with the role of internal audit. The Policy on Internal Audit affirms the function of internal audit as a provider of assurance services to departmental management. Audit engagements are structured to the specific needs of the organization, as determined through risk assessments and consultation with senior management. While internal audits provide assurance that the appropriate monitoring and control frameworks are in place, they should not be considered the primary mechanism for monitoring and reporting on individual policies. Internal audit plans are department focused and therefore do not provide consistent coverage across the government.
Evaluations are distinct from audits. While audit provides assurance services, evaluation is a management tool that has two main purposes:
There are nine policies in the study sample with a specific requirement for departmental internal audits or evaluations of the policy. However, Contracting, Transfer Payments and Government Security Policy are the only policies where we found that internal audits or evaluations are being completed.
We observed that information from departmental audits is not easily shared and disseminated in a meaningful way within TBS. Internal audit reports are circulated across TBS, and are reviewed independently in many areas, but there does not appear to be a centralized approach to assessing the impact of the information contained the audit reports.
Area for Consideration
We recommend that the requirement for internal audits of compliance of specific policies be discontinued as a generic policy requirement since it does not reflect the way in which internal auditing is planned and conducted. The risk-based audit approach undertaken by TBS and departments does not align with the specific policy monitoring required for TBS and departments. Internal audit should be viewed as a provider of assurance over and above the other monitoring and reporting mechanisms adopted for policies. If appropriate, the policy could refer to the role of internal audit by making reference to the Policy on Internal Audit.
We also recommend that consideration be given to identify emerging risks and share audit results including:
In developing the Framework, a number of effective TBS monitoring activities from select policies were identified and are included as key elements. These are briefly listed in the table below.
|
Element of Framework |
TBS Policies |
TBS Effective Practice |
|
Existing Legislation External Bodies |
Employment Equity Official Language Access to Information |
|
|
Effective Working Groups/Committees |
Federal Contaminated Sites Management Policy Integrated Risk Management (IRM) Official Languages |
|
|
Integration with Key TBS Financial Processes |
Contaminated Sites Management Policy Privacy Impact Assessment |
|
|
Executive Level Reporting |
Employment Equity Official Language Access to Information |
|
|
Communications |
Integrated Risk Management |
|
|
Intelligence gathering |
Various |
|
|
Knowledge Sharing Tools |
Privacy Impact Assessment Integrated Risk Management Contracting |
|
|
Advice to departments |
Government Security Policy |
|
|
Strong Relationships with departments |
Contaminated Sites Management Policy Integrated Risk Management |
|
The senior management of Treasury Board Secretariat (TBS) acknowledges the necessity of having effective policy monitoring practices in place at TBS and generally agrees with the recommendations stemming from this review, which aim to improve these practices at TBS.
Leading practices in policy monitoring activities at TBS were identified and presented in a Policy Monitoring Framework ("the Framework") as a tool for TBS to achieve a systematic approach to defining policy monitoring and reporting for policies that are developed in the future. It was suggested that adoption of the Framework would lead to improved policy monitoring.
We agree to take steps towards adopting the critical elements of the proposed Policy Monitoring Framework ("the Framework") outlined in the report including, as a priority, four of the nine recommended areas of focus plus one additional area:
On the issue of risk, we agree with the general proposals for evaluating the risks associated with each policy area, including the needs for periodic risk assessments to determine appropriate policy monitoring levels, and for ensuring the alignment of necessary capacity (resources and skills) to accomplish policy monitoring. Specific attention will be paid to the way in which high-risk policies are monitored, and how structured monitoring can be built in to the policy framework.
In the context of the present TBS Executive review of strategic business re-engineering initiatives (of which the TB Management Policy Review is one), we will review the internal distribution of resources to support the necessary monitoring of high-risk policies, and recommend necessary incremental investments or resource reallocations to that end. These decisions will be made during this fiscal year.
As recommended by the Review of Policy Monitoring, and consistent with the principles of the current TB Management Policy Review, we will continue to consider and implement improvements in the areas of:
In general, we will ensure that policy monitoring is conducted in a way consistent with the new approach to strategic oversight and risk assessment being adopted by TBS – including through related assessments of departmental and government-wide management and expenditure performance, based on the Management Accountability Framework. We will address the identified requirements for policy monitoring during the course of the TB Management Policy Review, whose main timelines for implementation are as follows:
General Remarks
The recommendations made in the review are meant to provide guidance to the TB Management Policy Review, as it proceeds to streamline TB policies and reporting requirements. Through the work of the TB Management Policy Review, a coherent and consistent approach to these recommendations will be taken across the policy suite.
The TB Management Policy Review will examine ways in which policy monitoring and reporting requirements can be better linked to the Management Accountability Framework and the Expenditure Management Information System. This will help to ensure better knowledge capacity, better knowledge transfer and less redundancy in the reporting of results.
The review suggests that "the requirement for internal audits of compliance of specific policies be discontinued as a general policy requirement since it does not reflect the way in which internal auditing is planned and conducted." Recognizing that risk-based audit planning practices are accepted across government, we appreciate the argument for discontinuing general policy requirements for internal audits of compliance within policies. We will look at possibilities for discontinuing the audit requirements articulated in specific policies, on a policy-by-policy basis. However, existing requirements will not be discontinued outright until other options for providing similar oversight capacity are identified and in place. Until then, the requirement for internal audits of compliance with specific policies will remain.
We would suggest that specific audit requirements within policy instruments should be contemplated for circumstances of generalized risk related to the policy, and in consultation with TBS internal audit policy experts. In the context of broader reviews (e.g. to strengthen compliance and sanctions related to the Financial Administration Act), we are presently exploring the range of mechanisms for ensuring effective compliance with policy and legislation. The resulting recommendations will influence how we address audit requirements in the TB policy suite.
Our study has revealed that TBS conducts effective policy monitoring in certain policy areas. We also observed that policy areas generally understood the level of risk, a fundamental component of the Framework. However there is an inconsistent application of the monitoring elements throughout TBS and certain practices are more effective as TBS monitoring tools.
The recommendations within the report are:
Key Performance Indicators
Level of Risk
Committed Funding
Ongoing Reporting and Analysis
Policy Objectives and Outcomes
Effective Implementation
Actionable Items
Audit and Evaluation
TBS and Departmental Executive Leadership/Sponsorship
A more detailed description of the recommendation for each practice is included in the body of the report.
The areas that TBS should focus on initially to strengthen the achievement and monitoring of policy outcomes are:
Employment Equity Policy
| Official Languages | - Language of Work in "Bilingual" Regions |
| - Communications with the Public |
Access to Information
Federal Contaminated Sites Management Policy
Privacy Impact Assessments Policy
Government Security Policy
Integrated Risk Management Framework including Risk Management Policy
Policy on the Prevention and Resolution of Harassment in the Workplace
Transfer Payment Policy
Contracting Policy
A policy for Continuous Learning in the Public Service of Canada
Policy on Delegation of Authorities
Policy of Delegation of Organizational Authority
Policy on Electronic Authorization and Authentication
Smoking in the Workplace
Telework Policy
Agriculture & Agri-food Canada
Canada Revenue Agency
Canadian Heritage
Canadian Institutes of Health Research
Citizenship & Immigration Canada
Correctional Service Canada
Health Canada
Human Resources Development Canada
Indian and Northern Affairs
Industry Canada
National Defence
Public Works and Government Services Canada
Transport Canada
[1] The Monitoring Framework – New Zealand- Ministry of Pacific Island Affairs
[2] The Canadian Institute of Chartered Accountants – Guidance on Control Issued by the Criteria of Control Board
[3] June 24, 2003 Policy Council Briefing Note- Monitoring and Reporting Requirements in TB Policies
[4] June 24, 2003 Policy Council Briefing Note- Monitoring and Reporting Requirements in TB Policies
[5] The Monitoring Framework – New Zealand- Ministry of Pacific Island Affairs
[6] The Canadian Institute of Chartered Accountants – Guidance on Control Issued by the Criteria of Control Board
[7] The Canadian Institute of Chartered Accountants – Guidance on Control Issued by the Criteria of Control Board
[8] The Monitoring Framework – New Zealand- Ministry of Pacific Island Affairs
[9] The Canadian Institute of Chartered Accountants – Guidance on Control Issued by the Criteria of Control Board
[10] The Monitoring Framework – New Zealand- Ministry of Pacific Island Affairs
[11] John Carver's Boards that Make a Difference
[12] The Monitoring Framework – New Zealand- Ministry of Pacific Island Affairs
