2004-612 Formative Evaluation of the Professional Development Regime for PWGSC Procurement Resources, Final Report

October 27, 2005

Table of Contents

• Executive Summary
• 1. Introduction
  • 1.1 Background
  • 1.2 Methodology
    
    
• 2. Evaluation Framework - Profile
  • 2.1 Component Profile
    • 2.1.1 Origin
    • 2.1.2 Governance Structure
    • 2.1.3 Resources
    • 2.1.4 Delivery Approach
    • 2.1.5 Target Populations
    • 2.1.6 Planned Objectives and Results
      
      
  • 2.2 Logic Model
    • 2.2.1 The Life-Cycle Approach to Professional Development
    • 2.2.2 Generic Logic Model
    • 2.2.3 Activities, Outputs and Immediate Outcomes
    • 2.2.4 Intermediate and Ultimate Outcomes
      
      
• 3. Evaluation Framework - Issues Matrix
  • 3.1 Evaluation Issues and Indicators
    • 3.1.1 Align Professional Development Program with Government, Department and Branch Needs and Priorities
    • 3.1.2 Design Professional Development Delivery Approach; Gap Analysis
    • 3.1.3 Professional Development Program Delivery
    • 3.1.4 Professional Development Program Monitoring, Review and Evaluation
    • 3.1.5 Program Results/Impacts
    • 3.1.6 Cost Effectiveness/Alternatives
      
      
  • 3.2 Evaluation Methodologies/Data Sources
    
    
• 4. Formative Findings and Conclusions
  • 4.1 Align Professional Development Program with Government, Department and Branch Needs and Priorities
  • 4.2 Design Professional Development Delivery Approach; Gap Analysis
  • 4.3 Professional Development Program Delivery
  • 4.4 Professional Development Program Monitoring and Evaluation
  • 4.5 Program Results/Impacts
  • 4.6 Cost Effectiveness/Alternatives
  • 4.7 Overall Conclusions
    
    
• 5. Recommendation
  
  
• Appendix A - Course listings

Executive Summary

Authority for the Project

This formative evaluation was undertaken in accordance with the Audit, Assurance and Ethics Committee approved Evaluation Plan for the 2004-05 fiscal year.

Objective

To assess PWGSC's progress towards meeting the goals of professional development for procurement resources.

Scope

For this engagement the professional development regime was considered to be the various mechanisms and entities that contribute to the procurement training and development available to procurement personnel in PWGSC. The scope of this formative evaluation examined how these mechanisms and entities were functioning to develop the procurement expertise of purchasing personnel in the Department. The scope did not include the examination of classification or recruitment aspects of procurement resources.

Background

Since 1993 the procurement arm of PWGSC has had the responsibility for coordinating and delivering training and professional development to its procurement personnel. In 2003/04, the Acquisition Branch (AB) of PWGSC centralized its professional development, training services and change management activities into a new organization, the Professional Development and Change Management Directorate (PDCMD) that was to provide a uniform and consistent approach to meeting future capacity requirements. The PDCMD also has a leadership role to contribute to the professional development of the procurement community within the public service and to contribute to the development of strategies that ensure that the government has enough sufficiently skilled procurement personnel to manage the business of procurement.

In 2004, the Task Force Report on the Government-Wide Review of Procurement identified procurement as a key central government activity. It further specified that the government procurement community must be supported by the acquisition of the appropriate procurement skills, training and expertise as an essential element of an improved procurement process within government. Since human resource capabilities are an important ingredient for delivery of procurement, a formative evaluation was planned.

Methodology

The methodology employed for this project adhered to the Audit and Evaluation Branch standard approach for formative evaluation work, which included the development of an evaluation framework. The work was executed through document and background reviews, stakeholder interviews, review of administrative data and comparison, where possible, with similar programs in other countries.

Evaluation Approach

In the course of developing the evaluation framework, it became clear that professional development for procurement resources in PWGSC was operating within a climate of significant change both organizationally within the Department, during the last two years, and governmentally with respect to the procurement function and that it lacked some of the key elements of a comprehensive, life-cycle based professional development program.

Consequently, the evaluation team proceeded on the basis that:

1. it was premature at this time to attempt to fully evaluate the program against all of the issues identified in the evaluation framework, given that certain issues could be addressed based on the research conducted in the course of developing the evaluation framework and on the analysis of available information, and other issues could not; and
2. this formative evaluation could best provide value to management by developing an evaluation framework using a logic model based on a generic life-cycle model of professional development programs, assessing available program information and undertaking some comparative research into other jurisdictions.

Key Findings and Conclusions

In addition to a logic model and an evaluation issues matrix that can be used by PDCMD in Acquisition Branch, a set of formative findings were derived for the professional development of procurement resources in PWGSC, as administered by PDCMD and referred to as the Professional Development Program (PDP).

Program Planning:

Issue: Has the program established a clear policy, accountability and planning framework?

• There is no formal policy and planning framework in place specifically for procurement professional development within PWGSC.

Such a framework would include a clear statement of strategic, corporate objectives and goals for professional development as well as guidelines, policies and standards for the design and conduct of professional development activities. In the absence of such a framework, the PDP reacts on an ad hoc basis to organizational and individual demands.

Program Design:

Issue: Is there an effective process for identifying training requirements for individual procurement practitioners?

• There is a process for identifying and referring candidates for professional development but needs are not determined on the basis of an assessment against a skills framework based on designated procurement competencies.

At the moment, the PDP lacks some key elements of an effective process for ensuring that AB procurement resources can identify their skill requirements and the specific professional development activities that can meet these requirements.

Issue: Is there an effective process for meeting demand for training?

• Sufficient information was not available to enable drawing a firm conclusion on this issue.

In the PDP, there is a process for assessing demand, based upon compilation of individual learning plans. However, in the absence of program data on the percentage of training requirements met, the evaluation team was unable to determine how effectively the PDP meets this demand.

Issue: Does the program have the capacity to respond effectively to corporate directives for professional development?

• Conclusions cannot be drawn on this issue at this time, based on the methodologies employed in this formative evaluation.

While the program appears to respond well on an ad hoc basis to senior management requests for additional or changed program delivery requirements, this does not form part of a coherent annual planning or demand identification process nor is the PDP a key participant in the overall corporate level procurement program planning function.

Because PWGSC is part of a government transformation of the procurement process over the next three years, a repositioning of the PDP is imminent to meet new needs in PWGSC and perhaps others in the larger federal government procurement community. Since these needs are not yet precise, the evaluation team did not pursue research or testing on this issue at this time. Consequently, the capacity of the PDP to effectively respond is an unknown at this time.

Program Delivery:

Issue: Are appropriate training delivery systems in place?

• While the PDP appears to be effective in delivering training, it may be considered to be less effective in the sense of working with potential partners to deliver a comprehensive professional development program, while optimizing the use of internal resources.

Based on interviews with PDP officials, other PWGSC officials and officials of Treasury Board Secretariat (TBS) and the Canada School of Public Service (CSPS), and on our review of the PDP course delivery database, while the PDP appears effective, in the sense of providing training relevant to the needs of the procurement community, it does not appear to be working in an integrated manner with other stakeholders at this time to provide an integrated and coordinated set of professional development activities that are clearly linked to specific competencies of the AB procurement community.

With respect to optimizing the use of internal resources, it is not possible to determine whether existing delivery systems are efficient. Currently, the PDP appears to be meeting the existing level of demand for professional development with internal staff playing a direct role in training delivery. However, it is not clear whether program resources would be adequate for the PDP to implement a comprehensive life-cycle based approach to professional development or for it to exercise an expanded role and mandate in professional development across government, without changes to the way it makes use of internal staff.

Program Monitoring and Evaluation:

Issue: To what extent is a systems approach used to monitor and evaluate program (as distinct from course) delivery?

• The PDP does not currently use a systems based approach in the conduct of program or course evaluation.

Participants in some courses complete individual course evaluations of the training that they have received. These documents are completed in hard copy and are supplied to PDP, however, they are not systematically used to review the appropriateness of the course, the delivery approach, nor are they used by PDP to review program choices about how courses or groups of courses should be delivered. There is no comprehensive approach to assessing the overall program of professional development or training in terms of specific topics, areas of specialization or methods of professional development.

Program Results/Impacts:

Issue: To what extent has the PDP contributed to participants’ ability to provide improved procurement services on behalf of PWGSC?

• Based on the research possible within the scope of this formative evaluation, the evaluation team cannot offer any conclusions on the degree to which the PDP contributes to participants' abilities to provide improved procurement services on behalf of PWGSC.

The PDP does collect data that would allow for the evaluation of impacts of its professional development activities on the effectiveness of individuals in carrying out their procurement responsibilities. However, in order to evaluate this, longitudinal (before and after) surveys of employees and their supervisors would be required, and this was not feasible within the parameters and timeframe for this formative evaluation.

Cost-Effectiveness/Alternatives:

Issues: To what extent are the resources appropriate given current and anticipated roles and responsibilities? Are there alternative approaches to management and to course delivery that are more cost effective or efficient?

• Based on interviews with PDP and other officials, any mandate expansion or modification would necessitate changes to the PDP resource mix in relation to roles and responsibilities expected to be exercised by PDP staff.
• Given that the future mandate may change it is not possible at this time to conclude what alternatives might be more cost effective or efficient.

Currently, the PDP has responsibility for all procurement training program planning, design, delivery, and monitoring in AB. Based on the interviews conducted, the PDP appears to be meeting current demands for training. In the absence of workload parameters/metrics, resource capacity could not be assessed. However, PDP officials indicated that their internal resources are currently stretched to meet demand. Additional demands, such as would result from the PDP being given responsibility for professional development for the broader procurement community in the federal government or from adoption of a life-cycle based approach, could mean that the existing resource mix would be inadequate in terms of number of trainers and/or expertise, to continue its current approach of involvement in all aspects of design and delivery. Since the PDP needs to examine more strategic and integrated approaches to managing professional development, options could include strategic partnerships with educational institutions, other government departments, outside service delivery organizations, and national procurement associations for the design and delivery of individual courses and activities.

Overall Conclusions

It is the conclusion of this formative evaluation that professional development for procurement resources lacks some of the key elements of a comprehensive, life-cycle based professional development program. However, the PDP has been highly reactive and agile at providing training to meet ad hoc demand and respond to senior management direction. During the last two years, professional development as been operating within a climate of significant change both organizationally within PWGSC and governmentally with respect to the Task Force and the proposed agenda for the procurement function.

Given the likelihood of an expansion or modification of its professional development mandate, the PDP will need to take a more strategic approach to its responsibilities. Consequently, the PDP will need to strategically review how it carries out the professional development program. Consideration should include how it can work with external partners, such as the CSPS, the private sector, universities and community colleges, and others to achieve its program objectives in a way that optimizes the use of core internal resources and include an examination of the existing professional development activities to assess their continued relevance and effectiveness. If its mandate is expanded or modified, the PDP in AB will have to determine the appropriate mix of resources (numbers and expertise) to discharge the roles and responsibilities expected of it.

The timing is appropriate for a program re-design which would ideally clarify the mandate, roles and responsibilities of the professional development program in Acquisitions Branch in the context of any recommendations of the Task Force and the other initiatives and define the corporate objectives and priorities for professional development in procurement, having examined options for linking competency-based certification programs (e.g. the TBS program) to PWGSC's professional development program. Part of this re-design exercise should implement a life-cycle approach to professional development, elaborated through a delivery strategy that clearly specifies the roles of PWGSC and partners for various aspects of professional development and through determination of resource requirements for implementing the renewed program.

The program re-design should also include the creation of an ongoing performance measurement strategy with associated data requirements and means to capture information. Once the program and its environment have stabilized and matured, a comprehensive evaluation of the PDP could be carried out.

Recommendation

It is recommended that the Assistant Deputy Minister, Acquisitions Branch:

1. Undertake a program re-design to align the professional development of procurement personnel with evolving governmental, departmental and Acquisitions Branch strategic directions and to establish a renewed mandate, objectives and priorities for professional development, a life-cycle approach to professional development, and a program delivery strategy with performance measurement capacity and with provision for future evaluation when the Professional Development Program is stable, against the evaluation framework developed.

1. Introduction

This report contains an evaluation framework and a formative evaluation of the professional development regime for procurement resources, which operates within the Acquisitions Branch of Public Works and Government Services (PWGSC).

The primary professional development and training organization in PWGSC is the Professional Development and Change Management Directorate, which has developed through a series of reorganizations over the past two years along with the department.

Authority for the Project

This formative evaluation was undertaken in accordance with the Audit, Assurance and Ethics Committee approved Evaluation Plan for the 2004-05 fiscal year.

Objective

To assess PWGSC's progress towards meeting the goals of professional development for procurement resources.

Scope

For this engagement the professional development regime was considered to be the various mechanisms and entities that contribute to the procurement training and development available to procurement personnel in PWGSC. The scope of this formative evaluation examined how these mechanisms and entities were functioning to develop the procurement expertise of purchasing personnel in the Department. The scope did not include the examination of classification or recruitment aspects of procurement resources.

1.1 Background

PWGSC is Canada’s largest public purchaser of goods and services. Each year it purchases more than $10 billion of goods and services and approximately 60,000 contractual documents are managed. Since 1993, the procurement arm of PWGSC has had the responsibility for coordinating and delivering training and professional development to its procurement personnel.

In 2004, the Task Force Report on the Government-Wide Review of Procurement identified procurement as a key central government activity. It further specified that the government procurement community must be supported by the acquisition of the appropriate procurement skills, training and expertise as an essential element of an improved procurement process within government.

Since human resource capabilities are an important ingredient for delivery of procurement, a formative evaluation was planned to assess PWGSC’s progress towards meeting the goals of professional development for its procurement resources. This formative evaluation applies only to procurement related training and professional development provided in the Acquisitions Branch. While the Professional Development and Change Management Directorate is tasked with providing or facilitating the provision of all training (e.g. pre-retirement, communications, writing), this evaluation addressed only procurement related professional development.

1.2 Methodology

The methodology employed for this project adhered to the Audit and Evaluation Branch standard approach for formative evaluation work, which included the development of an evaluation framework. The work was executed through document and background reviews, stakeholder interviews, review of administrative data and comparison, where possible, with similar programs in other countries.

2. Evaluation Framework - Profile

2.1 Component Profile

The discussion of the component profile for professional development is structured according to the following elements:

1. Origin
2. Governance Structure
3. Resources
4. Delivery Approach
5. Target Populations
6. Planned Objectives and Results

2.1.1 Origin

In April 1993, Supply Operations Services Branch (SOSB), now Acquisitions Branch (AB), was given the responsibility for coordinating and delivering Professional and Technical Training programs for procurement personnel. The Training and Development Division was established to carry out this responsibility. This group organized professional and technical training (procurement-related) and worked with Human Resources Branch to address non-procurement training courses for SOSB staff.

By 1998/99, the procurement training curriculum was further expanded to address issues of complex procurement. In addition, in 2000-01, an internship program was established within the procurement community, also administered by the Training and Development Division.

In 2003/04, AB centralized its professional development, training services and change management activities to provide a uniform and consistent approach to meeting future capacity requirements. The Professional Development and Change Management Directorate (PDCMD) was formed as the result of the merger of three organizations providing professional development: Training and Development Division; E-systems Training - Automated Buying Environment; and Major Project Service Line.

This new organization also has a leadership role to contribute to the professional development of the procurement community within the public service and to contribute to the development of strategies that ensure that the government has enough sufficiently skilled procurement personnel to manage the business of procurement.

2.1.2 Governance Structure

Figure 2.1 provides an overview of the governance structure for professional development within AB. The Director of the PDCMD reports to the DG of Risk, Integrity and Strategic Management Sector.

Figure 2.1

The PDCMD is comprised of four divisions. The Professional Development Division (PDD) is responsible for the development of strategies and the conduct of needs assessments for the procurement community. The PDD is responsible for the administration of training services, and the development and delivery of procurement courses, such as internal initiatives or programs with academic institutions or private sector organizations. The Human Resourcing Division administers the Intern Officer Program as well as recruitment along with the HR Management Division.

2.1.3 Resources

For the 2004-2005 fiscal year, the PDD was managing the AB training budget of $600,000, which includes an amount of $78,000 set aside for Intern Officer training. The following table outline the resources allocated to professional development within the PDCMD.

¹ This includes a portion of the time of the Director and the Director's offices staff allocated to professional development.

A small percentage of cost for the other divisions of PDCMD can also be attributed to professional development and these include the Intern Officer Program in the HR Resourcing Division and financial support within the HR Transaction Division.

Over the last 3 years, the PDD indicated that its training budget had been reduced significantly but the number of employees to be trained has increased. PDD has recently lost two of its experienced senior procurement trainers who were the first level core course instructors (Introduction to Procurement, Contracting for Services and Bid Evaluation). New bilingual trainer resources have been hired; consequently, these new resources might be experiencing a learning curve that could impact course delivery timeframes.

2.1.4 Delivery Approach

The administration and delivery of procurement training within the AB is managed by the PDD, which provides guidance on training opportunities; processes Personal Learning Plans; registers employees and processes payments for training; monitors training reports and coordinates and plans procurement continuum training courses.

The continuum is comprised of entry level, intermediate level and advanced level courses. Procurement continuum courses are provided to AB employees, and to Other Government Departments (OGDs) and other professional organizations on a tuition fee basis. In-house PDD trainers provide formal training to PWGSC employees, OGD’s and organizations. Of those, some provide training in systems. Other training courses are provided by Subject Matter Experts (SMEs) within PWGSC, or they are contracted out to external resources.

The PDD also promotes and coordinates the provision of commercially available procurement training courses with the National Institute of Government Purchasing (NIGP) and the Purchasing Management Association of Canada (PMAC). These organizations offer national and international professional development programs.

Appendix A includes a comprehensive list of courses offered through the PDD in the past two fiscal years. Immediately below are the primary courses offered by PDD directly and indirectly employing outside service providers.

Existing Procurement Training Courses:

1. Provided by PDD instructors:
   • Introduction to Procurement Training
   • Contracting for Services
   • Automated Buyer Environment Training
   • Government Electronic Tendering Services
   • ABE Contract Management Milestone Pilot Training
   • E-Purchasing training
     
     
2. Delivered by Subject Matter Experts with Facilitation
   • Limitation of Liability (LOL) Training
   • Risk Management (RM) Training
   • The PMPlus Professional Development Program
   • Introduction to Intermediate Complex Procurement Program
   • Green Procurement (Delivered by outside instructor)

Intern Officer Program

The Human Resourcing Division, also within the PDCMD, provides the training, management, advice and administrative support in regards to the Intern Officer program. This program is a key part of PWGSC’s human resources renewal plan. Its objective is to address immediate and mid-term skills shortages by providing Intern Officers with the skills, knowledge and experience to plan, negotiate and administer contracts. It is a two-year program in which new procurement recruits must complete required training and gain appropriate experience in accordance with the standards of competence established for the program. The internship consists of four (4) six (6) month assignments which include monthly trainee meetings, formal classroom training (20 days of training for the Procurement Continuum) and on-the-job training. After one year, a PG 1 passes before a review panel and is appointed to a PG2.

Organizational Performance and Learning

The Organizational Performance and Learning (OPAL) area of Corporate Services, Human Resources and Communications Branch is responsible for the support and enhancement of continuous learning of the individual, team and organization in support of PWGSC business plans and priorities. OPAL also ensures that the Integrated Training System (ITS), which manages the accurate processing of Personal Learning Plans (PLPs) and scheduling of HR courses and Procurement Continuum course training for Acquisition Branch, meets PWGSC user requirements. The ITS helps maintain the integrity, efficiency and effectiveness of the system and provides all related training and support services.

Treasury Board Secretariat – Professional Development and Certification Program

The Professional Development and Certification Program (PDCP) was initially developed for procurement specialists and then expanded to include all employees involved in life-cycle asset management functions. As one of the key elements of the PDCP, the Canadian General Standards Board have developed a set of competencies and associated knowledge, experience and training requirements for the federal government procurement, material management and real property community. The PDD is a key player in the development of the PDCP.

The Canada School of Public Service

The Canada School of Public Service (CSPS) has been working in partnership with TBS (and PWGSC) to develop or redesign material management courses related to the certification program and has been delivering courses which include general procurement content to some government departments (i.e. Project Management). The CSPS has been designated as a focal point for government-wide training and professional development and is intended to operate on a cost-recovery basis. They are expected to be the main course provider for the TBS Certification Program but PWGSC (specifically the PDD) is expected to be the training entity responsible to provide the procurement component of the program.

2.1.5 Target Populations

The primary target group for professional development has been the procurement community within AB, PWGSC (approximately 1800). In addition, subject to availability of spaces, the PDCMD has made training available to other branches of PWGSC and to procurement staff in other federal government departments and agencies.

Recent initiatives, in particular, PWGSC’s “The Way Forward” and the Task Force Report on Government-Wide Procurement indicate the government’s intention to transform the procurement process over the next three years. This transformation is likely to have implications for the scope of responsibility of the PDCMD with respect to professional development within the larger federal government procurement community.

2.1.6 Planned Objectives and Results

While there are no formally articulated strategic objectives for professional development of procurement resources, the operational objectives are as follows:

• support the effective delivery of procurement services within AB by providing procurement related training services;
• meet employees perceived training needs for procurement; and
• respond to specific departmental requests to provide specific procurement related training for employees.

During the course of this project, the evaluation team, in conjunction with program officials, developed immediate, intermediate and long term or strategic outcomes for professional development, based on a generic, life-cycle based model. These can be considered to form the value proposition for professional development of procurement officers in PWGSC.

2.2 Logic Model

The logic model, presented in Figure 2.3, illustrates the activities, outputs and outcomes based on a comprehensive life-cycle approach to professional development. PDCMD management, working with the evaluation team, has developed and confirmed this logic model as being representative of the direction towards which professional development in AB for procurement should evolve.

2.2.1 The Life-Cycle Approach to Professional Development

Based on research by professional development specialists, an effective professional development program should incorporate a comprehensive, life-cycle based approach. This approach should include the following critical elements, as illustrated in Figure 2.2:

• Alignment of professional development strategies and plans with their business mission and corporate goals;
• Continuous gap analysis to determine professional development needs;
• Design and delivery of professional development programming; and
• Evaluation of professional development strategies and activities to determine effect, including business impact.

Professional development must be firmly and clearly aligned with corporate goals, and supported by meaningful performance indicators for management and reporting purposes. When organizations ensure that professional development strategies are aligned with the business needs identified in their corporate plans, they are able to demonstrate the efficiency and effectiveness of their investment in professional development and its contribution to organizational effectiveness.

Figure 2.2

2.2.2 Generic Logic Model

The life-cycle model shown in Figure 2.2, above, has been accepted by PDCMD management as describing, at a high level, the approach that they now intend to employ.

The logic model in Figure 2.3 and the underlying life-cycle model upon which it is based are consistent with what is perceived as best practice in the professional development community. PDCMD management has acknowledged that the logic model represents a fully matured program logic that does not currently completely exist in PWGSC. However, they will be evolving a Professional Development Program (PDP) towards that objective, recognizing that they will face significant challenges in attempting to fully implement this model. It is noteworthy that officials in other countries when interviewed by the evaluation team indicated that they are facing similar challenges.

Figure 2.3 (View detailed figure by clicking on link)

2.2.3 Activities, Outputs and Immediate Outcomes

There are four primary activities undertaken by a Professional Development Program (PDP) related to procurement. These activities and the outputs and immediate outcomes resulting from them are described below.

Activity #1: Align the Professional Development Program with government, department and branch needs and priorities

This activity encompasses:

• identifying governmental, departmental, and branch policies and strategic directions for procurement;
• determining the priorities related to required competencies for the procurement community and identification of associated processes for professional development; and
• defining the guidelines, polices and standards that will guide the design and delivery of professional development.

The two outputs that flow from this activity are:

• Strategic objectives, statement of government, department and branch need, priorities and objectives
  
  This is the output that arises from an understanding of the importance or significance of procurement training/professional development at each of the organizational levels: government-wide, departmental and branch. The output should be the actual statement of objectives for each professional development level and there should be three separate specific statements related to procurement training that support each separate level of objective. In each case the PDP will specifically acknowledge what it will deliver against each of the three levels and generate a statement that this forms part of its mandate.

• Professional Development Program governance structure, delivery guidelines, policies and standards
  
  This output consists of the frames of reference within which the program is to be designed, delivered, administered, governed and the level of resources and standards to which the program is to adhere. This provides a policy and standards framework within which all applicable programs are delivered. This output is recognition of the interplay between the stakeholders that may be involved in delivery of the program. It resolves the questions of who reports to whom and who might be the primary delivery agents. It determines who is involved in design and what, if any, certification approaches are considered. This is the element that provides coherence for the program at each level.

Activity #2: Design Professional Development Program delivery approach; Gap analysis

The basis upon which the design of the PDP’s service delivery is conducted is grounded in the completion of a gap analysis which contrasts the existing PDP service delivery design with the service delivery design which should exist according to the generic life-cycle logic model. The correction of the gaps identified through this means will then form the basis of strategic plans to redesign the PDP to address demonstrated governmental, departmental, branch and individual procurement training needs and priorities.

There are three outputs for this activity:

• Professional Development Program strategic implementation plan
  
  At a program level this is the document that describes how the program will implement the directions that it has received. Following the gap analysis and the design of the professional development delivery approach, this plan describes how the professional development program will be implemented and the resources necessary to carry that out. This output will be produced as often as the activity #2 is undertaken.

• Individual Learning Plans
  
  Individual learning plans should link to the branch level plan as well as the departmental level plan. Based upon the gap analysis, the aggregated individual learning plans should demonstrate the extent to which the branch level, departmental plans and indeed governmental plans are captured. The extent to which individual learning plans contribute to the overall organizational professional development strategic plan is a critical linkage.

• Continuum of relevant procurement and other courses and other delivery mechanisms
  
  This output is the continuum of procurement and other courses and other delivery mechanisms that the program has identified - as a result of the gap analysis - consistent with the overall program delivery approach. This output represents the optimum delivery mechanisms for the delivery of professional development for the four levels over the following year(s). This is likely an annual output and would articulate how the courses/curriculum/training are actually addressing the identified gaps. For example if a gap in relation to a particular subject area (i.e. contract management) is identified, then the continuum should articulate how the needs are met, not merely state the courses available.

The immediate outcome flowing from activities #1 and #2 and related outputs are:

• Optimized professional development program fully responsive to all requirements
  
  This immediate outcome is a reflection of how the PDP has responded to all four (governmental, departmental, branch and individual) levels. It reflects the emphasis that the PDP will place on each of the four levels. For example, in the first year it may choose to emphasize government priorities rather than departmental, or it may choose to emphasize branch priorities rather than individual. This is an explicit recognition of how the PDP, in fact, is delivering its program within its resource envelope.

Activity #3: Professional Development Program delivery

This activity describes the technical aspects of the PDP delivery, including the program of course curricula, training course instructional materials and tools and delivery modalities. It is this activity that allows the PDP to determine what is the optimum manner to meet the demand identified in the first two activities.

For this activity there is a single output with many individual subsets of outputs.

• Training manuals, courses guidelines, trained facilitators and course schedules
  
  This output defines the mechanisms, tools and products that form the actual delivery of the program. The materials that are used are to be consistent with the standards, and policies that guide the course design and delivery.
  
  These include:
  
  • Individual Course Training Manuals
  • Individual Courses Guidelines
  • Individual course training and support materials (checklists, templates, contracts, contribution agreements, evaluation grids, selection criteria, model requests for proposal)
  • Teaching and training guides (presentations, tests, lesson plans)

The immediate outcome arising from this activity is:

• Knowledge and skills of procurement officers align to competencies for their positions
  
  This immediate outcome reflects the degree to which procurement officers’ skills and knowledge are suited to the positions that they occupy and the services that they deliver.

Activity #4: Professional development program monitoring and evaluation

This activity describes the manner in which the program is monitored, adjusted and reviewed to determine the degree to which it delivers against stated objectives and how successful it is.

There is one output from this activity.

• Client satisfaction studies, course monitoring, performance measurement, etc
  
  This output includes the tools used by PDP management for monitoring and reviewing both individual courses and course delivery mechanisms and alternatives. The results are to be used as a basis to revise course design and delivery on a regular basis. This will help to assure high quality course design and delivery. It also includes reviews and studies by management and any formal program reviews that address issues of relevance and success. The relevance and success of the program can be evaluated by considering the degree to which courses have addressed the governmental, departmental, branch and individual needs and priorities.

There are two immediate outcomes from this activity.

• Enhanced course participation, course design, cost effectiveness, and relevance of curriculum
  
  This immediate outcome reflects the degree to which course delivery, design, cost effectiveness and even participation have been optimized based upon the effective use of monitoring, review, and evaluation of how existing professional development activities have been themselves refined based upon the evidence gathered from existing courses and training methods.

• Improved ability to anticipate and manage capacity and future demand
  
  This immediate outcome reflects the degree to which the PDP is aligned with stated government, department and branch needs and objectives. As well, it speaks to the degree to which the program is able to adjust to changes in trends and demand.

2.2.4 Intermediate and Ultimate Outcomes

Following from the immediate outcomes of more competent and skilled procurement practitioners as well as from improved course participation, course design, cost effectiveness and relevance of curriculum, the intermediate outcome that should be seen is:

• Adequately skilled procurement resources to deliver organizational and government procurement needs

Achievement of this, in turn, should lead to the ultimate outcome:

• Effective government procurement and enhanced value for money procurement services for Canadians.

3. Evaluation Framework - Issues Matrix

This chapter includes a discussion of potential evaluation issues, questions and methodologies for an evaluation of the PDP. The main sources for the development of the evaluation issues, questions and indicators are the outputs and outcomes of the PDP described in the previous chapter.

3.1 Evaluation Issues and Indicators

The evaluation issues and indicators in the evaluation matrix are set out in Table 3.1 on the following page and are discussed in the following sections of the report.

3.1.1 Align Professional Development Program with Government, Department and Branch Needs and Priorities

Issue: Has the program established a clear policy, accountabilities and planning framework?

Clearly defined objectives, policies, priorities, standards and requirements are important to the PDP for several reasons. Firstly, they provide a framework for decisions around the design of the program; what professional development to deliver, and to whom. Secondly, they provide clear direction to the delivery agent(s) for the program. Thirdly, they provide a basis for monitoring, reviewing, and evaluating program performance in terms of achievement of objectives and adherence to values and principles. Finally, they ensure clear understanding of the responsibilities and accountabilities of the PDP, employees, supervisors, delivery agents and others for ensuring that effective procurement-related professional development is provided to all employees who require it.

Good planning provides a basis for identifying the need for program; for estimating financial and other resources required; for negotiating with potential delivery agents; and for measuring achievements, in terms of numbers of procurement officers receiving professional development support.

This question aims at determining the extent to which the PDP has established a clear policy, accountability and planning framework. Key indicators for this question are:

• Strategic/Operational Plans
• Well-defined program priorities
• Program budget appropriate to achieve objectives
• Program delivery standards and guidelines
• Well-defined accountabilities for program design, delivery, monitoring and evaluation and participation.

3.1.2 Design Professional Development Delivery Approach; Gap Analysis

Issue: Is there an effective process for identifying training requirements for individual procurement practitioners?

An effective needs identification and referral process would include:

1. an effective communications program to ensure employees and their supervisors are aware of the importance of professional development; and of how to access information about professional development;
2. availability to all employees of tools – ideally online – for assessing their competencies against the required competencies of their classification or responsibility level; for identifying professional development opportunities that would meet their needs; and fro registering for these activities.

An ineffective process for identifying and referring candidates to the PDP could have several undesirable impacts on the program. These include a failure to achieve program objectives for the number of procurement community members trained; underutilization of facilities; and, most importantly, failure to provide training to candidates who could have benefited from it.

Key indicators of an effective referral process are:

• All employees have access to competencies-based needs assessment /professional development registration tools
• % of eligible, targeted procurement community members who enroll
• % of eligible, targeted procurement community members who undertake professional development activities
• Clarity of responsibilities for referral candidates for professional development
• % of professional development requests met

Issue: Is there an effective process for meeting demand for training?

An ineffective process for determining the aggregated level of demand for the various training courses and other professional development activities and for implementing plans to meet that demand could result in a failure to provide employees with the training they have requested. This would have a dramatic impact on the effectiveness of the program.

Key indicators that the PDP can respond effectively to demand are:

• % of identified training requirements that the PDP is able to meet each year;
• Course subscription rates;
• Course attendance; Elasticity of capacity management

Issue: Does the program have the capacity to respond effectively to corporate directives for professional development?

The PDP has to be flexible and responsive to changes in corporate priorities for professional development. Indeed, it should be capable of anticipating shifts, the competencies that will be required and changes in the demographic composition of the procurement community. External events such as parliamentary inquiries, changes in regulations or policies can result in the emergence of new and or increased demands for professional development services. The PDP must be able to develop and deliver professional development in response to these directives within timeframes that are acceptable to senior management.

This question aims at determining the extent to which the PDP can respond or even anticipate the need to adjust professional development training to address changing needs.

Key indicators for this question are:

• Ongoing operational planning processes
• Number of professional development course modifications
• Timeliness of response to directives for new courses or other professional development activities

3.1.3 Professional Development Program Delivery

Issue: Are appropriate training delivery systems in place?

Indicators here would probe the deliverables actually provided in support of the training. More specifically those indicators could include:

• Existence of well-defined learning objectives
• Consistency/Comprehensiveness of course content vis-à-vis learning objectives
• Effectiveness of learning approach
• Appropriate course facilities/class sizes
• Training of course providers
• Appropriateness/effectiveness of instructional materials (manuals, handouts, videos etc) (e.g. level of information, currency)
• Availability of instructional materials at all sites/ to all students in both official languages.
• Perceived effectiveness of course providers

3.1.4 Professional Development Program Monitoring, Review and Evaluation

Issue: To what extent is a systems approach used to monitor, review or evaluate program (as distinct from course) delivery?

The development and use of participant feedback is the first level source of monitoring and performance information normally available to training programs. Feedback from participants, in the form of post-course assessment forms, follow-up surveys or other forms can provide information useful for making adjustments to course design or content or for evaluating instructors or facilitators.

These should be supplemented however, with periodic androgogical assessments of courses and instructors to ensure that the course designs and delivery reflect best practices in adult education and that the content of the courses is clearly linked to course objectives.

From a program point of view it will be important to track:

1. the percentage of the population that is receiving training that meets identified needs; and
2. the extent to which the complement of courses addresses the full range of d priorities identified in the planning processes.

3.1.5 Program Results/Impacts

Issue: To what extent has the PDP contributed to participants’ ability to provide improved procurement services on behalf of PWGSC?

The PDP aims at improving participants' ability to provide improved procurement services on behalf of PWGSC. Ideally this question would be answered by comparing participants with non-participants. However, it can be very difficult to accurately survey these different populations in an effective manner that isolates training as the key factor. Another alternative would be the use of baseline and subsequent follow-up surveys that would provide “before” and “after” comparisons.

Nevertheless, given the importance of this question, it will be important to attempt to establish measures of whether the initiative makes a difference to PWGSC.

Potential indicators that could be used to address these questions include:

• Perceptions of management regarding improved knowledge
• Perceptions of procurement community clients
• Perceptions of PDCMD officials
• Perceptions of clients
• Percentage of identified competencies acquired by procurement community/practitioners

3.1.6 Cost Effectiveness/Alternatives

Issue: To what extent are the resources appropriate given current and anticipated roles and responsibilities?

Indicators that would assist in assessing this include:

• Cost per training activity
• Cost to achieve individual competencies

Issue: Are there alternative approaches to management and to course delivery that are more cost effective or efficient?

Indicators that would assist in determining the comparisons here would include:

• Cost effectiveness of alternative course design and delivery mechanisms (CSPS, Private Sector/In house, NIGP, PMAC)
• Potential for full recovery of course delivery costs

3.2 Evaluation Methodologies/Data Sources

This section describes data sources and methodologies identified in the Evaluation Matrix.

Document Review

These consist of planning and management documents such as:

1. strategic and operational planning documents related to the PDP itself;
2. financial planning documents;
3. program policies, standards and guidelines; and
4. documentation on program priorities.

Administrative Data

Administrative data refers to numerical data stored by a variety of means including shared and stand-alone databases, spreadsheets and paper records. For this formative evaluation, the evaluation team made use of program statistical data on the population of courses offered and on the numbers of participants, and of program financial data.

Stakeholder Consultations

A number of internal and external stakeholders in the PDP were consulted/interviewed. Such stakeholder consultation was an important methodology for this formative evaluation.

Androgogical Assessment by Learning Expert

While not employed as part of this formative evaluation, an assessment of the PDP from the point of view of learning theory and practice, especially adult education principles, should be part of any future evaluation. The androgogical assessment would focus on the design of the PDP; the extent to which the content and materials support the learning objectives; the effectiveness of the facilitation approach; the adequacy of facilitator training; the adequacy of facilities and equipment and ancillary services; accessibility of training; and the use and effectiveness of post-course feedback and evaluation questionnaires.

Surveys

While not part of this formative evaluation, survey instruments could be developed to support and complement existing administrative data as well as to seek out critical answers to questions that impinge on expected results or program impacts. This methodology would be specifically useful in assessing both the degree to which professional development participants valued the training as well as assessing the degree to which organizations that have sent participants perceive the value of the professional development activities. Two types of survey instrument could be useful in this evaluation: surveys of referring agents/clients; and surveys of course participants.

4. Formative Findings and Conclusions

This chapter provides current formative findings and conclusions related to the evaluation issues and questions set out in the previous section. The professional development of procurement resources as administered by the PDCMD is referred to as the Professional Development Program (PDP).

In the course of developing the evaluation framework, it became clear that professional development for procurement resources in PWGSC was operating within a climate of significant change both organizationally within the Department, during the last two years, and governmentally with respect to the procurement function and that it lacked some of the key elements of a comprehensive, life-cycle based professional development program. Consequently, the evaluation team proceeded on the basis that:

1. it was premature at this time to attempt to fully evaluate the program against all of the issues identified in the evaluation framework, given that certain issues could be addressed based on the research conducted in the course of developing the evaluation framework and on the analysis of available information, and other issues could not; and
2. this formative evaluation could best provide value to management by developing am evaluation framework using a logic model for the PDP based on a generic life-cycle model of professional development programs, assessing available program information and undertaking some comparative research into other jurisdictions.

The evaluation issues, with formative findings and conclusions are discussed in the following sections.

4.1 Align Professional Development Program with Government, Department and Branch Needs and Priorities

Issue: Has the program established a clear policy, accountabilities and planning framework?

FINDINGS

There is no formal policy and planning framework in place specifically for procurement professional development within AB. A policy and planning framework would include a clear statement of strategic, corporate objectives and goals for professional development in procurement as well as guidelines, policies and standards for the design and conduct of professional development activities. In the absence of such a framework, the PDP in AB is little more than a training program reacting on an ad hoc basis to organizational and individual demands.

The PDP’s policies and procedures are in accordance with PWGSC HR policies and procedures (i.e. Departmental Learning Policy, PWGSC Learning Strategy, Minimum Training Investment and PWGSC Framework on Apprenticeship or Professional Training Program). There is also the recognized procurement certification program that is being developed by TBS. The PDP has been actively involved with the development of this program, which is based on an established standards developed by the Canadian General Standards Board. To fully implement the TBS Certification Program, PDP will need to determine the level of funding required.

As part of the evaluation of this issue, the evaluation team contacted officials in the US, Australia and the United Kingdom to determine how professional development of the procurement community is managed in those jurisdictions. Some of the governments contacted provided responses to interview questions posed but the primary source of data was their web sites.

The US has a strong governance structure in place with legislation and policies specifically related to procurement training. The Federal Acquisition Institute (FAI) offers and facilitates a variety of training programs and initiatives. Nevertheless, they do not have a comprehensive common program for the whole of their civilian procurement workforce. They are currently working on an overarching federal policy to provide common, mandatory training to all procurement resources nationally.

The UK appears to be somewhat more advanced then either Canada or the US in terms of strategic management of the procurement function and of professional development in the procurement community. The UK Office of Government Commerce (OGC), an independent Office of the Treasury, is responsible for improving the efficiency and effectiveness of government procurement. The OGC has developed a Skills Framework for the program, project management and procurement communities and a self-assessment tool, which is available on their web site that can be used by the procurement community to assess their skills. However, they do not appear to have linked the skills to specific job classifications or responsibilities. Rather, individuals can use the framework to assess the skills they require to carry out the various roles required in their current job. They can then assess themselves against the required skills and identify gaps. Ultimately, the framework will be supported by the addition of training paths and crricula for skill areas. In general, procurement practitioners are expected as a minimum to complete the Certificate of Competence in Purchasing and Supply. Higher level posts require further qualifications such as the Chartered Institute of Purchasing and Supply's (CIPS) MCIPS graduate diploma, or an equivalent qualification in a relevant discipline. The UK is currently undertaking a strategic review of how the government procurement community might be structured and managed.

In Australia, while the Commonwealth Government does not have a procurement competency development program, most of the Australian states do. For example, the South Australian Government has developed the Procurement Capability Development Program (pCAPd) that has specific stated objectives. The program is optional. They are closely working with the Australian subsidiary of the Chartered Institute of Purchasing and Supply (CIPS), other Australian governments and private sector firms to look at professional accreditation at the post graduate level.

Based on the above, management of professional development of the procurement community in these other states does not differ greatly from the way it is managed in Canada. However, it is noteworthy that these other governments are all looking at more strategic approaches to ensuring their procurement communities have the skills and knowledge to maximize their effectiveness.

Within Canada, the procurement function of the federal government and PWGSC is going through further transformation as a result of the Task Force Report, the recent Budget and the Way Forward. Further changes are likely over the next few years. This climate of change will make it even more imperative that PWGSC have in place a strategically driven professional development program. The following questions will need to be comprehensively addressed:

• What will the governmental, departmental and branch priorities for a professional development program for the procurement community be?
• What will the competencies required of the procurement community be? Who will play the lead role in defining these?
• How and by whom will the professional development activities be designed and developed?
• How and by whom will those professional development activities be delivered?
• What will the quality control/monitoring of PD activities be?

CONCLUSION

• There is no formal policy or planning framework in place specifically for procurement professional development to guide program design and delivery in AB.

4.2 Design Professional Development Delivery Approach; Gap Analysis

Issue: Is there an effective process for identifying training requirements for individual procurement practitioners?

FINDINGS

For the past several years there has been an annual process for identifying and referring candidates for professional development. It starts with a call up letter to all employees advising them that they may now plan what training they would like in the next year. Each employee completes this form in consultation with his manager, who must approve the course selection. An employee may select from a range of general courses as well as those specialized in his or her area of concentration or expertise. This includes both procurement related and non-procurement related courses. (e.g. pre-retirement planning or complex procurement strategies). The manager is to balance his or her unit’s overall human resource requirements for the upcoming years and then determine if the employee’s course or training selection is appropriate. These forms, once completed, are sent in hard copy to the PDP.

There is no branch wide or department wide framework against which specific professional development training needs are assessed. A manager is without any formal departmental or branch guidance as to what training he or she should be emphasizing. They can employ the TBS Certification Program Standards to identify competencies. There is also a detailed list of departmental competencies identified under procurement reform. These various competencies would need to be reviewed with an eye to creating a skills framework based on designated procurement competencies. However, due to the diversity of the Acquisition Branch unique skill sets, training must also be addressed as required.

As noted earlier, the UK has developed a Skills Framework of essential skills required by individuals working in the procurement area and a self-assessment tool procurement personnel can use to assess their skills against the skills framework. PWGSC needs to implement this kind of approach and link the skills framework to specific professional development activities that employees can take advantage of to develop or augment their skills.

CONCLUSION

• There is a process for identifying and referring candidates for professional development but needs are not determined on the basis of an assessment against a skills framework based on designated procurement competencies.

At the moment, the PDP lacks some key elements of an effective process for ensuring that AB procurement resources can identify their skill requirements and the specific professional development activities that can meet these requirements.

Issue: Is there an effective process for meeting demand for training?

FINDINGS

From the point at which course selections are made known to the PDP, a structured process is followed to determine how best to match the demand for courses with the availability of professional development services and programs. All individual learning plans are reviewed by PDP personnel and the priority criteria for course selection that has been indicated by the employee are reviewed.

PDP personnel assess the aggregate demand for courses to determine if sufficient numbers are selecting particular courses within particular timeframes. The information currently available in the PDP did not permit a comprehensive assessment of whether the PDP is able to meet all demands for training.

It should be pointed out that the other countries about which the evaluation team obtained information are not much further ahead in estimating and meeting demand for professional development.

The US does have a competency–based development guide although it is not linked to levels. They do have an Acquisition Career Management Information system, which provides information and produces reports on the acquisition workforce. However, they are still struggling with identifying demand and they rely mostly on estimations.

The UK has on-line skills assessment but has not yet implemented on-line selection and registration processes for professional development activities.

The Australian government has guidelines for identifying procurement certification and development needs. They rely on PACCER, who develops courses based on the established national Procurement Contract Management Competency Standard. The program is then based on the units of competency of the National Public Services Training package (PSETA). The competency standards cover a very broad range of subject groups including procurement. Each of the competencies is set at a particular level and has clear standards of knowledge and experience. The competencies are designed to be measured on the job and not just course based. They run their training programs to address these competencies and a student who completes a listed set gets a recognized, portable qualification which will give the credit in further studies.

CONCLUSION

• Sufficient information was not available to enable drawing a firm conclusion on this issue.

In the PDP, there is a process for assessing demand, based upon compilation of individual learning plans. However, in the absence of program data on the percentage of training requirements met, the evaluation team was unable to determine how effectively the PDP meets this demand.

Issue: Does the program have the capacity to respond effectively to corporate directives for professional development?

FINDINGS

One key indicator of an effective training program is its ability to respond, in a timely fashion, to directives from senior management for the design and delivery of training in specific areas. In this regard it is worthwhile noting that the PDP has more recently concentrated on responding to senior level direction to training delivery with the emphasis on environmental or green procurement and specific issue-based course delivery (e.g., fraud awareness and contract management). Based on interviews with program officials, the PDP has been able to design and deliver this training effectively and in a timely fashion. Further, as Table 4.1 indicates, the PDP has provided training in these and in more traditional areas related to procurement to large numbers of individuals within PWGSC and from other departments.

Because PWGSC is part of a government transformation of the procurement process over the next three years, a repositioning of the PDP is imminent to meet new needs in PWGSC and perhaps others in the larger federal government procurement community. Since these needs are not yet precise, the evaluation team did not pursue research or testing on this issue at this time, beyond the initial interviews conducted with program management and other stakeholders as part of the process for developing the evaluation framework. Consequently, the capacity of the PDP to effectively respond is an unknown at this time.

CONCLUSION

• Conclusions cannot be drawn on this issue at this time, based on the methodologies employed in this formative evaluation.

While the program appears to respond well on an ad hoc basis to senior management requests for additional or changed program delivery requirements, this does not form part of a coherent annual planning or demand identification process nor is the PDP a key participant in the overall corporate level procurement program planning function. The PDP is currently in the planning phase of repositioning itself to meet the new needs in PWGSC. The PDP has indicated that it will be looking at annual planning and demand identification with an eye to participating in the overall corporate level procurement program planning function.

4.3 Professional Development Program Delivery

Issue: Are appropriate training delivery systems in place?

FINDINGS

This is very difficult to report on at this time, as the program data currently available does not form a basis for evaluating whether program delivery systems are appropriate. Training and other professional development activities are undertaken mainly by the PDP and also by the Canadian School of the Public Service (CSPS) and by outside organizations such as the Procurement Management Association of Canada (PMAC) and the National Institute of Government Procurement (NGIP).

The two governments for which the evaluation team was able to obtain information rely heavily on the private and non-governmental sectors for course delivery. The USA uses private contractors for all course delivery. They also have strategic partnerships with universities, other government departments and various associations. Likewise, the South Australian government has engaged an external firm to deliver all training. The PDP is unique among these three jurisdictions in that a large part of training administration and delivery is done in house.

At present, there is no plan of delivery that utilizes a structured series of course delivery options (i.e., in-house, private sector, NIGP, CSPS, etc.) Training is delivered efficiently in the sense that courses are scheduled, delivered, attended and intermittently evaluated by participants; however, from a systems point of view it cannot be said that the most appropriate system is in place. With training being provided by multiple organizations and with no current common standard for training delivery that is related to a common set of targeted competencies it cannot be said that the training systems are appropriate or not, and it is difficult to identify the training provided as forming part of an overall “system”.

Training needs ought to respond to changes in the emphasis placed on procurement by government, the department and the branch. These are currently in a state of flux. There is currently a lack of clarity as to which organization (i.e. PWGSC, CSPS) will take the lead in the professional development related to procurement. Target populations may change and with it the nature of the training system. A related aspect is whether the PDP is making the most effective use of its internal resources in the way that it designs, develops and delivers training and other activities. As noted above, other governments appear to be moving towards the use of external delivery agencies to deliver professional development activities. This approach has some advantages:

1. it releases internal resources to focus on more strategic aspects of the professional development program; and
2. it allows for flexibility in responding to fluctuations in demand, something that is more difficult when internal staff are delivering training.

Furthermore, if the PDP is given responsibility for professional development of the broader procurement community within the federal government, it is likely that internal resources will not be available for it to continue to play a direct role in delivery of training.

PDP is currently establishing a MOU with DND for the delivery of Procurement Continuum courses to other government departments. The PDP would need to determine what additional resources would be required to be able to effectively work with other potential partners in delivering a comprehensive professional development program (i.e. TBS Certification Program).

CONCLUSION

• While the PDP appears to be effective in delivering training, it may be considered to be less effective in the sense of working with potential partners to deliver a comprehensive professional development program, while optimizing the use of internal resources.

Based on interviews with PDP officials, other PWGSC officials and officials of TBS and the CSPS, and on our review of the PDP course delivery database, while the PDP appears effective, in the sense of providing training relevant to the needs of the procurement community, it does not appear to be working in an integrated manner with other stakeholders at this to provide an integrated and coordinated set of professional development activities that are clearly linked to specific competencies of the AB procurement community.

With respect to optimizing the use of internal resources, it is not possible to determine whether existing delivery systems are efficient. Currently, the PDP appears to be meeting the existing level of demand for professional development with internal staff playing a direct role in training delivery. However, it is not clear whether program resources would be adequate for the PDP to implement a comprehensive life-cycle based approach to professional development or for it to exercise an expanded role and mandate in professional development across government, without changes to the way it makes use of internal staff.

4.4 Professional Development Program Monitoring and Evaluation

Issue: To what degree is a systems approach used to monitor and evaluate program (as distinct from course) delivery?

FINDINGS

PDP participants in some courses complete individual course evaluations of the training that they have received. The PDP evaluates and monitors its courses, either through selected interviews or in writing for the degree to which training supports business objectives, the degree to which on-the-job performance improves, the consistency of delivered training with departmental and service-wide policies and standards. These documents are completed in hard copy and are supplied to the PDP, however, they are not systematically used to review the appropriateness of the course, the delivery approach, nor are they used by the PDP to review program choices about how courses or groups of courses should be delivered. There is no comprehensive approach to assessing the overall program of professional development or training in terms of specific topics, areas of specialization or methods of professional development.

The PDP has indicated that it is currently investigating the feasibility of acquiring a Management Learning System to support a comprehensive approach to assessing he overall program of professional development.

The US is struggling with the same issues as Canada in terms of monitoring and assessing their program. They have no means by which to measure the effectiveness of training, however, they are working on an internet-based solution that which would enable the compilation of course evaluation sheets and would provide a forum to conduct post course evaluations.

In South Australia, the contract with the external firm used by the South Australian government stipulates that a review of the course curriculum is to be performed every six months. They have not gone through this process yet as the program has been in operation for only six months.

CONCLUSION

• The PDP does not currently use a systems based approach in the conduct of program or course evaluation.

4.5 Program Results/Impacts

Issue: To what extent has the PDP contributed to participants’ ability to provide improved procurement services on behalf of PWGSC?

FINDINGS

The impacts of professional development programs are extremely difficult to measure, for several reasons. Firstly, it can take considerable time before impacts on the workplace are sufficiently robust to be measurable. Secondly, many other factors impact on service delivery expertise besides training.

When changes are made to procurement policies as a result of TB or Departmental decisions, the PDP has indicated that it creates or adjusts course content. Also, as a result of feedback forms received from course participants, the PDP has indicated that adjustments are made to Procurement Continuum courses on a continual basis. In both these examples, the PDP has indicated that procurement staff and managers have provided comments in terms of value added to their knowledge base, which could have a positive impact in terms of improved procurement services on behalf of PWGSC.

Key indicators should involve the subjective perceptions of individuals, clients, and management along with using longitudinal surveys with baseline measures and downstream measures. Another more objective approach to measure the impacts of professional development activities might be to assess the degree to which individuals have acquired mandatory competencies essential to their positions.

At present, the PDP does not systematically collect data that would enable it to evaluate program impacts using either of the methods above. While baseline data could be developed fairly easily, based on the first approach outlined above, development of data based on the second approach is contingent upon the development of mandatory competencies linked to specific positions or responsibility levels.

CONCLUSION

• Based on the research possible within the scope of this formative evaluation, the evaluation team cannot offer any conclusions on the degree to which the PDP contributes to participants' abilities to provide improved procurement services on behalf of PWGSC.

The PDP does collect data that would allow for the evaluation of impacts of its professional development activities on the effectiveness of individuals in carrying out their procurement responsibilities. However, in order to evaluate this, longitudinal (before and after) surveys of employees and their supervisors would be required, and this was not feasible within the parameters and timeframe for this formative evaluation.

4.6 Cost Effectiveness/Alternatives

Issues:
To what extent are the resources appropriate given current and anticipated roles and responsibilities?
Are there alternative approaches to management and delivery to course delivery that are more cost effective or efficient?

FINDINGS

Currently, the PDP has responsibility for all procurement training program planning, design, delivery, and monitoring in AB. As noted earlier in this report, based on the interviews conducted, the PDP appears to be meeting current demands for training. In the absence of workload parameters/metrics, resource capacity could not be assessed. However, PDP officials indicated that their internal resources are currently stretched to meet demand. Additional demands, such as would result from the PDP being given responsibility for professional development for the broader procurement community in the federal government or from adoption of a life-cycle based approach, could mean that the existing resource mix would be inadequate in terms of number of trainers and/or expertise, to continue its current approach of involvement in all aspects of design and delivery. Since the PDP needs to examine more strategic and integrated approaches to managing professional development, options could include strategic partnerships with educational institutions, other government departments, outside service delivery organizations, and national procurement associations for the design and delivery of individual courses and activities.

The PDP would need to determine the necessary resources and funding requirements to enable these kinds of partnerships and to support the role-out of the TBS Certification Program. Due to the unique nature of PWGSC’s business, PDP anticipates that these partnerships would only be able to meet a portion of the need. It is expected that there will always be a requirement for job-specific unique project procurement training.

CONCLUSION

• Based on interviews with PDP and other officials, any mandate expansion or modification would necessitate changes to the PDP resource mix in relation to roles and responsibilities expected to be exercised by PDP staff.

Given that the future mandate may change it is not possible at this time to conclude what alternatives might be more cost effective or efficient.

4.7 Overall Conclusions

It is the conclusion of this formative evaluation that professional development for procurement resources lacks some of the key elements of a comprehensive, life-cycle based professional development program. However, the PDP has been highly reactive and agile at providing training to meet ad hoc demand and respond to senior management direction. During the last two years, professional development as been operating within a climate of significant change both organizationally within PWGSC and governmentally with respect to the Task Force and the proposed agenda for the procurement function.

Given the likelihood of an expansion or modification of its professional development mandate, the PDP will need to take a more strategic approach to its responsibilities. Consequently, the PDP will need to strategically review how it carries out the professional development program. Consideration should include how it can work with external partners, such as the CSPS, the private sector, universities and community colleges, and others to achieve its program objectives in a way that optimizes the use of core internal resources and include an examination of the existing professional development activities to assess their continued relevance and effectiveness. If its mandate is expanded or modified, the PDP in AB will have to determine the appropriate mix of resources (numbers and expertise) to discharge the roles and responsibilities expected of it.

The timing is appropriate for a program re-design which would ideally clarify the mandate, roles and responsibilities of the professional development program in Acquisitions Branch in the context of any recommendations of the Task Force and the other initiatives and define the corporate objectives and priorities for professional development in procurement, having examined options for linking competency-based certification programs (e.g. the TBS program) to PWGSC's professional development program. Part of this re-design exercise should implement a life-cycle approach to professional development, elaborated through a delivery strategy that clearly specifies the roles of PWGSC and partners for various aspects of professional development and through determination of resource requirements for implementing the renewed program.

The program re-design should also include the creation of an ongoing performance measurement strategy with associated data requirements and means to capture information. Once the program and its environment have stabilized and matured, a comprehensive evaluation of the PDP could be carried out.

5. Recommendation

It is recommended that the Assistant Deputy Minister, Acquisitions Branch:

1. Undertake a program re-design to align the professional development of procurement personnel with evolving governmental, departmental and Acquisitions Branch strategic directions and to establish a renewed mandate, objectives and priorities for professional development, a life-cycle approach to professional development, and a program delivery strategy with performance measurement capacity and with provision for future evaluation when the Professional Development Program is stable, against the evaluation framework developed.