|
|
     |
||
 |
     |
 |
|
Purpose The evaluation of the Sectoral Partnership Initiative (SPI) was planned originally to include three phases: 1) formative; 2) monitoring and data collection; and 3) summative. Since most sector councils were only early into their operational period when the SPI evaluation was initiated, a formative evaluation was conducted during 1994 and 1995. The subsequent monitoring and data collection phase was designed for implementation in two possible rounds of data collection. This report draws from the Round One data collection which was completed by mid-1997. Significant changes have impacted upon the SPI, including the federal Youth Strategy, the EI Act, and federal-to-provincial devolution of labour market training. Given these changes, it became evident that it would not be worthwhile to pursue the rest of this evaluation. The formative evaluation of the SPI dealt primarily with rationale and delivery issues. The monitoring and data collection stage of the evaluation was intended to cover all of the evaluation issues. These issues revolve around:
The emphasis of this report is on new information collected. For completeness, however, findings from the formative evaluation, based largely on eleven sector council case studies, have been integrated with this report. This report summarizes findings from both phases of the evaluation conducted to date. The Sectoral Partnership Initiative
Each sector council is not expected to meet all of SPI's objectives. In addition, the Initiative is evolving and these original objectives were supplemented with greater emphasis now being given to creating systemic changes in human resource (HR) practices. Although training participation had been funded in the past, SPI will now only fund sector councils for activities related to developing the training infrastructure and not for the councils to directly fund firms and employees participating in training. The original budget for the SPI, fiscal years 1993/4 to 1997/8, was $250 million, however, after various budget reductions the five-year budget stands at $135 million (O&M and contribution program dollars combined). As of September 2, 1997, total program expenditures under SPI were approximately $87.9 million. As of the Summer, 1997, there were 19 operational sector councils and two sector councils at the developmental stage. Two sector councils have ceased operations. Evaluation Methodology Of the 19 operational sector councils, seven were included for intensive study in Round One of the monitoring and data collection phase of this evaluation. As well, limited data was collected on the two sector councils that have ceased operations. Methods included:
For surveys done, usually a suitable comparison group could not be constructed and results reported for participants are sometimes based on relatively small numbers. For project-level case studies undertaken, interview results are generated from a success story perspective. Although the results for some sector councils are based on quantitative data sources such as surveys, each council is effectively a case study. Experiences of one sector council may not have been faced by others, and rarely by all sector councils, making generalizations difficult. Similar to the formative evaluation, a quantitative counting of problems or benefits across the sector councils is not as important as learning from the results for each council. The methodologies, to the extent possible, were designed to collect a common core of information from each of the sector councils. In addition, data were collected on unique aspects of each sector council. Key Findings and Conclusions The rationale for SPI has not diminished since the formative evaluation and, in the face of devolution of training to the provinces, SPI may have become even more relevant by providing human resource-related programs and services that are national in scope. In recent research examining the rationale for SPI, there is support for the HR approaches sponsored by this Initiative as a way to share costs of training-related investments and to capitalize on the expertise of the various partners. Past and recent public opinion surveys have confirmed that public support for lifelong learning is extremely high and there is strong public support of the partnership approach. The SPI approach, which attempts to depoliticize training and focuses on joint solutions, resonates very well with the general public. Employers and employees surveyed at the formative evaluation stage also confirmed the sector council concept is a valid approach. However, it is often difficult to translate verbal support into financial support.
Structure and Organization Similar to their products and services, there is a great deal of diversity in the design and delivery of sector councils. Despite this diversity, there are some common elements of success, based on the evidence collected to date. In fact, thus far, the structure or organizational elements of a council have been the best predictor of its future survival. The key elements of success include:
Agreements There are few statements on intended impacts and effects in sectoral agreements and there are limited provisions in the current sectoral agreements for ongoing data collection and evaluation. The absence of any financial provisions has made it difficult for sector councils to devote resources to these activities, even if the results could ultimately provide information that could improve the effectiveness of their delivery results and outcomes. Strengthening sectoral agreement provisions on reporting requirements for accountability could help to clarify result commitments and strategies to measure performance (i.e., data collection and evaluation/review) regarding council contributions to overall SPI strategic objectives as well as for program delivery (outputs). Not all partnerships will yield lasting results. Agreements must ensure that provisions are incorporated to ensure maximum use is obtained from the products and services developed by the sector council. The sector councils must also prepare for the potential dissolution of project level partnerships and ensure that provisions are made in the agreements for ownership of products, client lists, etc.. Ownership is emerging as a crucial issue for several councils as joint development projects are undertaken with third parties and products, such as curriculum and supporting materials, are provided to participants. Protecting ownership of intellectual property is a difficult and complex issue, but may be crucial to the survival of a sector council. On the other hand, the collection of royalties and efforts to protect copyright could actually conflict with the broader SPI objectives of increasing learning opportunities. Reinforcing sectoral agreement provisions to deal with the issue of ownership in the context of regular functioning or partnership dissolution could help to clarify matters in this regard. Products and Services There is a great deal of diversity in the products and services offered by the sector councils, reflecting different industry needs and challenges. A crude description of sector council products and services may be based on their focus on standards and/or training. Most sector councils are involved in the development of occupational standards. Examples of products and services include: development of occupational standards, training curriculum design, certification/accreditation, funding training participation/training leveraging, and human resource research. The SPI is a large scale experiment in HR approaches which provides the opportunity to examine the strengths and weaknesses of various strategies. One potential threat to fully benefiting from this diversity is duplication of efforts across the sector councils. To avoid duplication and to strengthen the efforts of all sector councils, the activities and experiences of the sector councils need to be communicated. This communications need may have exceeded the capability of the informal communications network and may require more formal mechanisms. Awareness, Take-Up and Satisfaction Awareness and take-up of sector council programs and services is still relatively low. This observation is compounded by the fact that some of this training was not incremental. While some of this problem is simply due to the recent entry of the products into the marketplace, marketing is still a significant weakness. Several councils have recognized this weakness and staffed marketing positions or contracted out. In too many circumstances, however, product testing and marketing occurred after full production and entry into the market with products and services. There are numerous barriers to implementing occupational standards and training and each small impediment can suppress participation. In general, a lack of flexibility in the delivery of the programs and services can severely restrict take-up rates. For example, the requirement to take courses in fixed time periods can conflict with production scheduling. Another related consideration is the role of unions in the delivery of programs and services. While union involvement may be essential at the design phase, for some sectors it may be equally essential to have a non-union labour interface as an option for the delivery of programs and services. Overall, there were high levels of satisfaction among stakeholders and participants with the programs and services provided by sector councils. However, despite the high levels of satisfaction, in at least one survey the results indicated that participating employers were equally, if not more satisfied with training from other sources. The fact that employer or employee participants choose sector council training may reflect the unique access provided or the opportunities for standards and certification that may be lacking in the other options. Among the more flexible options, is computer-based technology. This type of training, however, is self-directed and it is important to ensure the nature and content of the materials is clearly understood, including the skill and motivation required by the learner. A common criticism of the training was the quality of the trainers. As train-the-trainer programs become more prevalent, it will become increasingly important to monitor the trainers and provide additional training or new trainers as appropriate. As noted by key informants, quite often the most highly skilled members of the workforce made the worst instructors despite the wealth of their knowledge. Systemic Change and Effective Partnerships The key informant interview and case study evidence collected in this evaluation supports the achievement of systemic change. This change has occurred within the Canadian learning system and the way in which educational institutions respond to the needs of firms and workers. College recognition of workplace-based and sectoral council-based training for credit is one example of systemic change to the learning system. Effective partnerships are a key factor in achieving systemic change. Partnerships bring significant direct and in-kind contributions to sector councils in the form of the time of partners, contributions toward operational expenses, as well as direct contributions to development and training costs. Partnerships have also played a critical role in achieving efficient development and implementation of council programs and services. Joint development projects are not without their problems and the sector councils must carefully consider potential outcomes, costs, benefits and legal implications. Self-Sufficiency Given the evidence to date, the three year timeframe for achieving core operational self-sufficiency appears to be unrealistic for most sectors, unless there is already a strong partnership/working relationship. The process of developing working partnerships is frequently slower than expected, which only compounds the problems of trying to design and bring products to market within three years using a consensus decision making process. Of the seven operational sector councils included in Round One data collection, three had achieved financial self-sufficiency. Self-sufficient councils were still conducting projects consistent with SPI objectives, in fact, they were all still receiving project funding under SPI. Focusing on a single product is a risky venture since the entire council can be at risk if the product fails in the marketplace. Diversity can be difficult to manage, but a modest degree of diversity is likely the safest approach. Simply having successful product and service sales may not be sufficient to ensure self-sufficiency. Some sector councils with relatively modest program take-up levels have achieved core self-sufficiency from membership fees and certificates/renewal fees. There is a need to develop ongoing core support in addition to support from product sales. This type of ongoing core support is particularly challenging for cross-sectoral councils. Although there are compelling reasons to require industry support and self-sufficiency, there can be inherent conflicts to charging fees such as less take-up due to cost. In addition, royalty and copyrights have become important issues as sector councils participate in joint development agreements. Self-sufficiency goals would compel the sector council to attempt to protect its copyright, yet this may conflict with goals for broader take-up by imposing a cost to potential users. Employer Impacts The evidence from administrative data, surveys and project-level case studies suggests that council-sponsored programs had an impact for employers on increasing training and interest in training activities. When it comes to developing a training culture, the process is even more important than the product. Without employer involvement, there can be little expectation that the process will have an impact at the firm level. The involvement may be as little as simply planning or tracking their training activities to full joint labour-management participation in HR decisions. The general consensus of employers was that the training and standards improved the skill levels of their workforce and there tended to be a related increase in productivity. There was some objective evidence of productivity and profitability improvements, but this was on a very limited scale. Other related impacts reported include improved school-to-work transition and client satisfaction. Employee Impacts Council-sponsored HR activities are likely to have little impact on an employee's interest in training since there is virtually a mystique surrounding the training and lifelong learning. The main benefit of sectoral initiatives would be to provide more opportunities for training, removing some of the supply level barriers. Another advantage of sectoral training is the strong link to the workplace and the standards and certification that is frequently available. These opportunities facilitate the transition from intentions to actions. Council-sponsored training was perceived by the majority of workers surveyed to have positive short-term impacts on various job-related skills and many reported using these skills at work. Indeed, many participants surveyed cited increasing or upgrading skills as the main reason for participating in council initiatives. Survey participants tended to be split on the perceived impacts of the training on their job mobility, flexibility, security and earnings. Some participants reported a perceived increase in mobility as a result of participation in training, and many expected the training to have positive impacts on their abilities for job advancement. For other training participants, the effects on such aspects were less clear. Differences in rated impacts may be due to the fact that different conditions prevail from sector to sector and across firms, which influences the reasons why employees participate in training and their expected results. The ability of workers to realize benefits depends, among other things, on the workplace itself. In addition, it is too early to measure the longer-term benefits of occupational training. Efficiency and Cost-Effectiveness The evidence indicated significant efficiencies in HR activities as a result of the sectoral approach. In general, the shift to joint development with educational institutions and private companies has significantly reduced the costs incurred by sector councils to develop their products and services. Some efficiencies have off-setting costs, including revenue sharing, but also additional management and administrative costs. A potential for inefficiencies can occur if there is close overlap or duplication of programs and services offered through other means. Alternatives For some sectors, the alternatives to sectoral initiatives are minimal since the types of training being promoted and designed are for new occupational standards. In some sectors or occupations, there may be few delivery agents because the market niche is too small to develop tailored products and services. In other sectors, however, the role of the councils may have to shift substantially to adjust to the growth in training options. Some of these challenges present opportunities as new market needs and roles for the council become apparent. The extent to which a sector council is a true partnership and a broad representation of the industry, the involvement of educational institutions, private trainers and industry/occupational associations eliminates the distinctions between competing alternatives and the SPI approach. The more comprehensive the partnership approach is, the less relevant the alternatives issues become.
| |||||||
![[Previous Page]](/web/20060126223157im_/http://www11.hrdc-drhc.gc.ca/edd-img/back.gif){kind=small}
![[Table of Contents]](/web/20060126223157im_/http://www11.hrdc-drhc.gc.ca/edd-img/content.gif)
![[Next Page]](/web/20060126223157im_/http://www11.hrdc-drhc.gc.ca/edd-img/next.gif){kind=small}