Hallux Consulting Inc. (Hallux) was engaged by Industry Canada (IC) to conduct a formative evaluation of Student Connections (SC). The evaluation was undertaken in order to:

Evaluate the program’s design and implementation;
Identify opportunities for improving the effectiveness or efficiency of program delivery; and,
Assess the adequacy of performance measurement systems.

The key steps in the evaluation were:

Reviewing program documentation provided by IC and summarizing it in relation to the evaluation issues; and,
Key informant interviews with IC program management, the staff of the Association of Canadian Community Colleges (ACCC) — the third party delivery agent for SC, and with Administrative Centre (AC) Coordinators at the colleges and universities contracted by the ACCC to deliver SC.

The findings of this study are presented under the three evaluation issues:

Program design and implementation
Continuous improvement
Performance measurement systems

Evaluation Issue No. 1 — Program Design and Implementation

SC’s program objectives require a design that balances reaching as many Canadian small and medium enterprises (SME) and seniors as possible to deliver information and communications technologies (ICT) training while providing meaningful, short-term work experiences to as many youth as possible. Under its current design SC has a national scope, with a presence in every province and most major cities. It trains over 15,000 clients and employs 300 youth each year.
The program’s design is viewed as offering numerous benefits, and generates a great deal of enthusiasm and support among all involved with its delivery. SC is also valued by the SME (and seniors) that take SC training and by the youth who provide the training as Student Business Advisors (SBA). Semi-annual surveys done of SME, seniors and SBA by an independent company consistently show high levels of positive responses to questions concerning the program’s value and benefits.
SC remains an unique program in most of the regions in which it is delivered. Its focus on technology training for SMEs and seniors distinguishes it from other initiatives aimed at the general population. SC’s use of youth trainers is also distinctive, as is its use of them in a capacity that is directly related to their educational focus.
All AC stated that they would provide training services in the minority official language if requested Three of them experience sufficient demand for such training that they offer a fully bilingual program.
SC’s governance and accountability structure has a variety of mechanisms that support decision-making for ongoing planning, management and resolution of issues. The structure combines mechanisms such as mandatory reporting and a formal committee structure with a close working relationship among the three parties responsible for delivery of SC (IC, the ACCC, and the individual AC). These allow for timely exchanges of information and views concerning the program, its design and evolution.
SC is satisfying IC’s accountabilities to the approved program terms and conditions and of the Youth Employment Strategy (YES), which is the source of funding for SC. The extent to which SC is meeting its objectives and generating its results are systematically monitored by surveys. These have consistently shown that it is.

Evaluation Issue No. 2 — Continuous Improvement

SC’s location within its host post-secondary institution has a significant influence on the program’s success, as does the presence of either a full-time coordinator or a coordination team that has the support of the institution’s senior management. Emphasis is therefore placed on the positioning of ACs within their host institutions, and on trying to ensure that those appointed as AC Coordinators possess the appropriate combination of personal and professional qualities.
Starting in 2003-2004, SC has been delivered through a contribution agreement with the ACCC. This change was accompanied by several measures designed to improve the accountability, results measurement and transparency of the program. Since then, the ACCC has implemented other measures to support new ACs that have joined SC.
Notwithstanding these improvements, significant challenges have arisen since 2004. These challenges, which were identified as risks in the 2004 evaluation of SC, stem from changes in the program’s operating parameters brought about by the contribution agreement and by certain Youth Employment Strategy restrictions. They are threatening SC’s ability to maintain its national scope.
Funding is widely regarded as inadequate. It also threatens the program’s ability to maintain its national scope.
SC has a very limited marketing budget. Its experience has been that national promotions are ineffective. Consequently it focuses on networked and grassroots approaches assisted by ACCC and IC. These are labour-intensive, but they work, particularly in established centres.
There are mixed views on the value of the ACCC’s contribution to the delivery of the program. While new centres and program management at IC endorse the ACCC, its added value is questioned by a significant number of established AC.
SC’s operating parameters and funding level are not its only challenges. The persistent issue with the program’s name being confusing for some clients, the difficulties than can arise if the AC is not positioned appropriately within its host institution, turnover of AC, the short-term nature of the program’s contracts, and a perceived unwillingness to share impede delivery of the program.
Outdated products are another risk that threatens the program’s ability to maintain its national scope. The ACCC has therefore developed a product renewal strategy to address this issue. The strategy is based on distributed product development among AC, coupled with contractual and/or in-kind contributions for development and quality assurance.
The ACCC’s product renewal strategy is unproven.

Evaluation Issue No. 3 — Performance Measurement Systems

SC has a robust set of measurement systems that provide a rich data set for monitoring the program’s performance. The ability of AC to meet quantitative performance targets and benchmarks is monitored on a monthly, quarterly and annual basis. Failure to meet targets quickly becomes evident, and when it does the ACCC intervenes directly to provide advice and assistance.
In addition to the scheduled monitoring, the ACCC has instituted rotating audits of the AC that review procedures, data quality and compliance with government and program policies. The ACCC would like to supplement the audits with spot checks and formal lessons learned analyses, but lacks the funding to do so.
The data quality assurance processes used by the ACCC and IC are complemented by local procedures.
SC periodically refines its performance measurement processes to ensure that they continue to generate useful information concerning the program.
The 2004 SC Risk-based Audit Framework (RBAF) identified four key risks to SC. Measures to mitigate these risks are in place, but they are not all effective. In addition, a new risk, stemming from the changes in SC’s operating parameters, has since arisen for which there may be no effective mitigation.

Conclusions and Recommendations

SC is an unique initiative that consistently draws very positive responses from both its participants and its target beneficiaries as well as meeting its national targets and benchmarks. Because it is delivered by IC but funded by the YES, it must balance delivering practical, relevant ICT training to SME and seniors with providing youth with meaningful work experiences that enhance their employability. Recent changes to the program’s operating parameters, combined with the lack of an increase in funding since 1999 and outdated training products and services are compromising the program’s ability to maintain that balance. They also pose a risk to the program’s ability to maintain its national scope. SC needs to take action to address to these issues. It should:

develop a contingency plan for product renewal in the event that the ACCC’s current strategy fails;
explore options for securing additional funding and getting greater flexibility in the terms and conditions governing youth employment in the program;
examine its reporting requirements to determine if the reporting burden on AC, which increased when SC began to be delivered through a contribution agreement, can be reduced;
explore options for easing the pressures new AC face when they join the program and must establish themselves in local markets for SC products and services.

Final Report (PDF - 266 KB - 36 pages)
Management Response (PDF - 40 KB - 3 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.