Evaluation of the Contributions Program for
Non-Profit Consumer and Voluntary Organizations

March 30, 2001

Summary

• The purpose of this evaluation is to assess the effectiveness of the Contributions Program and to identify areas for improvement. The objective of the program is to strengthen the consumer's role in the marketplace by providing research project contributions and development project contributions to consumer and voluntary organizations.



• The methodologies used for this evaluation included a program documentation review, key informant interviews (n=20) and 11 case studies.

Evaluation Findings

Relevance

• Overall, respondents agreed that the program is relevant and is consistent with the Department's priorities. The program is needed to support good research on topics of interest to policy making and to increase consumer groups' research capacity, as well as to maintain the existence of the groups that depend on this program for their survival.

Service Delivery

• The program is successful in reaching existing Canadian consumer organizations and is making sufficient efforts to promote itself with other voluntary groups.



• Organizations are generally satisfied with the proposal application and selection process, which are perceived to be fair and efficient. An issue lies in the selection of priority research areas for the program, for which consumer organizations are not being consulted on a formal basis.



• Respondents are unanimous in their praise of the support provided by the Office of Consumer Affairs staff, both at the proposal preparation stage and during the project.



• The program is monitored by means of mandatory quarterly reports submitted by each group, as well as by conducting a post-project methodology assessment of a sample of projects, with the assistance of an external expert. The majority of consumer organizations find the monitoring mechanisms to be appropriate.

Impacts

• According to case study results, research projects are deemed to have informed and sensitized government to the particular issues being studied. In half of these cases the projects have influenced or are expected to influence government policy. Similarly, the majority of projects informed and sensitized industry members and half of these have actually provoked changes in industry practices. The consumer organizations themselves also benefit from increased knowledge, competency on specific issues, and new opportunities.



• According to the External Assessor of Reports, most projects are relevant to the program. Many projects appear to suffer from a poor presentation of the results (at least those reported to the Department shortly after project completion B other communication products may have been produced after the review of the External Assessor). Communications appears to be a key improvement area. Many (between one-quarter and one-third of the projects) have also experienced difficulties in attaining the expected results.



• Stakeholder and government respondents also generally support the view that the program has a positive impact on consumer organizations capacity to produce quality research and to provide useful input into public policy. Impacts on consumers, however, are difficult to measure or remain expected.



• Of the three development projects evaluated, two demonstrated clear results in terms of additional revenues generated, which is a very positive impact for the program.

Conclusions and Recommendations

• According to evaluation findings, the strengths of the program include its relevance, proposal review process, external assessment process, and its capacity to meet tangible needs on the part of government, industry, consumer organizations and consumers. Findings also suggest a number of remaining challenges for the program, including the on-going effort to increase the quality of funded research and the communication of results.

The following recommendations are proposed:

• Consultations should be conducted with consumer organizations before the annual priorities are set for the funded projects.



• Consumer organizations should be consulted on any review (e.g., electronic automation) of the proposal process to ensure that it meets their needs.



• The challenge surrounding communication of project results may be addressed by modifying the program's requirements to include a more detailed communication plan, and by ensuring that adequate funds are provided to support these communication activities.



• The program should encourage the use of seasoned researchers and provide sufficient funds to enable consumer organizations to purchase the adequate expertise to conduct their research.



• The program should explore alternative ways of helping organizations become more self-sufficient, such as by conducting a review of best practices in the area.

Reactions of Program Management

The management of the Program is in general agreement with the evaluation findings, and will act upon the first three recommendations made in time for next year's program, in 2002-03. In the coming year, the Office of Consumer Affairs will also explore means to improve organizations' capacity to reach self-sufficiency. Recommended changes are, for the most part, administrative and will not require modifications to the present program Terms and Conditions.

Adobe Acrobat Version (PDF - 180KB - 47 pages)

Note: to read the PDF version, you need Adobe Acrobat Reader on your system. If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page. If the accessibility of PDF is a concern, you can have the file converted to HTML or ASCII text by using one of the access services provide by Adobe.