19.8 Regional development is a major concern at the various levels of government in Quebec. Over the past 15 years, federal responsibility for regional development has been gradually decentralized to respond more effectively to regional needs and to foster greater involvement by the communities affected. Establishment of the Federal Office of Regional Development - Quebec (FORD-Q) as a separate department in June 1991 was a result of this decentralization.
19.9 Although FORD-Q was created in 1991, its activities until 31 March 1995 were based primarily on a strategy developed in 1988. They were essentially a continuation of the activities implemented at that time and were heavily impregnated with the objectives, practices and operating methods of Industry, Science and Technology Canada (now Industry Canada), which was formerly responsible for regional development in Quebec.
19.11 This mandate is broad, complex and ambitious. To attain the objectives the mandate entails, FORD-Q was given responsibilities for defining the overall direction of regional development policy in Quebec, for negotiating the necessary development agreements with the Quebec government in partnership with the federal departments involved, for recommending the allocation of funds from the federal regional development reserve (eliminated in the 1995 federal Budget) and for ensuring the implementation of the development initiatives that are within its scope.
19.12 With an average annual budget of $200 million, FORD-Q carries relatively limited weight in comparison to all the other instruments of intervention established by the federal government for equitable distribution of the national wealth.
19.14 All of these programs use the same mechanism: financial assistance in the form of contributions to commercial and non-commercial undertakings. The funding is intended to be used to acquire capital assets or to conduct studies such as feasibility or marketing studies; in some cases, the contributions finance the operating costs of non-commercial undertakings. The amount of assistance to a project varies depending on the program and the region of Quebec.
19.15 Since February 1994, FORD-Q has been managing the federal share of the Canada-Quebec Infrastructure Works Agreement; this agreement was excluded from the scope of our audit.
19.16 Responsibility for the Canada-Quebec Subsidiary Agreements on Industrial Development, which supported large businesses' major projects, was transferred to Industry Canada in the fall of 1994. Our Office audited the agreements and the findings are reported in Chapter 21 on Industry Canada. However, the data presented in the following paragraph and in Exhibits 19.3 and Exhibit 19.4 encompass all the projects approved and funds committed under those agreements up until the time of their transfer.
19.17 Overall, the Department has committed $1.15 billion to 4,139 projects since 1988. We found that:
19.19 Between 1988 and 1995, the Department had on average 250 employees and an average annual budget of $200 million, of which $170 million was allocated to grants and contributions. However, because of cuts announced in the February 1995 federal Budget, the Department will have to cope with significantly reduced budgets and a smaller staff in the future.
19.21 In late 1993, the Department initiated an in-depth review of its orientation and its future activities. This exercise resulted in a major reorientation of its strategy and the development of a new program direction, which came into effect on 1 April 1995. This topic is covered in greater detail at the end of the chapter.
19.24 In the context of the Department's new orientation, we also sought to determine what lessons the Department could draw from its past experience.
19.25 The quantitative information in this chapter was obtained from the various government sources cited. However, unless otherwise indicated, we assessed the reasonableness of this information but did not audit it.
19.27 The Department set objectives for each of the programs that were in effect from 1 April 1988 to 31 March 1995. (See examples in Exhibit 19.2. ) Although the objectives were all valid, they were not sufficiently precise to enable the Department to guide its activities, make optimal use of its resources and evaluate its results.
19.28 The Department did not set objectives at a more operational level than the program objectives. Precise, results-oriented operational objectives would have helped establish a clear orientation, ensure the best choice of projects to be funded and facilitate measurement of results.
19.30 An examination of the funds committed shows that the programs responded to a range of applications and types of applicants (see Exhibit 19.4) , and supported projects in a large number of sectors throughout the province.
19.31 Such a large number of programs raises questions about possible overlap within the Department itself and about the efficiency of program management.
19.32 For example, according to the Department's database, a total of $186 million in funding has been committed in the tourism sector since 1 April 1988. These contributions were approved under 15 different programs. Some of the programs focussed exclusively on tourism development, while others were more general in nature. This generates costs not only for the development and implementation of each program but also for its administration.
19.33 FORD-Q should ensure that its new program framework that came into effect on 1 April 1995:
19.36 The scarcity of data on results may be explained in part by the lack of evaluation frameworks for most of the Department's programs. Although Treasury Board recommends that evaluation frameworks be developed as soon as possible after programs are put in place, FORD-Q developed very few.
19.37 Developing evaluation frameworks very soon after a program is instituted has many advantages. Among other things, it makes it possible to define the objectives and expected results clearly and to identify performance indicators. It also helps in identifying relevant data that must be collected. These data can then be used for program management and evaluation and for improving accountability.
19.38 FORD-Q should develop performance indicators as soon as a program is instituted in order to permit ongoing monitoring of results and to improve program management and accountability.
Department's response: The Department agrees. An evaluation and performance measurement framework is already being drawn up and will become operational over the next few months.
19.40 Despite the limitations and problems involved, the Department needs to evaluate the results of its activities and measure the impact of its assistance programs in order to determine their strengths, weaknesses and efficiency, and to draw lessons from them. This will also enable the Department to provide better information to Parliament and to the public.
19.41 The Department has adopted a structure for managing program evaluations. Management has established selection criteria for significant financial assistance programs. The Department has also developed evaluation plans, but they have not been followed. Although these plans were approved by senior management, most of the evaluations scheduled were not conducted because of a lack of funds and a decision to evaluate only programs whose characteristics were similar to those of the new program framework.
19.42 Since its creation in 1991, FORD-Q has carried out three program evaluations. We examined two of them: one of the Manufacturing Productivity Improvement Program and the other of the Enterprise Development Program, Industrial Component. Both of these programs are among the largest in terms of the number of projects and the funds committed.
19.43 Our review revealed that the evaluations carried out were timely, the methodology was appropriate and the observations were relevant and generally reliable. The evaluations dealt with program relevance, value added and effectiveness, among other matters. However, some important issues were not addressed.
19.44 For example, neither of these two program evaluations linked the attainment of the program's objectives to the attainment of the objectives of the Canada-Quebec agreement under which the programs were carried out, or to the objectives set out in the Department's mandate. The evaluators were therefore unable to determine whether the programs had benefited disadvantaged regions or promoted the creation of sustainable employment and income.
19.45 Furthermore, while both evaluations compiled a list of possible positive and negative impacts of the programs, certain impacts were not measured. In one case, the Department did not measure any negative impacts such as encouragement of non-viable activities, exceeding the limit of borrowing capacity or displacement of competition. We found no explanation for the choice of impacts that were measured by the Department.
19.46 The number of impacts selected for review affects the cost of a program evaluation, and the cost must be in relation to the size of the program. Hence, it is obviously impossible to examine every impact. However, a sufficient number of positive and negative impacts must be examined if the evaluation conclusions are to present a realistic picture.
19.47 FORD-Q should examine a sufficient number of relevant issues and impacts in a program evaluation to obtain adequate information on program performance.
Department's response: The new evaluation and performance measurement framework will respond to this concern.
19.50 In its Part III, FORD-Q informs the reader about the Department's mandate, its primary activities and actions, and the allocation of its financial resources. This information is very useful since it makes it possible to understand the context in which the Department operates. However, this rather descriptive information is incomplete in the sense that it does not enable Parliament to determine the extent to which the Department has attained its objectives.
19.51 Apart from the results of the three evaluations of financial assistance programs we have already mentioned, and the amount of investment generated by certain programs, the activity measurements presented are generally operational statistics, such as amounts spent or committed and numbers of projects approved.
19.52 The results of the two program evaluations we examined were not presented in a completely objective manner. The Department emphasized primarily the positive aspects and did not report the evaluators' negative comments, such as the fact that the programs had limited impact and that a large number of projects would have gone ahead without the Department's assistance.
19.53 With respect to the investment generated by its programs, the information presented by the Department let the readers suppose that the proportion of public funds invested in projects is limited to FORD-Q's contribution, whereas, in many cases, other levels of government have also contributed to the same projects.
19.55 However, parliamentarians, the media and the general public consider that the number of jobs created is a significant statistic. For the last two years, the Department has been providing information on jobs created and maintained. In presenting its 1994-1995 Estimates to the Standing Committee on Industry, the Department stated that since 1988 its activities had made possible the creation and maintenance of 56,000 jobs in Quebec.
19.56 We found that the employment data were incomplete and did not correspond to actual results. Indeed, the data were not always entered in the Department's information system and, when they were, they generally represented the applicants' forecasts at the time of application for financial assistance and had rarely been revised or updated.
19.57 FORD-Q should ensure that information reported to Parliament provides sufficient details about the efficiency of operations and the attainment of legislative objectives and the objectives of the programs for which the funds were appropriated.
Department's response: As the Auditor General mentions, the Department has already made many improvements to its Part III of the Estimates and will continue to do so. Particular attention will be paid to information on efficiency and effectiveness.
19.59 We analyzed the documentation in the project files, held discussions with the Department's representatives and met with some of the recipients of contributions.
19.61 The program eligibility conditions and the administrative guide specify the basic analytical criteria to be used when reviewing applications for financial assistance. These criteria are: incrementality; the potential economic benefits for Quebec and Canada; and the commercial and economic viability of the project and the applicant. Analyzing applications according to each of these basic criteria and deciding to approve financial assistance requires considerable judgment.
19.62 The scope of the analysis should reflect the size of the contribution, the type of applicant and the nature of the project. However, departmental management has not specified the scope of the analysis it expects in relation to the size and complexity of projects. The guidelines available to employees are general and apply to all projects without distinction, regardless of their size or nature.
19.63 We found that applications for financial assistance were not always analyzed against each basic criterion and that, for projects of similar size or type, different employees performed the analysis with varying degrees of thoroughness.
19.64 Extrapolating our findings to the population we can conclude, with a confidence level of 97.5 percent, that one or more criteria may not have been met in at least 22 percent of approved projects with contributions totalling $84 million (14 percent of the value of the target population).
19.65 We noted that 85 percent of these projects involved contributions of less than $200,000. In our view, this finding is especially important, since the Department's new program direction focusses primarily on funding projects of this size.
19.67 Analyzing an application against the criterion of incrementality is undoubtedly difficult. Since the assistance provided is free, any applicant would find this type of financing attractive. The Department is therefore faced with the difficult task of evaluating the effect of the contribution on the location, scope and timetable of the project.
19.68 In two program evaluations conducted by the Department, the evaluators found that about 30 percent of the projects would still have been carried out as planned without financial assistance. Our review determined that in the case of projects submitted by non-commercial undertakings, incrementality was fairly obvious. However, in 27 percent of the projects submitted by commercial undertakings, we noted factors that raised questions about the need for government assistance.
19.69 For example, some contributions were granted although a high proportion of the project costs had been committed before the application was submitted or the Department's decision made. In one case, this proportion was as high as 70 percent at the very outset of the analysis of the application.
19.70 For other projects, review of the documentation shows that the applicant was in a very good financial situation, which normally would have enabled it to obtain private sector financing. We acknowledge that a good financial situation does not necessarily mean that an application for assistance should be rejected, as long as other incentive factors justify the need for assistance.
19.71 For example, one business was granted a $90,000 contribution despite the fact that it had paid nearly $2.5 million in dividends the same year it applied for financial assistance. The only incentive factor documented in the file referred to sharing the financial risk.
19.72 In the case of applications for financial assistance submitted by subsidiaries of large companies, the argument used to justify financial assistance is that if the contribution is not granted, the project will be implemented outside Canada. The funding provided by the Department is generally used to increase the rate of return on the project to the level required by the parent company. In general, the file documentation did not refer to the applicants' target rate of return.
19.74 In the tourism sector, the gross economic effects were normally quantified; in other sectors, they were rarely quantified and most often stated in general terms such as improving competitiveness and increasing exports.
19.75 In addition to identifying the gross economic benefits of a project, the Department must determine whether there are factors that might substantially reduce such benefits.
19.76 In some cases we found documents that included reservations about market potential and existing competition in the applicable market, which raises the possibility of a negative impact. However, we found no evidence of any analysis indicating that the Department had assessed that impact. ( See example in Exhibit 19.5 .)
19.78 Project officers must review the financial forecasts and basic assumptions and assess the probability that they will be realized. To this end, they must analyze the market, the competition and the prior experience of the applicant's owners and managers, and determine the extent to which its financial situation would enable it to carry out the project successfully.
19.79 In 11 projects that we examined, we noted that at the time the application was analyzed there were indications that cast doubt on the applicant's financial capacity or the profitability of the project. Of the firms involved in the 11 projects, five were forced to shut down before final payment of the contribution.
19.81 FORD-Q is rarely the only organization that helps finance this type of undertaking. While it mainly funds construction of buildings or acquisition of equipment, other levels of government generally contribute to operating costs.
19.82 The basic criteria used to analyze applications for financial assistance are much more applicable to commercial than non-commercial undertakings. The Department has not developed guidelines for adapting the analysis of applications to the characteristics of non-commercial undertakings.
19.83 For example, the Department considers that the viability of non-commercial undertakings is based either on the ability of the undertaking to become self-financing within a given period of time or on the receipt of recurring grants from various levels of government.
19.84 We found that the proportion of income anticipated from the private sector tends to be significantly overestimated and that, accordingly, the undertaking's ability to become self-financing within a reasonable period is overestimated. Two examples are provided in Exhibit 19.6 .
19.85 When organizations are highly dependent on government funding for their operations, FORD-Q should investigate their potential for obtaining recurring funding over the medium term. Because of the very limited resources at all levels of government, a major reduction in the funding allocated to some of these organizations is certainly possible.
19.86 Some of the Department's employees have substantial experience in the financing of non-commercial undertakings. The Department could draw on this expertise to identify lessons and specify the basic criteria and factors to be considered when analyzing applications, thereby improving the quality of its funding decisions.
19.87 Study projects. Contributions for conducting studies such as feasibility and marketing studies currently represent 3 percent of funds committed, although studies account for 19 percent of approved projects. However, the funding of proposed studies is likely to become more significant under the new program direction. The average contribution awarded to date for this type of project is $40,000.
19.88 We found that the quality of analysis of these applications varied significantly. The Department needs to establish what it considers a reasonable minimum analysis.
19.89 Although these contributions are smaller than those for capital projects, it is still necessary to analyze the financial and human capacities of the undertaking in order to ensure that the applicant will be able to act on positive conclusions of a study. This will provide the Department with reasonable assurance that the contribution will result in economic benefits.
19.90 Of the nine studies with positive conclusions, we noted that three applicants did not have the financial capacity when the application was analyzed to implement the project that normally would follow the study. So far, the three projects have not been carried out.
19.92 Our discussions with departmental officers revealed that they were aware of progress toward the completion of projects. However, this information was not recorded clearly in the files and was rarely entered in the Department's information system. The economic benefits achieved were never compared or analyzed in relation to the initial objectives, either during project implementation or at the time of the final payment.
19.93 Since FORD-Q does not know the results achieved for each project, its ability to identify the results of programs is reduced, as is its ability to intervene in the short term when programs are under way.
19.94 To improve program delivery and ensure that approved contributions allow its objectives to be attained, the Department should:
Under the new IDEA-SME program, the evaluation and performance measurement framework being drawn up will include, among others, economic impact performance indicators for projects supported by the Department. As well, the management information system is being revamped in order to allow for the gathering of all the information required to measure along these parameters. This new evaluation framework and the revised information system will be up and running by early 1996. The Department will draw up simplified guidelines to support the work of its officers. It will ensure that its expectations regarding the quality and scope of work and documentation regarding key decisions are clearly communicated.
19.96 We found no internal report explaining the discrepancy between the actual processing time and the stated target. In an internal audit report submitted in December 1994, the time needed by an applicant to provide all the information required for the Department to make a decision was estimated at about 30 days. However, the internal auditors were unable to undertake a more detailed analysis because information that would have enabled them to identify all the actions taken and the dates of those actions was not in the files.
19.97 As part of its new program direction, the Department has set an official service standard. In view of the type of activity it anticipates, it currently estimates that 30 days would be a reasonable standard.
19.98 FORD-Q should ensure that it collects sufficient information to be in a position to analyze the discrepancy between actual processing time and the stated standard and to take any necessary action, particularly reviewing the service standard.
Department's response: FORD-Q has already begun to establish service standards for its clientele. Processing time is part of the quality service that we would like to offer our clients. This exercise is also intended to simplify decision-making procedures for projects involving an application for financial assistance. Implementation of the service standards program, combined with the revamping of our information system, will enable us to evaluate performance in terms of processing-time standards and take any necessary corrective measures.
19.100 In April 1990, Treasury Board issued a policy on repayment of financial assistance. The Department subsequently applied this policy, using the exemptions allowed. For example, contributions of $100,000 or less can be exempted from the requirement.
19.101 In October 1994, the Department decided that all contributions above $25,000 granted to commercial undertakings would be repayable. Budgetary constraints and the possibility of retaining the amounts recovered after 1 April 1996 had prompted the Department to review its policy in this regard.
19.102 In view of the potential significance of repayable contributions, it would be in the Department's interest to determine the overall level of risk it is prepared to assume. In the case of repayable contributions, as in the case of loans, this risk can be expressed as an acceptable loss rate or desirable recovery rate.
19.103 We do not expect that FORD-Q should take no risks, since this would be incompatible with a regional development mandate. Rather, our view is that it should determine the recovery rate it considers desirable to provide guidance for decision making while continuing to assist small and medium-sized businesses.
19.104 The Department should determine the recovery rate it considers desirable and guide its funding decisions accordingly.
Department's response: Since April 1995, the Department has implemented new programming that includes a decision to lower the repayment threshold for repayable contributions from $100,000 to $25,000. It has also set objectives for recovery of its contributions.
It should be noted that the Department manages a number of important programs to which the concept of repayable contributions does not apply, namely the Canada-Quebec Infrastructure Agreement, and the Community Futures Program.
19.107 A number of factors will have a definite impact on federal regional development activities and on the pattern of interdepartmental relations. Among them are the termination of the Canada-Quebec Economic and Regional Development Agreement; the elimination in 1995 of the reserve for regional development in Quebec; the significant reduction in the budgets of federal departments and agencies; and the major reorientation of FORD-Q. Aware that all these changes have an impact on its ability to carry out its mandate of promoting economic development, the Department developed a more articulated strategy for federal partnership.
19.108 As of 31 March 1995, five memoranda of understanding on specific sectors of co-operation had been signed and others were being negotiated. For example, one such memorandum signed with the National Research Council covers exchanges of information in the field of innovation. Another, signed with Foreign Affairs and International Trade Canada and Industry Canada, covers the delivery of a financial assistance program for exports in all regions of Quebec.
19.109 In June 1994, the Department also established a multi-tiered, multilateral co-ordination structure to promote discussion and exchange of information at policy and operational levels. A number of task forces involving representatives of some dozen federal departments and agencies have been created to examine various regional issues identified by stakeholders. As of 31 March 1995, the frequency of meetings, the work performed and the progress made varied with the particular committee and issue.
19.110 We acknowledge the Department's efforts to date to establish federal co-ordination mechanisms. It is still too early to determine whether these new mechanisms will promote effective co-operation and consultation. The challenge for the Federal Office of Regional Development - Quebec over the coming years will be to encourage the departments involved to work together toward greater convergence and combination of efforts. FORD-Q has no legal authority over the other federal organizations in Quebec. However, it must seek their participation in formulating a comprehensive approach to regional development.
19.112 Our audit dealt with the process of developing and implementing the new program framework for customer services and activities.
19.114 We found that the Department took into account in a reasonable manner the available information on its previous activities and its internal and external environment. It considered the general political and economic climate, the government's policies and priorities and the needs of small and medium-sized businesses (SME), among other things.
19.115 The resulting product. Under its new program direction, FORD-Q will become a federal service centre for small and medium-sized businesses, geared to the distribution of information, project development, partnership and access to financing along three major lines: (1) innovation, research and development, and design; (2) market development; and (3) entrepreneurship and the business climate.
19.116 This new orientation has resulted in the development of a single program that is intended to be flexible and tailored to the realities and needs of the regions. This new program, IDEA-SME (Innovation - Development of markets - Export trade - Assistance to entrepreneurship), replaces all previous programs.
19.117 On 31 March 1995, uncommitted funds from all financial assistance programs still in effect as of that date were transferred to the new program. Over the next two years, a major share of the available budgets will go to the payment of commitments made under previous programs. The budget available for financial assistance under the IDEA-SME program will be approximately $18 million in 1995-1996.
19.118 This reorientation entails some major changes in the way things are viewed and done. As described in Exhibit 19.7 , these changes affect, among other things, the Department's product, its role, its approach, its management framework and the skills of its staff.
19.122 In view of the amount and complexity of the work that remains to be done, the implementation phase will require a sustained effort by the Department's management and staff to ensure the relevance and quality of services, client satisfaction and the success of the new program direction.
19.123 To ensure successful implementation of the new program, the Department should:
As mentioned by the Auditor General, the audit was completed on March 31, 1995. Since then, a global plan for program implementation has been established with precise deadlines for the accomplishment of work.
Over and above that, FORD-Q has established working groups to prepare proposals to ensure implementation of the IDEA-SME program, focussing on four areas of concern: management systems, management information systems, quality management and programming tools. Each working group has prepared an action plan. These plans are followed up and their progress monitored on a regular and structured basis by a co-ordinating committee, which will report on the results of the working groups to an Operations Committee made up of representatives of each FORD-Q directorate.
19.125 The 1995-1996 fiscal year will be a transitional period crucial to the successful implementation of the new program direction.
19.126 In this context, the Department should draw on the experience of its employees and learn from its previous activities in order to develop a more results-oriented management framework.
19.127 From the outset, the Department will have to set specific objectives, develop performance indicators and implement information systems for collecting data. Regular information on the results and efficiency of the Department's activities are necessary to provide better guidance of actions and direction for employees, to evaluate performance and to improve accountability to Parliament.
19.128 Finally, the Department will have to provide its employees with directives that will ensure more consistent quality of analysis. This analysis must be sufficient to ensure that the projects selected will help the Department achieve its regional development objectives, at a reasonable cost.
Audit Team
Lucie Chagnon
Raoul Cyr
Louise Grand'Maison
Luce Joncas
Annie Prémont
Sylvie Soucy
Tina Swiderski
For information, please contact Micheline Éthier Massicotte, the responsible auditor.