The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:
" To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system ."
To ensure the excellence of the research CIHR funds, applications to CIHR for funding of research projects are evaluated by peer review. Peer review is carried out by committees of experts (grants committees) which span the entire spectrum of health research from Biochemistry and Molecular Biology at one extreme, to Population and Public Health at the other. The purpose of this document is to inform applicants and members of these grants committees about relevant CIHR policies and procedures.
CIHR is governed by a Council (GC) that has authority over the expenditure of CIHR's funds. The GC establishes programs and policies, and determines the committee structures, memberships, policies, and procedures. Committee members and staff work within the policies established by the GC. (Committee Chairpersons and Scientific Officers do not appoint committee members).
CIHR grants committees are asked by the GC to review applications and make recommendations on their relative merits. The GC decides which applications are to be supported. Inevitably, committee members will form opinions on CIHR policies. CIHR welcomes discussion and encourages committees to set aside time for this purpose at the end of their meetings. However, committees may not take actions that are against CIHR's established policies.
CIHR funds research through both open and strategic competitions. Open competitions are competitions in which proposals in any area of health research are accepted, and are regularly scheduled twice annually. Strategic competitions may be sponsored by CIHR or by one or more of its Institutes. Applications are solicited in specified areas of research only, and are invited through a Request for Applications (RFA) process . Strategic competitions are announced regularly two times per year. In general, the same policies and procedures are followed for both types of competitions, unless otherwise specified in the RFA.
The prime responsibility of CIHR grants committees is to evaluate applications submitted for a particular competition, to rate them so that they may be ranked in order of priority and to recommend the funds needed to support the research if the application is approved.
Recommendations from committees are considered initially by a Standing Committee of the GC, the Standing Committee on Oversight of Grants and Awards Competitions (SCOGAC), which in turn makes recommendations for funding to the GC. The GC very rarely departs from the priority rating recommendations of committees in making its final decisions, and modifies budget recommendations only if budgetary restrictions on CIHR make it necessary. In the case of Institute Strategic Initiatives, which are funded from a strategic budget approved by the GC on the basis of the Institute Strategic Plan, recommendations from SCOGAC go directly to the Scientific Director of the Institute(s) for final decision.
Committees are asked to flag applications that raise special concerns, for example, ethics issues (human or animal), laboratory safety, eligibility of applicant or institution, overlap with other funding, compatibility with CIHR's policies etc. Such concerns, however, must not prevent a committee from evaluating an application and, unless they directly affect the scientific merit, must not influence the rating. These concerns will be taken into account by the GC when making a decision on whether to approve the grant. Issues of ethics will be taken up with the research institution whether or not the project is funded.
All information contained in applications, internal and external reviews and committee discussions is strictly confidential. Committee members must not discuss with applicants or reviewers any information relating to the review of a specific application, or offer opinions on the chances of success or failure. All requests for information on an application or a reviewer report should be referred to staff in the Research Portfolio at CIHR responsible for the committee in question.
By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application is made available to the applicants when they are notified of CIHR's decision. However, the identity of external reviewers and committee reviewers will not be revealed by CIHR to applicants.
CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. No committee member with a conflict of interest may participate in the review of an application. As a result:
must declare a conflict of interest and leave the room when such a proposal is up for review. The Chairperson is responsible for resolving areas of uncertainty.
The Chairperson and Scientific Officer are subject to the same conflict of interest guidelines as regular committee members.
All committee members are asked to sign a form agreeing to abide by the Confidentiality and Conflict of Interest policies when they join the committee. All external reviewers are sent a copy of the policies with the application(s) they are asked to review.
New, renewal and resubmitted applications are normally reviewed at the same committee meeting. They are to be considered on a "level playing field" as the GC expects to apply the same criteria and cutoff for all types of applications. In evaluating the likelihood of success of the proposed research, committee members are reminded to take stage of career and previous funding of the applicant into account and vary the emphasis placed on track record and training experience as appropriate. Resubmissions may contain a two page response to previous reviews of the application, and although this may be helpful, they are still to be rated relative to other applications in the competition and not on how the application may have improved relative to its previous submission.
Some applications may have been submitted as part of a partnership program or Strategic Initiative (e.g., Neuromuscular Research Partnership, Regional Partnership Program, Institute Priority Announcement etc.). These applications are to be reviewed and rated as regular operating grants, except when special instructions are provided. They will either be funded out of a separate envelope or measures will be taken to ensure that funding of these proposals does not influence the funding of other applications to the open grants competition reviewed by the same committee.
2.6.1 Project vs. Programmatic Operating Grant Proposals
Applications may be for support of a single project, or for a program of research including several distinct projects. While programs may be more typical of biomedical applications and projects more typical of health services applications, either format is acceptable for all areas of research. Programmatic grants differ from project grants in that they:
All applications received by the appropriate deadline date are entered into the competition. Material arriving after the deadline date will not be considered unless it relates to the acceptance of previously submitted manuscripts, or the outcome of other applications submitted by the applicant or the appointment status of the applicant.
Within two weeks after the deadline, the Chairpersons and Scientific Officers of the grant committees meet to:
These assignments take into consideration the committee(s) suggested by the applicant and the applicant's suggested reviewer list, but are not bound by them. Chairpersons and Scientific Officers may suggest that extra members be added to their committee for a certain competition or to review one or more specific applications (perhaps by teleconference) if there is not appropriate expertise among the regular members of the committee or if the application pressure is too great. A decision by the Chairperson and Scientific Officer that a committee should review an application is binding on that committee.
CIHR staff then send the applications to external reviewers as required and to the committee reviewers. Internal and external reviewers receive full copies of the applications assigned to them four to six weeks before the committee meeting. Each committee member receives books/CDs containing all the applications (minus appendices) assigned to their committee at least two weeks before the meeting and copies of all the external reviewer reports. External reviewer reports are sent as far ahead of the meeting as possible, though some arrive late and may be available only on the day of the meeting. If external reviews have been solicited, CIHR staff make every effort to ensure that at least one external review is received for each application.
Committee meetings are held usually within four months of the deadline date and last not more than three days. A maximal load for a committee meeting should be not more than 80 applications for a 3-day meeting with no member asked to review more than 12 (and preferably 10) applications (not including the applications for which he/she is asked to be reader).
SCOGAC considers the committee recommendations as soon as possible after the committee meetings and makes funding recommendations to the GC for final decisions. A list of successful applicants is posted on the CIHR website as soon as it is available.
Applicants are informed of the results of the competition in two stages:
Applications which have been flagged for Special Attention (overlap, eligibility, ethics, human pluripotent stem cells research, etc.) are withheld as "pending". The applicant will be contacted if further information is required. The additional information may be discussed with the committee Chairperson and/or reviewers if necessary prior to a final decision regarding funding.
CIHR is currently gradually introducing the use of electronic documents into the review process. In committees/programs where electronic submission and review is underway, applicants will be submitting electronic applications and given access to reviews and decisions electronically via a secure website, "ResearchNet", designed specifically for this purpose. Similarly, internal and external reviewers will be posting their reviews electronically, and committee members will receive access to external reviews as soon as they are available via the same site. To date, the process of review and decision itself as described above has not been changed, although the easy and rapid availability of all documents and potential for communication among applicants and reviewers may make feasible changes that would previously have been difficult if not impossible. Nevertheless, no changes will be adopted without thorough discussion by representatives of the research community, e.g., SMIPR, the Subcommittee on Monitoring and Innovation in Peer Review.
Together, the Chairpersons and Scientific Officers are responsible for ensuring that applications received by CIHR are reviewed by the most appropriate committee for evaluation. On accepting an application for review by their committee, Chairpersons and Scientific Officers accept responsibility for ensuring that the committee performs a fair review.
Chairpersons and Scientific Officers:
The committee Chairperson is directly responsible to the CIHR for ensuring that the committee functions smoothly, effectively and objectively, according to CIHR's policies for the Operating Grants Program and its committees. He/she establishes a positive, constructive, fair-minded environment in which the research proposals are to be evaluated. In addition to the duties shared with the Scientific Officer, the Chairperson:
In addition to the duties shared with the Chairperson, the Scientific Officer:
4.3.1 Scientific Officer Notes
The purpose of Scientific Officer notes is to tell the applicant what the determining factors were, from the point of view of the committee, in the rating of the application, and how the budget was determined. The notes provide the applicant with insights into the committee discussion of the application, which are not available though the comments of the reviewers.
The notes should include:
The notes should not include:
The Scientific Officer should read back the draft summary of discussion for validation and to obtain further possible input from the committee.
CIHR staff:
It is expected that committees will work in a more consistent manner if they follow defined procedures.
5.1 All committee members should be familiar with the objectives of the projects and the applicant's experience in all applications to be assessed by their committee as well as with the external reviewer reports. Internal reviewers therefore need only briefly summarize these.
5.2 For committees not reviewing open operating grant applications (MOPs), the assessment of each application begins with both internal reviewers announcing their initial ratings, to one decimal place. All applications rated below 3.0 by both internal reviewers, and for which the external reviews are essentially in agreement, will not be discussed further by the committee unless a committee member requests it (triage). In these cases, the applicant will receive a copy of all internal and external reviews, but there will be no Scientific Officer notes. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers.
For committees reviewing open operating grant applications (MOPs), a new procedure has been introduced to streamline discussion of applications, effective with the Fall 2006 competition. Prior to the committee meeting, reviewers are asked on ResearchNet to identify the applications they reviewed that they feel are the most competitive; i.e., that not only deserve to be funded but that have a good probability of being funded. As a guideline, reviewers should aim at placing approximately 50% of the applications they reviewed in this category, although this may be higher or lower depending on the overall quality of the pool of applications they reviewed. At the committee meeting, the assessment of each application begins with both internal reviewers announcing their initial ratings, to one decimal place. If neither reviewer has placed the application in his/her top group of competitive applications, and if the average of the internal reviewers' ratings is <3.50, then the committee is given the option of not discussing the application. If any committee member objects, the application must be discussed. If an application is not discussed, the applicant will receive a copy of all internal and external reviews, but there will be no Scientific Officer notes. Committee members do not vote on the rating; it is calculated as the mean of the initial ratings of the two internal reviewers. The expectation is that this process will result in up to 30% of the proposals not being discussed, thereby streamlining the committee meetings and allowing committee members to devote more time to discuss competitive applications in the fundable range.
5.3 For applications that pass this first stage, discussion proceeds with the first internal reviewer presenting his or her assessment of the application, describing the strengths and weaknesses of the proposal (see Factors in the Assessment, below). The second reviewer follows, concentrating on points of agreement or disagreement with the first, and elaborating points that may not have been addressed by the first reviewer. One of the internal reviewers or the Scientific Officer then addresses the comments of the external reviewers, pointing out areas of agreement and disagreement with the internal reviewers. The reader may wish to comment further on issues that have already been raised, or may wish to raise additional issues, as appropriate.
5.4 The Chairperson leads discussion of the proposal by all committee members. In addition to points raised by the internal reviewers, views expressed in the external reviewer reports must also be taken into account; the Scientific Officer introduces points not adequately addressed by the two internal reviewers.
5.5 The Chairperson briefly summarizes the discussion.
5.6 Following the discussion summary, the Chairperson seeks a "consensus rating" from the two internal reviewers (the internal reviewers may revise their initial ratings to reach a consensus rating, if they see fit). If a consensus cannot be reached, the mean value of the ratings of the two internal reviewers is used (round up, if necessary, to obtain a single decimal point). All committee members, including the two internal reviewers but excluding the Chairperson and Scientific Officer, then cast individual confidential votes within +/-0.5 of the consensus rating (the internal reviewers are not bound to the consensus rating, i.e., they can also vote +/- 0.5 from the consensus rating). The rating assigned to the proposal is the average of these confidential votes. Regardless of what the final consensus rating is, all committee members vote since they have been privy to the discussion. However, if the final consensus rating is <3.5 (i.e., not in the fundable range), the budget discussion should be brief. (See 5.7 below).
To ensure consistency, committees must adhere to a common scale. It is particularly important that committees use the full scale and apply the same convention in assigning ratings. To facilitate this, the following scale and descriptors should be utilized:
|
Descriptor* |
Range** |
Outcome |
|
outstanding |
4.5 - 4.9 |
May Be Funded - |
|
excellent |
4.0 - 4.4 | |
|
very good |
3.5 - 3.9 | |
|
acceptable, but low priority |
3.0 - 3.4 |
Not Fundable - May or |
|
needs revision |
2.5 - 2.9 | |
|
needs major revision |
2.0 - 2.4 | |
|
seriously flawed |
1.0 - 1.9 | |
|
rejected |
0.0 - 0.9 |
* Only applications rated 3.5 or higher are eligible for CIHR funding. The range 3.0 to 3.4 should be used for applications which, while technically and conceptually acceptable, are not considered to be a high priority for CIHR funding, perhaps because the topic is not considered relevant to an important health issue, or because the work proposed seems unlikely to yield major advances in knowledge, or because the approach is not particularly innovative. Please note that applications rated 3.0 to 3.4 are not eligible for CIHR funds, including those from partnership programs, and may not be discussed by the committee; however, applicants are encouraged to re-apply after addressing the reviews. Applications rated below 3.0 are so flawed in some respect that they do not represent a good investment of public funds, and would require significant rewriting to be considered acceptable. Such applications will normally be triaged, and not be discussed by the committee.
** In the committee meetings, reviewers assign scores to one decimal place, but the final average rating is calculated to two decimal places. When two decimal places are used, the ranges become 4.50 - 4.99, 4.00 - 4.49, etc.
5.7 A budget is recommended (see Section 8).
5.8 The term of support for the grant is determined (see Section 9).
5.9 Special concerns of eligibility, ethics, overlapping sources of funds, human pluripotent stem cell research or other points are discussed and, if necessary, flagged for CIHR staff to address.
5.10 Once all applications have been reviewed, if the committee feels that any application(s) has been treated inconsistently, re-review of one or a small number of applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before. The committee does not review the overall rankings of all applications at the end of the meeting as individuals with conflicts of interest would inevitably be present during such a process.
5.11 An essential component of any committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. Committee members will therefore remain for the entire meeting and make travel arrangements accordingly. Staff and the committee Chairperson will provide advance guidance on the probable length of the meeting.
CIHR's objective is to support research which will lead, directly or indirectly to the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products, and a strengthened health care system. Evidence of originality, clear thinking and the potential for high quality publications and/or high impact are sought in preference to elaborations on existing knowledge, lack of focus or large numbers of not very important papers.
6.1.1 The individual review should briefly outline the aim(s) and description of the project - purpose of the proposal, the hypothesis to be tested or question to be addressed, the objectives to be achieved, the approach proposed, the progress made to date.
6.1.2 This is followed by an assessment of the application (see Section 6.2). It should be clear and concise, using objective and non-inflammatory language. The reviewer should state his/her opinions, with justification. The review should also include comment on the external reviews, pointing out areas of agreement and disagreement. (This might have to be done very shortly before the meeting since not all external reviewer reports will be available at the time the review is initially written). Since the applicant receives all reviewer reports, it is not necessary to repeat the statements of the external reviewers, but if the committee member disagrees with them, reasons for doing so should be clearly stated.
6.1.3 The review should contain an assessment of the strengths and weaknesses, the originality and potential impact of the research described in the application, the appropriateness of the research plan, the past productivity of the applicants and the suitability of the research environment. The most useful reviews are those which provide constructive advice to the applicant, enabling him/her to improve the quality and efficiency of the proposed research.
6.1.4 The review should include comments on the budget requested and a formal recommendation which is consistent with the scientific recommendations. If budget cuts are recommended, there should be clear and detailed reasons for the cuts.
6.1.5 If necessary, the reviewer may comment on such factors as the ethical acceptability of the research ( e.g., adequacy of consent forms, appropriateness of techniques) , possible overlaps of the application with other sources of funding (as described in the CV Module), possible issues of eligibility or any other issues but these concerns should not influence the rating or budget recommendations, unless they bear on the scientific merit of the application. It should be noted that CIHR does not require ethics forms per se as part of the application. Responsibility for ensuring that all research meets ethical standards has been delegated to the local institution. However, CIHR may wish to ensure that certain details have been considered by the local Research Ethics Board (REB) or Animal Care Committee (ACC) and may withhold funding until it is certain that such issues have been addressed.
The following questions are among those which should be considered. It is understood that reviewers and committees may weigh these questions differently from one application to another.
The Applicant's Productivity, Experience and Training
The Research Proposed
The Research Environment
In addition, requests for " programmatic support " should be judged by criteria such as:
The rating given to an application reflects the evaluation of many factors, as described under section 6.2. CIHR does not provide rigid rules for how they should be weighted, since no two applications are identical.
On occasion, if an application contains sections of varying quality and part(s) is regarded as particularly compelling, a committee may prefer to base its rating and budget on the parts regarded more favourably and reduce the term accordingly. It may also happen that a proposal is particularly exciting but feasibility of the entire program depends on a positive outcome to an initial experiment In such a case, the committee may recommend a shorter term grant to enable the applicant to obtain "pilot" data that can then be used to support a subsequent follow-up proposal. If funded, the applicant will be told of this, but will not be instructed to work only on the aspects preferred by the committee, unless questions of ethics are involved.
In deciding on the rating, committee members should use the full scale. The rating should not be influenced by such factors as overlapping funds, eligibility or issues of ethics which do not have scientific import. These should be discussed after the application has been rated and the budget determined; if the committee agrees, the concerns should be flagged for attention by the CIHR secretariat.
CIHR's objective is to provide the funds needed to allow the research approved to be carried out effectively. The appropriate budget is very much a matter for judgement by the committee. Some areas of research are more expensive than others. Committees may recommend funds for only the part of the proposed research which they regard more highly than others. Committees may be of the opinion that a highly rated researcher can effectively use a large grant to increase productivity without sacrificing quality; conversely, the progress made by an applicant may be judged of high quality, but less than expected from previous funding, and the committee may recommend a more modest budget.
Discussion on budget may be influenced not only by the perceived costs of the work proposed, but also by the judgements of the rate at which the committee feels that the applicant should be able to progress. The latter may correlate with the rating value; thus an application rated as merely good is not likely to be given the same budgetary latitude as one regarded as excellent.
Committees should use a zero base approach to determine the funds required for the research thought worthy of support. A percentage change from current funding is not an appropriate rationale for a budget. Committees should not make their budget recommendations in the context of perceptions of CIHR's budget; the GC itself will modify the amount for each grant if it feels it is necessary.
For equipment requests built into operating grant applications, recommended budgets should be zero-based and justified item by item. Recommendation of only a global amount without itemized justification is not acceptable.
Consideration of the budget should follow this sequence:
Research staff (research associates, research assistants, technicians, etc.) should be determined by the actual needs for the techniques and work required for the research, as well as by the speed at which the committee wishes the work to proceed. The salary scales put forward by the institution should be followed, especially if the positions of people already employed are to be continued. A starting salary should not be substituted for the higher salary of a named incumbent with a record of continued employment with the applicant. Applicants (Principal Investigators or Co-Investigators) may not be employed on the grant or on any other CIHR grant, except in the case of Research Associates and trainees, when the salary or stipend should be addressed in the budget justification.
Graduate students may be hired as research personnel on a grant. In general this is on a part-time basis, i.e., hourly. This situation is to be distinguished from a graduate student receiving a stipend from a grant (see below), in which case the work done is part of the training of the student and constitutes the thesis or comparable academic requirement.
It is CIHR's policy that awarding of trainee positions should be on the basis of the quality of the training environment. The committee's judgement of the quality of training will be influenced by such factors as the rating given the project, the nature of the research program and the project(s) on which the trainee(s) might work, the track record of the applicant in training young researchers, and the environment within which the applicant is working. It is helpful, but not essential, that the applicant has identified the project to which the trainee(s) will be assigned.
A budget should be established for materials and supplies, etc., to include all non-personnel requirements for funds. Committees may wish to build these totals by categories (e.g., animals, isotopes, chemicals or other reagents, glassware, computer software packages, payments to subjects, access to databases, data analysis, printing, publication costs, travel for research personnel or research subjects). Alternatively, committees may wish to make blanket assessments of the usual operating costs per research worker (which will vary according to the type of work being done), though special costs (e.g., travel for collaboration, or unusual animal care or maintenance costs) should not be forgotten.
In general, CIHR grant funds may be used to cover the direct costs of research only and may not be used for indirect costs, e.g., library, heat and light, office furniture, regular telephone rates etc. (For further details, see document entitled "Use of Grant Funds" or a member of CIHR staff.) Some applications request an amount for what may seem to be indirect costs or "overheads". These need to be justified in terms of their direct contribution to the research, e.g., a contribution to salary of glassware washer or technician to operate a common piece of research equipment is allowable. A departmental "tax" to cover other than research expenses, e.g., for library acquisitions, graduate student stipends, secretarial pool etc. is not allowable. The latter should normally be covered by the institution.
The sum of personnel, trainees and materials and supplies gives a total annual operating budget recommendation. For some types of research, e.g., biomedical laboratory research, this sum is usually rolled over from year to year. Some projects, e.g., clinical trials, epidemiological studies etc. require differing amounts of support in different years. In these cases, the recommended amounts, by year, should be specified.
Equipment costs may be requested for identified items required to do the research. Funding for equipment should be requested in the first year (regardless of what year it is required) and will be awarded as a one time payment during the first year of the grant (i.e., it is not part of the operating base). Equipment requested through an operating grant will have the same rating as the grant.
CIHR encourages individual applicants, as much as possible, to request equipment in the context of an operating grant rather than as a stand alone equipment application. Therefore, the committees should make every effort to evaluate the request when reviewing the operating grant and not recommend a separate application for equipment.
Cost quotations must be provided for equipment or service contracts greater than $10,000. Two competitive quotes as well as letters from an appropriate institutional official documenting the availability and status of similar equipment are required for items costing more than $25,000. In some cases, i.e., when the equipment is to be used by three or more investigators, a multi-user equipment application may be more appropriate.
The committee may receive proposals requesting equipment and/or maintenance alone. These should be rated according to the quality of science the equipment is to support and the necessity for the equipment. Reviews should include comments on the proposed uses and need for the equipment and/or maintenance, the appropriateness of the equipment and/or maintenance requested, and the availability of similar equipment or services to the applicant's laboratory. Cost quotations are required as described above.
Committees are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., they should not reduce recommended budgets to take into account potential overlap. However, if there is evidence of overlap with other actual or potential sources of funds, the application should be flagged by the committee, with comments made on the overlapping sources, and the extent of the overlap. This information will be followed up by CIHR staff, with further consultation with committee members as required and adjustments made as appropriate. The CV module requests a summary and description of overlap for any grants held or applied for by the applicant.
Grants are usually for 2, 3 or 5 years. The length of support recommended by the committee will reflect their view of the time needed to make significant progress, and the expectation that the applicant(s) will be productive. Committees should recommend a term of support sufficient to allow the applicant(s) time to show progress. As a general guide, 5-year grants are for highly rated research programs which are judged to be stable and consistently productive. Two or 3-year grants are for projects which are less well rated or for which the committee has some concerns and believes that a fairly rapid re-evaluation is justified.
Committees should consider that a new investigator, starting a new program, will probably not have time to show progress unless a 3-year initial grant is given. CIHR therefore encourages committees to recommend 3-year initial grants for new investigators, and even 5-year new grants in the case of outstanding young investigators with already impressive track records.
Some proposals may be for support of discrete, time-limited projects in which case the term should match the recommended project duration.
Committees are often reluctant to rate highly an application which is truly novel, in the absence of preliminary data. In this situation, committees are encouraged to support such innovative applications with a high rating, but mitigate the risk by recommending funding for only 1 or 2 years, or provide limited funding which will allow the applicant to obtain preliminary data in support of a more substantive future application.
Revised: October 2006
