2.1 Governance and Decision Making
2.2 CIHR Funding Opportunities
2.3 ResearchNet
2.4 Recommendations from Committees
2.5 Confidentiality
2.6 Conflict of Interest
2.7 Biases
3. Role of Committee Chair Person
5. Guidelines for Members of Salary Awards Committees
5.1 The External Referee Reports
5.2 Hints about Preparing a Review
6.1 Sequence of Steps for the Review of each Application
6.2 Committee Standards
6.3 Review of Meeting
6.4 Attendance
6.5 Procedure at the End of the Meeting
6.6 After the Committee Meeting
Guidelines for the Assessment of Applications for Salary Support
Sequence of Steps for Review of Each Application
The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:
"To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system".
To ensure the excellence of the research CIHR funds, applications to CIHR for training and salary support are evaluated by peer review committees. Peer review is carried out by committees of experts, which cover all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health). The purpose of this document is to inform members of CIHR Awards Committees about relevant CIHR policies and procedures.
CIHR's Governing Council (GC) agreed with the direction suggested by the International Review Panel of creating a single committee to account for all research-related decision-making within CIHR. The Research and Knowledge Translation Committee (RKTC) has been established to develop, implement, and report on CIHR's Research and knowledge translation strategy, in accordance with the CIHR Act and the overarching strategic directions set out by the Governing Council. This includes approving funding for all research and knowledge translation initiatives.
CIHR funds research through both open and strategic competitions. Open competitions accept proposals in any area of health research. Strategic competitions are sponsored by CIHR or by one or more of its Institutes and applications are solicited in specified areas of health research. CIHR funding opportunities are announced regularly two times per year. In general, the same policies and procedures are followed for both types of competitions, unless otherwise specified in the funding opportunity.
CIHR has introduced the use of electronic documents into the review process for some funding opportunities (e.g. Doctoral Research Awards (DRA)). Applicants submit electronic applications and are given access to reviews and decisions electronically via a secure website, "ResearchNet", designed specifically for this purpose. Internal and external reviewers post their reviews electronically, and therefore committee members receive access to external reviews as soon as they are available.
The prime responsibility of CIHR Awards Committees is to evaluate applications submitted for a particular competition and to rate them so that they may be ranked in order of priority and, if applicable, to recommend funds needed to support the research if the application is approved.
Recommendations from peer review committees (PRCs) are considered initially by the Research and Knowledge Translation Committee's (RKTC) Subcommittee on Programs and Peer Review (SPPR) which in turn makes recommendations for funding to the RKTC. The RKTC approves funding for all research and knowledge translation initiatives and reviews reports on Institute Strategic Initiative funding decisions which are made by the respective Scientific Director
Committees are asked to flag applications with special concerns, for example, ethical issues, laboratory safety, eligibility, overlap with other funding, compatibility with CIHR's policies etc. Such concerns, however, must not prevent a Committee from evaluating an application and, unless they directly affect the scientific merit, must not influence the rating. These concerns will be taken into account when making a decision on whether or not to fund the award.
All documents and information provided for the purpose of peer review by CIHR are subject to the conditions of the Privacy Act and the Access to Information Act. Therefore, they and any discussions thereof, must be treated as strictly confidential and may not be used for any purpose beyond that for which they were originally intended. Committee members / reviewers must not discuss with the applicants or referees any information relating to the review of a specific application, or offer opinions on the chances of success or failure. All enquiries or requests for information on an application or a referee report should be referred to the Research Capacity Development Branch.
By law, applicants have access to their own application files. Therefore, all written material used in evaluating an application will be made available to the applicants when they are notified of CIHR's decision. The identity of external referees and Committee reviewers, however, will not be revealed by CIHR to applicants. However, a list of peer review committee members will be published on the CIHR website 60 days after the RKTC approves funding for a competition cycle and will be included in the CIHR Power of Volunteers booklet.
The maintenance of confidentiality also requires procedural safeguards. All materials related to the review process must be stored in a secure manner to prevent unauthorized access. They must be transmitted using secure carriers and technologies. When documentation is no longer required, it must be destroyed using a secure method such as burning or shredding or returned to CIHR for destruction.
CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that it appears that way. No Committee member with a conflict of interest may participate in the review of an application. Committee members who:
(or proposed supervisor)
must declare a conflict of interest and leave the room when such a proposal is up for review. The Chairperson is responsible for resolving areas of uncertainty. The Chairperson is subject to the same conflict of interest guidelines as regular Committee members.
All committee members must agree to abide by the Confidentiality and Conflict of Interest policies when they join the committee. Depending on the competition, the members are asked to sign a form or complete the task on ResearchNet.
Discrimination or bias in the review based on age, nationality, gender or other factors is unacceptable. Moreover, applicants are encouraged to provide information on any interruption in their career (e.g., sick or parental leave). As a result, it is expected that reviewers will better understand the circumstances of each applicant, thereby enabling them to evaluate the applications fairly and without prejudice.
The Committee Chairperson is directly responsible to CIHR for ensuring that the Committee functions smoothly, effectively and objectively, according to CIHR's policies. He/she establishes a positive, constructive, fair-minded environment in which the applications are to be evaluated. The success of the peer review system is critically dependent upon the willingness and ability of all Committee members to be fair and reasonable; to exercise rigorous scientific judgement; and to understand, and take into account in a balanced way, the particular context of each applicant.
The Chairperson must be knowledgeable of policies affecting the procedures of CIHR Awards Committees and should ensure that:
Note that a delegate is appointed as chair in the event of a conflict of interest.
A CIHR Program Delivery Coordinator, who acts as Recording Secretary, is present for the entire meeting. A senior CIHR staff member may also be present, usually at the beginning and end of the meeting, to discuss any particular concerns regarding CIHR policies or procedure.
CIHR staff:
This section applies to the following programs / Committees:
The primary responsibility of the CIHR Salary Awards Committees is to provide an assessment of each applicant based on three levels of evaluation:
The applications may be reviewed by one to four external referees. Referees receive a complete copy of the application to be reviewed. If the applicant has submitted a grant application simultaneously with the CIHR New Investigator application, only the grant application is distributed to external referees, and the reports given to the Awards Committees are therefore the reviews of the grant proposal only.
Committee members may be asked to review applications where the scope of the science may not fall easily within their own areas of expertise.
Reviewers may consult a colleague (usually from their own institution) regarding a specific application, bearing in mind the confidential nature of the review process.
Committee members will be assigned applications for review as either first or second reviewer. In preparing their reviews, committee members are asked to refer to the Guidelines for the Assessment of Applications for Salary Support. (see Appendix 2). This will help the Committee to arrive at an opinion that is based on objective and well-defined criteria. Reviewers should begin with their evaluation of the application. The evaluation is the part of the review that is most likely to have the greatest impact on the committee; therefore, reviewers should be clear and concise. Reviewers should conclude with a summary of their assessment. This summary should not be longer than 150-200 words and may be in the form of a list.
Reviewers may refer to the assessments made by the external referees, but they should consider that the applicant will have access to the external referee(s') report(s). For this reason reviewers should not repeat the external referee(s') statements. If reviewers disagree with the external referee(s), they must state clearly why this is so. Finally reviewers should prepare a short (60-100 words) explanation of their recommendation, which should be stated explicitly. Reviewers must avoid statements or even words that may be construed as personal, emotional or derogatory. Please note that reviews will be forwarded to the applicant.
Two general considerations:
Each Committee member, in addition to reviewing their assigned applications, is expected to read all applications prior to the Committee meeting, and to be prepared to participate in the consideration of each application.
To ensure consistency, CIHR expects Committees to adhere to a common scale. It is particularly important that Committees use the full scale and apply a consistent convention in assigning ratings.
To facilitate this, the following scale and descriptors should be utilized:
| Descriptor | Range | Outcome |
|---|---|---|
| outstanding | 4.5 - 4.9 | May be Funded - Will be Discussed by the Committee |
| excellent | 4.0 - 4.4 | |
| very good | 3.5 - 3.9 | |
| good* | 3.0 - 3.4 | Not Fundable - May or may not be Discussed by the Committee |
| average | 2.0 - 2.9 | |
| below average | 1.0 - 1.9 | |
| not acceptable | 0 - 0.9 |
* Only applications rated 3.5 or higher are eligible for CIHR funding. The range 3.0 to 3.4 should be used for applications which, while rated as good, are not considered to be a high priority for CIHR funding. Please note that applications rated 3.0 to 3.4 are not eligible for CIHR funds, including those from partnership programs, and may or may not be discussed by the committee.
It is expected that Committees will work in a more consistent manner if they follow defined procedures. The sequence of steps for the review of each application is summarized in Appendix 3, and is outlined in detail below:
If a Committee feels that its standards have not remained constant over the period of the meeting, the Chairperson may re-open discussion of one or a few applications. As each application is reconsidered, any Committee member with conflict of interest will again have to leave the room. Following discussion, a consensus rating is determined by the two internal reviewers and voting proceeds as before.
An essential component of any Committee meeting is the final review of the whole meeting, and a discussion of policy. Committee members should therefore remain for the entire meeting and make travel arrangements accordingly.
Absence of a Committee member from a scheduled meeting could seriously jeopardize the Committee's work. Hence, it is mutually assumed that those who have agreed to serve on a CIHR Committee will attend all of its meetings. If unforeseen circumstances bar attendance on short notice, it is the affected member's responsibility to forward to CIHR the completed assessment forms so that they are available to the Committee at the beginning of the meeting.
After the meeting, CIHR staff calculates the average rating for each application, from all rating sheets completed, except those of the Chairperson. The applications are then ranked in order of priority, based on their rating. Based on the available budget, CIHR then decides the level (rating) above which applications are to be funded.
Guidelines for the Assessment of Applications for Salary Support
Applications are to be assessed in three general areas, described below, taking into account the stage of, and any interruptions in, the candidate's career. Specific criteria used to evaluate an application may vary across disciplines (e.g., influence of research on health policy may not be relevant to the assessment of a basic, biomedical application). Similarly, criteria will vary with stage of career (e.g., international recognition may not be relevant to a New Investigator application). Your commentary should include positive remarks as well as constructive criticism.
1. Track Record
2. Present Proposal
3. Environment and Support
4. Summary of Assessment
The reviewer should provide a summary of the overall application, highlighting both strengths and weaknesses. The goal of the summary is to help candidates understand the assessment they receive and their ranking within the competition.
Please do not indicate your numeric rating on your review.
A. Sequence of Steps for Review of Each Application:
| 1. Internal referees announce their ratings |
| 2. Triage: Review Terminated if rated 2.9 or below by both reviewers and external referees are in relative agreement and there is no objection from other committee members committee members do not vote the rating is calculated as the mean of the two internal reviewers voting members record a "T" on their rating sheets |
| 3. Internal Reviewers: provide just enough summary to introduce application consider strengths and weaknesses |
| 4. External Referees (if applicable): internal reviewers present external referees' comments |
| 5. Discussion of application should focus on: differences of view between reviewers factors that influenced ratings |
| 6. Chair provides a summary of conclusions from discussion: strengths and weaknesses Chair may prepare a brief note for applicant |
| 7. Consensus rating determined by internal reviewers: use full scale check consistency with previous applications if consensus cannot be reached, use mean of internals' ratings |
| 8. Individual ratings: rate +/- 0.5 of consensus rating proposed by internal reviewers confidential vote internals are not bound to the consensus rating |
| 9. Issues to be flagged (if necessary): eligibility ethics human stem cells |
B. Once all applications have been reviewed:
Examine consensus ratings and if committee believes there has been significant drift in standards, re-review of one or a few applications is permitted. Any committee member with a conflict of interest must again leave the room. Following discussion, the two internal reviewers determine a consensus rating and voting proceeds as before.
Revised September 2007
