Responsibilities of Reviewers
Avoid Conflict of Interest
Respect the Confidentiality of Applications
Be Aware of the Reviewer Community's Scoring of the Benchmark Application
Reviewing the Assigned Applications
Read the Applications
Rate the Applications
Provide Other Information for CIHR and Feedback to Applicants
Send Review Results to CIHR via ResearchNet
Be Prepared for a Re-Review Request from CIHR
Annex 1: The Benchmark Application
Annex 2: Criteria and Rating Scales
Annex 3: Calibrating Scores
Annex 4: Reviewer WorkSheet
Guide to Locating Information
Annex 5: Examples of Electronic Forms via ResearchNet
You must not be involved in the review if the applicant or the proposed research supervisor:
If you would be in conflict of interest, or might be perceived to be in conflict of interest, notify CIHR immediately and the application will be assigned to another reviewer.
Do not forward copies of applications or discuss them with others.
Step 1: Rate the Benchmark Application
A benchmark application appears in Annex 1. If you have already rated it, go to Step 2. If not, read then rate it using the six criteria and the scales provided in Annex 2.
Step 2: Compare Your Ratings to the Pattern for the Reviewer Community
View your ratings in relation to those of the community by completing the table in Annex 3. In cases where your rating falls outside the normal range, revisit your assessment and consider the reasons given by other reviewers for their ratings.
Read all of your assigned applications before rating any of them. As you examine each application, jot down notes to capture your impressions. The Reviewer Worksheet (Annex 4) provides a template that you could use. Note that the worksheet will not be filed with CIHR.
Be alert to unconscious bias related to gender, discipline or geographic location. Remember that:
You are free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. Note however that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.
Examine each application in detail and rate it against each of the six criteria described in Annex 2,
Reviews for Fellowship applications are now submitted to CIHR via ResearchNet. The electronic rating forms are available to reviewers you once you log on to ResearchNet.
Human Stem Cell Research: Indicate if the candidate's research involves human stem cells.
Other Comments for CIHR: Mention any ethical issues, et cetera.
Feedback for the Applicant: Prepare brief comments on the application for transmittal to the candidate by CIHR via ResearchNet after the competition. Carefully avoid language that might be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Cover both strengths and weaknesses, particularly those that could be realistically addressed by the applicant.
Please respect the deadline provided by CIHR by submitting your reviews via ResearchNet by the date specified via correspondance with CIHR staff responsible for the Fellowship program.
When all scores are received, CIHR will calculate an average for each applicant. CIHR will then identify applications which are at risk of an unfair decision because of a wide spread between the two reviewers' ratings. In such cases, CIHR will ask both reviewers to reconsider their initial assessment and resubmit scores. Usually this second review will reduce the gap between scores to an acceptable size. If it does not, CIHR will obtain a third review.
Just in case you are asked to do a re-review, keep the applications and your working notes on file until competition results have been announced.
Note to Reviewers
Information on a hypothetical candidate is presented as though it has been extracted for you from a CIHR application form by another reviewer.
The benchmark application presents information on a hypothetical candidate submitting an application to the Fall 2005 competition. It is intentionally generic; the candidate could be female or male, in any health research area, and training inside or outside Canada. Do not worry about this lack of specificity.
The information has been organized as follows:
A - Introduction to the Candidate
1. The Candidate's Plans
2. The Proposed Research Project
3. Honours, Awards and Academic Distinctions of the Candidate
4. The Candidate's Publications and Related Research Achievements
5. Characteristics and Abilities of the Candidate
6. Research Activity, Resources and Mentorship in the Training Environment.
Journal Articles
# | Year | Position of candidate's name in the list of authors | Contribution of candidate to the publication | Impact of journal* | Comment |
---|---|---|---|---|---|
1 |
1998 |
Second of three authors |
25% |
Medium |
From summer student work |
2 |
1999 |
First of three authors |
65% |
Medium |
From summer student work |
3 |
2000 |
First of two authors |
60% |
Low |
Case study from clinical work |
4 |
2002 |
Third of four authors |
20% |
Medium |
From Master's work |
5 |
2002 |
Second of three authors |
40% |
High |
Some overlap with article 4 |
6 |
2003 |
Second of five authors |
20% |
Medium |
From Master's |
7 |
2004 |
Sole author |
100% |
Low |
Review article. From PhD work |
8 |
2004 |
First of two authors |
85% |
Medium |
From PhD |
9 |
2005 |
First of three authors |
75% |
High |
In press |
10 |
2005 |
Second of five authors |
30% |
Medium |
In press |
11 | First of two authors |
75% |
High |
Submitted |
* Field normalized journal impact
Related Research Achievements
Characteristic or ability of the candidate | Sponsor #1 | Sponsor #2 | Sponsor #3 | Mean |
---|---|---|---|---|
Independence and critical thought |
|
|
|
|
Energetic and focused pursuit of knowledge |
|
|
|
|
Creativity |
|
|
|
|
Mean |
|
|
|
|
Rating benchmarks were:
3.0 to 3.4 | Above average in strength and frequency of demonstration |
3.5 to 3.9 | Strong and frequently demonstrated |
4.0 to 4.4 | Very strong and very frequently demonstrated |
4.5 to 4.9 | Exceptionally strong and always demonstrated |
Summaries of comments from the sponsors follow in the next three tables.
From the report by the candidate's PhD supervisor, 2002 to present
Characteristic or ability of the candidate | Summary of sponsor's comments | Sponsor's rating |
---|---|---|
Potential for independent research | Very positive. Indicated that in fifteen years of supervising students only two gave stronger signs of researcher potential. |
n/a |
Independence and capacity to think critically | Gave good examples of situations where the candidate demonstrated independence and initiative. There was no specific example of a situation where the candidate demonstrated very strong critical thinking. |
4.2 |
Capacity for energetic and focused pursuit of knowledge | Emphasized the candidate's organizational skills and their ability to motivate and work with others. Described the candidate as "someone who always meets their commitments". |
4.5 |
Capacity for creative thinking | Mentioned the candidate's key role in the development of a novel research instrument. |
4.1 |
Most significant achievement | Referred again to the design and development of the research methodology and instrument. |
n/a |
From the report by the candidate's Master's degree supervisor, 2000-2002
Characteristic or ability of the candidate | Summary of sponsor's comments | Sponsor's rating |
---|---|---|
Potential for independent research | Very favourable. Has kept in touch with the candidate and is following their career development with interest. |
n/a |
Independence and capacity to think critically | Cited the candidate's exceptionally well-written and strongly argued proposal for a Master's research project. Also mentioned leadership qualities. |
4.1 |
Capacity for energetic and focused pursuit of knowledge | Mentioned the candidate's ability to write papers and meet deadlines while remaining actively involved in rowing and graduate student affairs. |
4.4 |
Capacity for creative thinking | Indicated that the Master's research topic, although important, may not have allowed the candidate to fully demonstrate their creativity. |
3.8 |
Most significant achievement | Described the question and answer session after a conference presentation in which it was clear that the candidate's ideas were influencing the thinking of the leading researchers in the area. |
n/a |
From the report of the candidate's clinical supervisor, 1999.
Characteristic or ability of the candidate | Summary of sponsor's comments | Sponsor's rating |
---|---|---|
Potential for independent research | Supervised the candidate's clinical work in 1999. Felt unable to judge research potential but described the candidate as an "evidence-based" clinician. |
n/a |
Independence and capacity to think critically | Wrote of the candidate's ability to probe beyond the obvious diagnosis, always exploring the possibility of multiple causes and the full range of treatment options. |
4.0 |
Capacity for energetic and focused pursuit of knowledge | Described the candidate as "tireless, determined, and one of the best young clinicians I have ever encountered". Mentioned that the candidate had helped prepare a case study for publication. |
4.3 |
Capacity for creative thinking | Gave as an example a situation in which the candidate organized a team of specialists to provide around-the-clock care for a particular patient. |
4.0 |
Most significant achievement | Mentioned that the candidate had organized a music program in which volunteers facilitated concerts by patients. |
n/a |
Trainee | Period | Current position |
---|---|---|
Postdoctoral fellow |
|
Public servant (Ministry of Health) |
Master's student |
|
Postdoctoral fellow |
Postdoctoral fellow |
|
Assistant professor |
PhD student |
|
Self-employed |
Master's student |
|
PhD student |
PhD student |
|
Volunteer in an international aid program |
The Annex contains:
A - An Overview of the Six Selection Criteria
1. Evaluating the Candidate's Plans
2. Evaluating the Proposed Research Project
3. Evaluating the Honours, Awards and Academic Distinctions of the Candidate
4. Evaluating the Candidate's Publications and Related Research Achievement
5. Evaluating the Characteristics and Abilities of the Candidate
6. Evaluating the Research Activity, Resources and Mentorship in the Training Environment.
Fellowship selection criteria and weights are based on studies of the predictors of post-training research activity. They have been fine-tuned through two surveys of Fellowship reviewers.
The raw scores that you submit for each criterion on the 0 to 4.9 scale will be weighted automatically by CIHR in the calculation of an overall score.
The Six Criteria and their Weights in the Overall Score:
Criterion |
| |
Candidate data
| ||
Candidate's plans (Training expectations) |
|
|
Proposed research project |
| |
Honours, awards and academic distinction * |
| |
Publications and related research achievements ** |
| |
Assessments by Sponsors
| ||
Characteristics and abilities of the candidate |
|
|
Fellowship Training Environment
| ||
Research activity, resources and mentorship |
|
|
|
|
1 Includes health professionals who hold a PhD degree.
There are slight differences in the weighting of criteria for post-PhD Fellowship candidates and other candidates:
* Honours, Awards and Academic Achievements carry a weight of 15% for other candidates.
** Publications and related research achievements carry a 25% weigh for other candidates.
10% weight for all candidates
Working Definition
A description of the applicant's career intentions and proposal for achieving them.
What to Look For
Clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Faultless depiction of research career intentions and relevance of the proposed training. Ideal career path. |
4.0 to 4.4 | Clear, convincing depiction of research career intentions and relevance of the proposed training. Highly appropriate career path. |
3.5 to 3.9 | Very good depiction of research career intentions and relevance of the proposed training. Logical career path. |
3.0 to 3.4 | Reasonable depiction of research career intentions and relevance of the proposed training. Suitable career path. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Weak |
0 | Not acceptable |
10% weight for all candidates
Working Definition
A carefully planned, systematic study aimed at clearly answering a question in health research.
What to Look For
The ideal project is one that is best for the candidate given their education, experience and interests. It is the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training.
Bear in mind that it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Extraordinary optimization of: challenge to the candidate, scientific importance and feasibility of completion during the Fellowship period. An ideal project that is faultlessly outlined |
4.0 to 4.4 | Excellent optimization of: challenge, importance and feasibility. A highly suitable project that was superbly outlined. |
3.5 to 3.9 | Strong optimization of: challenge, importance, and feasibility. A very suitable project that was very clearly outlined. |
3.0 to 3.4 | Good optimization of challenge, scientific importance and feasibility. A suitable project that was well outlined. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Below average |
0 to 0.9 | Not acceptable |
5% Weight for Post-PhD applications, 15% for Others
Working Definition
Official recognition or prizes signifying special qualities of the recipient. Includes accomplishments in terms of formal education and scholarship.
What to Look For
In assessing this variable and other achievements of the candidate, it is essential that you take into consideration the career path that they have followed to date. Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Note relevance to research and whether the recognition is regional, national or international. Note the length of time required to complete academic programs and any indications of special academic distinction.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Extraordinary All aspects of the candidate's distinctions (their number, importance and breadth) indicate recognition of a very rarely encountered level of talent. |
4.0 to 4.4 | Excellent Several aspects of the candidate's distinctions (their number, importance or depth) indicate recognition of superb talent. |
3.5 to 3.9 | Very Good At least one aspect of the candidate's distinctions (their number, importance or depth) indicates recognition of a talent. |
3.0 to 3.4 | Good The candidate's distinctions indicate an above-average performance. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Below average |
0 to 0.9 | Not acceptable |
35% Weight for Post-PhD applicants; 25% for Others
Working Definition
Articles, chapters or books published (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights.
What to Look For
Evidence of achievement in research relative to opportunity to date. Bear in mind that opportunity to publish may vary according to research discipline and life course (e.g., time spent raising children).
For publications, observe the number of co-authors and the position of the candidate's name in the authors list. Note the candidate's role in publications and their estimated percent contribution to the work.
Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activity. Note the candidate's other professional activities. Consider any patents or copyrights to which the candidate contributed.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Brilliant All aspects of the candidate's publications and related research achievements (number, likely impact and breadth) indicate an extraordinarily productive and creative individual. |
4.0 to 4.4 | Excellent Several aspects of the candidate's publications and related research achievements (number, likely impact or breadth) indicate excellent productivity and creativity. |
3.5 to 3.9 | Strong At least one aspect of the candidate's publications and related achievements (number, likely impact or breadth) indicate very good productivity or creativity. |
3.0 to 3.4 | Good There is evidence of greater than expected involvement in publication and related research activities. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Below average |
0 to 0.9 | Not acceptable |
20% weight for all candidates
Working Definition
A perspective on the candidate provided by persons who are familiar with her/his characteristics and abilities.
What to Look For
Evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with career research achievement. Examine the sponsor's scores, recognizing that high scores are common while low scores are not. Read the supporting text carefully, taking note of the extent to which they justify the scores.
Look particularly for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent.
Look also for indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
If the candidate has had an opportunity to conduct research, look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Extraordinary An outstandingly critical, original and independent thinker. Exceptionally focused, energetic and creative. An ideal role model for others. |
4.0 to 4.4 | First-Rate A highly critical, original and independent thinker. Very focused, energetic and creative. Excellent potential for future research leadership. |
3.5 to 3.9 | Strong Clearly a critical, original and independent thinker. Definitely focused, energetic and creative. Very good potential for success as an independent researcher. |
3.0 to 3.4 | Good Appears to be a critical, original and independent thinker. Seems to be focused, energetic and creative. Above average potential for a productive career in research. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Below average |
0 to 0.9 | Not acceptable |
20% weight for all candidates
Working Definition
Elements of the research milieu that will contribute directly or indirectly to the quality of the candidate's research training experience.
What to Look For
Review information on the education, research experience, qualifications, honours and awards of the Fellowship supervisor. Examine the supervisor's publication record to get a sense of productivity, impact and collaboration.
Determine the space, facilities and personnel support available. Review the information on grants currently held, noting the extent to which the supervisor was "principal or co-applicant" for the funds. Get a sense of the resources available and the overall level of activity.
Review the supervisor's training record. Note for each person listed: the level of training, length of time with the supervisor, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisor. Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.
Rating Range | Benchmark |
---|---|
4.5 to 4.9 | Exceptional A vibrant, world-class research environment. Outstanding availability of research resources. Superb mentorship. |
4.0 to 4.4 | Excellent A highly active research environment. Excellent availability of research resources First-rate mentorship. |
3.5 to 3.9 | Strong A very active research environment. Very good availability of research resources. Strong mentorship. |
3.0 to 3.4 | Good An active research environment. Sufficient research resources available. Appropriate mentorship. |
2.0 to 2.9 | Mediocre |
1.0 to 1.9 | Below average |
0 to 0.9 | Not acceptable |
Position your scores relative to those of the reviewer community
For each criterion circle the range of scores that includes your rating for the benchmark application
Criterion |
Range of Ratings of the Benchmark Application | |||
Low |
A large majority of reviewers rated the application in these ranges |
High | ||
Candidate's Plans |
|
|
|
|
Proposed Research Project |
|
|
|
|
Honours, Awards and Academic Distinctions |
|
|
|
|
Publications and Related Research Achievements |
|
|
|
|
Characteristics and Abilities of the Candidate |
|
|
|
|
Research Activity, Resources and Mentorship in the Training Environment |
|
|
|
|
|
(The mean rating lay between the italicized numbers) |
|
Understanding Differences
For criteria on which your rating was Low or High relative to most other reviewers you should reread the benchmark application and then review the rating scale.
If after reviewing your score you still see no reason for shifting it into the normal range, consider the reasons why other reviewers did not give a higher or lower rating to the benchmark application.
Criterion | Reviewers' Reasons for Not Giving a Higher Rating |
---|---|
Candidate's Plans |
|
Proposed Research Project |
|
Honours, Awards and Academic Distinctions |
|
Publications and Related Research Achievements |
|
Characteristics and Abilities of the Candidate |
|
Research Activity, Resources and Mentorship in the Training Environment |
|
Criterion | Reviewers' Reasons for Not Giving a Lower Rating |
---|---|
Candidate's Plans |
|
Proposed Research Project |
|
Honours, Awards and Academic Distinctions |
|
Publications and Related Research Achievements |
|
Characteristics and Abilities of the Candidate |
|
Research Activity, Resources and Mentorship in the Training Environment |
|
This template is strictly for your working notes and will not be filed with CIHR.
Name of Applicant | |
Application Number | |
Background Information on the Candidate Refer to the CV Module of the Candidate |
Degrees held or in progress: Time for completion of degree programs: Research experience: |
Candidate's Plans Refer to the Training Expectations in the CIHR Training Module |
Link between proposed and prior training: Career goals: |
Proposed Research Project Refer to the Abstract and the Proposed Training Program in the CIHR Training Module |
Project: Expected duration of training: Suitability of project for applicant's career: |
Honours, Awards and Academic Distinction Refer to the CV Module of the Candidate and Refer to university transcripts |
|
Publications and related research activity Refer to the CV Module of the Candidate |
Papers: Presentations: Other: Candidate's role: |
Characteristics and Abilities of the Candidate Refer to sponsors' assessments |
Sponsor: Relationship to candidate: Notes: |
Sponsor: Relationship to candidate: Notes: | |
Sponsor: Relationship to candidate: Notes: | |
Research Activity in the proposed Fellowship Training Environment Refer to the CV Module(s) of the proposed supervisor(s) |
Name of proposed supervisor: Qualifications: Publication activity: Impact: Collaborators: |
Resources Available in the proposed Fellowship training environment Refer to the CV Module(s) of the proposed supervisor(s) |
Space, facilities and personnel support available (refer to page 8 of CIHR Training Module): Research funding available to the supervisor:Other key resources: |
Mentorship record in the proposed Fellowship training environment Refer to the CV Module(s) of the proposed supervisor(s) |
Trainees in the last five years - level and outcome: |
General Notes on the Application | Overall impression: |
Review Criterion | Where to Find Information | |
Application Module | Sections | |
Candidate's Plans | Training Module |
|
Candidate's Research Project | Training Module |
|
Candidate's Honours, Awards and Academic Distinction |
Applicant's CV module |
|
Candidates's Publications and Related Research Achievements |
Applicant's CV module |
|
Candidate's Characteristics and Abilities |
Sponsors' assessments |
|
Research Activity, Resources and Mentorship in the Fellowship Training Environment |
Training module: |
|
Fellowship Supervisor's CV module: |
|
- Form for Reporting Special Issues with an Application [ PDF (15.97 KB) | Help ]
- Form for Providing Feedback to Candidates [ PDF (16.94 KB) | Help ]