*This report was prepared by Patricia M. Baranek, PhD, Department of Health Policy, Management and Research, University of Toronto. Tel: (416) 921-5973; Email: pat.baranek@utoronto.ca, with the assistance of Heather Chappell, doctoral student, Department of Health Policy, Management and Research, University of Toronto.
Based on the national consultation in 1999 by five national organizations to identify priority areas in health services and policy research, fifteen research themes were identified. Two of the themes identified in Listening for Direction: A National Consultation on Health Services and Policy Issues were financing and public expectations. In particular, the document outlined the following priorities:
In 2001, the Institute of Health Services and Policy Research (IHSPR) combined the two priorities into a strategic initiative and issued a Request for Application (RFA), "Financing Health Care in the Face of Changing Public Expectations".The RFA resulted in the submission of six eligible applications, only two of which were successful. The Institute issued a second Request in 2002 and received 2 eligible applications. The two rounds of competition resulted in only 8 applications in total, 2 of which were funded. The weaknesses in the submissions were said to be similar to other competitions, namely, poor methods, a weak team of researchers and partners, and/or non-relevance of the research to policy.
Given the importance and urgency of the topic as identified in the national consultations, the IHSPR asked Professors Raisa Deber and Julia Abelson, two successful applicants, to organize and host a workshop to determine the reasons for the low response and success rates to both its competitions.
The objectives of the Workshop were to explore the underlying reasons for the low response, for the shortage of qualified applications, and to make recommendations to increase both the number of submissions and the number of successful applications in a future RFA in this area. In particular, the Workshop was to determine:
The workshop was held on September 15, 2003, at the University of Toronto. Participants included academic researchers, and policy, administrative and clinical decision makers. The workshop attempted to balance geographic representation, academic disciplines (health services, health policy, public participation, business, economics, political science, nursing, and medicine) as well as academic rank (doctoral students, junior and senior academics). Participants included successful and unsuccessful applicants to the RFAs, members of the review panels, and those currently conducting the type of work outlined in the original RFAs (and/or with skills and interests which could predispose them to pursue this type of research) but who had not applied. See Appendices 1 and 2 for the agenda and list of participants.
Diane Watson, Assistant Director, Institute of Health Services and Policy Research, outlined four tenets of the Institute's strategic plan and explicit areas of investment; i.e., to build a community of excellent researchers, to expand and enhance research resources, to support strategic research, and to translate that research knowledge to policy/decision makers. The two RFAs on Financing Health Care within Changing Public Expectations were part of the Institute's mandate to support strategic research. Despite the directions recommended by the community of researchers and decision makers and the explicit mandate of the IHSPR ("...support innovative research, capacity building and knowledge translation initiatives designed to improve the way health care services are organized, regulated, managed, financed, paid for, used and delivered..."), the results of the two competitions were disappointing. Because this area of research is still considered of high priority, the IHSPR is thinking of posting an RFA on financing and changing public expectations in December and is very interested in learning from the workshop participants, who represent both expert and upcoming researchers in these areas, how best to improve the RFA and its ability to attract successful submissions.
The discussion of issues and barriers to successful submissions and recommendations to address them fell into four areas:
The objective of the RFA was to support cross-disciplinary research projects regarding:
While some of the participants found the objectives of the RFA straightforward, others found that they lacked clarity or that the interesting questions were buried in the text. Consequently, the latter group was unwilling to risk investing the time and resources in developing a proposal.
Participants were unclear if submissions had to address both the finance objectives (1 and 5) as well as the public expectations objectives (2, 3, and 4). Many thought the objectives were too wide ranging and, as a result, too complex to address in the specified duration of funding unless the required team had already been assembled (which was usually not the case).
Given the disparate skills required to address such a comprehensive program of research, three years of funding was considered insufficient. Similarly, because of the time-limited and changing nature of strategic grants and priorities, there was concern that researchers would be unwilling to invest their time and resources in a program of research that was under threat of being discontinued in future years as a priority area by the Institute.
Some researchers noted that the two successful applicants had already been working in related areas. Policy makers tend to want results as soon as they identify a priority area; the research cycle takes a longer time period. In that connection, it was suggested that the IHSPR should ensure that objectives not be too prescriptive, and allow experienced researchers enough latitude to identify, define and begin work in areas 'ahead of the curve' - that is, which they believe are worthy and timely for exploration and will be recognized as policy relevant, rather than restricting research to those areas already so recognized.
Given the discomfort around aligning the two areas as described above, one recommendation forwarded was that objectives 1 and 5 be de-linked from 2, 3, and 4. Although this was generally supported, there was also support for attempting to gain potential 'value added' from linking the two topics and in attracting "fresh blood" and ideas from combining the two as well as enticing disciplines outside the traditional health and economics fields, such as psychology, history, sociology, and geography to enter the field.
Another set of issues related to the intended target audience for the resultant research and how researchers should link to policy makers. Some participants maintained that the type of research considered relevant and useful for decision makers (managers) is different from research that informs policy makers. Managers are interested in micro-level data and analyses while policy makers are more interested in the macro-level. Others did not draw a distinction between the two audiences.
In that connection, there were some questions as to how results would be used, particularly within a publicly financed health care system. Understanding the nature of public values, expectations and support is important for policy. As one example, issues of how to finance health care are clearly influenced by public values and public expectations. Given that the RFA required the research to "inform policy and management decisions in Canada," the fear was that researchers could be either explicitly or implicitly providing evidence to support particular ideological positions or political ends. Participants accordingly differed as to whether expected results should steer which research questions are asked. For example, there are undoubtedly values underlying different payment mechanisms; these in turn have varying impacts on costs, quality, or access to services. Most agreed that these are empirical questions that need to be explored. Rather than avoiding uncomfortable questions, researchers must be transparent and explicit in highlighting the inevitable trade-offs involved.
Response: Diane Watson stated that there were two drivers to RFAs: research that is achievable, and research that is needed. The challenge in strategic initiatives is to juxtapose the two. By definition, therefore, the funding is targeted to research that is hard to do. Given that a good many participants felt the link between the two areas was important to maintain, she proffered a suggestion that a renewed RFA could have three objectives of which researchers only need address one. That is, the RFA could fund any one of three streams of research:
In this way, financing and public expectations research could continue unconstrained by the need to link with the other1. At the same time, funding would be available for those who believe the relationship is an important one to explore and who are interested in pursuing it. Questions that would need to be resolved include whether each of the three objectives should have separate, dedicated funds which would be carried over if not spent, or whether submissions for the three objectives should compete with each other for funding.
Most believed that a successful submission to the two RFA competitions required skills from diverse disciplines and partnerships which to date have not had much exposure to each other and which take time to build. The public expectations and perceptions components of the RFA links with the skills of researchers in the behavioural sciences (including psychology, sociology, and media studies), and the financing components with those of economics, policy analysis, and health services research. While there was general consensus that there was value and enthusiasm in bringing diverse disciplines together, there were considerable challenges to overcome: namely, time and therefore, resources, to identify appropriate individuals, build relationships and knowledge of required areas; disciplinary cultures that do not reward collaborative work but solo research and publications; and finally, appropriate publishing venues. It was thought that CIHR could assist this process by creating better linkages with the Social Science and Humanities Research Council (SSHRC).
One participant expressed the sentiment that Canadian health services research itself was not deep enough in terms of numbers of active researchers, and required capacity building. Concern was expressed that this might be undermined by encouraging the entry of other disciplines. Concern was also expressed that researchers in the traditional areas of health finance could get crowded out by these other disciplines. Others argued that the research question should be the sole reason that dictates the need for interdisciplinary teams in submission rather than the desire on the part of the funder to encourage such collaborations for their own sake.
Successful knowledge transfer, it was suggested by some, requires partnerships and teams that include policy and decision makers from the inception of a research proposal. Others cautioned that the mobility of these personnel disrupts the continuity of input making it very difficult to include them on research teams, . Moreover, citing a particular research experience involving decision makers, one researcher commented that "speaking truth to power" may lead to the lack of cooperation or opposition from policy/decision makers.
Given the time and resources required to build appropriate partnerships, a longer term of funding and a phased approach to funding were suggested. Possible CIHR funding mechanisms were explored; namely, Pilot Project grants, Operating grants, Pilot Project plus Operating grants, New Emerging Teams (NET) grants, and Interdisciplinary Capacity Enhancement (ICE) grants. One suggestion was to identify and build the team in the first phase through the NET or ICE grants and to apply for research funding in the second phase through either pilot or operating grants. Applications for the first phase could come from a single individual or an established team, both willing to build a broader team to accomplish the program of research. Given the lack of research depth in both areas of financing and public expectations, one suggestion was to build capacity first in each area before trying to bring the two together. Because of discipline specific cultures and ideologies, it was suggested that time should be spent targeting individuals in other disciplines who were interested in crossing disciplinary boundaries. Conferences could also be used judiciously to identify and target potential teams.
As stated above, those participants who saw the objectives as either unclear or too broad ranging, were unwilling to risk expending the resources required to develop a submission for the RFA. Given the short duration of the funding and the uncertainty of its continuing in the future, a number of participants, especially the senior ones, were disinclined to spend the time and energy in building partnerships and a program of research that would be prematurely truncated. The plethora of other grant competitions and the tendency of senior researchers to gear themselves to the operating grant cycle also drew proposals away from the "RFA of the week." Furthermore, building capacity in this type of collaboration was not likely to be rewarded after the completion of the strategic grant through operating grant support given the low success rate of those competitions (estimated at 20%).
Others highlighted the difficulties and expense in obtaining the data necessary to undertake research in health care financing. Much of the data required for a fulsome analysis is not always available and would be prohibitive to collect. Privacy issues and cost limit access to available administrative data. The negotiations to access data with each province make cross provincial analyses onerous. While data on private financing is scarce, proprietary rights restrict access to what is available making it difficult to undertake public-private comparisons. It was suggested that the costs of collecting data or accessing available data can be factored into a submission's budget. Moreover, partnerships with agencies such as the Canadian Institute of Health Information should be encouraged to avoid the needless duplication of data collection. The Romanow Commission in its work collected considerable primary and secondary data. The CIHR should not only identify available data sets, e.g., polling data, but also create accessible repositories of available data sets, comparable to the National Medical Expenditure data set in the U.S.
Participants indicated that the RFA was not well disseminated and advertised, nor was its timing in the academic year opportune. Given the many grant opportunities with deadlines that span the entire calendar year, RFAs often do not make it onto researchers' radar screen. Many participants learn of competitions through various list servers rather than searching the CIHR website. It was suggested that CIHR could take a leadership role in creating a master list and dissemination strategy of all health granting competitions to avoid duplication of efforts, provide a service to researchers and ensure that they are well informed of the various opportunities. Some regularity in posting competitions would greatly improve notice by the research community, as well as advance notice of upcoming competitions. Participants were informed that with the CIHR more firmly established and staffed, competitions will follow predefined cycles.
As discussed above, the time in which to submit a proposal to the RFA was also considered far too short given the perceived requirement of multi-disciplinary teams.
There was considerable comment on the limitations of the review process and panels. Two main issues arose.
One related to the review of RFAs as compared to open grants, and the extent to which review panels do (or should) use different standards. Because the Institute only establishes a separate panel for an RFA if it attracts approximately 15 submissions, review committees for open competitions are frequently used to adjudicate an RFA. Given the different rating thresholds for funding in open competitions (rarely less than 3.7, and often over 4.0) and RFAs (some have been funded with scores as low as 3.0), some felt that the same review panel may have held RFA applications to the higher standard. In future, the rating threshold for funding applications to RFAs will be no lower than 3.5, not as demanding but closer to that of open competitions.
The second related to the workings of the review panels. In particular, the structure, qualifications, and criteria of review panels were felt to neither mirror nor reinforce the participation of researchers from other disciplines and the competition requirement of cross-disciplinary teams. The interdisciplinary requirement of the RFA, as well as many applications to the open competition, was seen to have been thwarted by review panels whose membership did not allow for an appreciation of some of the knowledge, methods, and skill sets potentially included in submissions. One participant commented that the Romanow Commission had sought out researchers outside of the health field to contribute background work. A number of these individuals had subsequently applied to the CIHR for funding, and had been rejected. It was suggested that transparent and consistent standards should be promoted, and that urgent attention needed to be paid to improving the membership of review panels.
The day's discussion included issues of clarity, inclusiveness, and prescriptiveness of RFA objectives; ways in which to identify, attract and build interdisciplinary teams; notification, suitable deadlines, timelines and cycles for RFAs; and enhancement of the interdisciplinary capacity of review panels, and standardization of the adjudication rules and procedures.
Although consensus was not reached in every case, the following is a summary of suggestions/recommendations made by participants.
1 Because the "public expectation" stream could be a catch-all description for research in that area, some participants believed that this third stream required more specification to link it to financing.
|
1.0 |
Welcome and Introductions (Raisa Deber) | 11:00 - 11:15 |
|
2.0 |
Overview of CIHR Directions and Request for Applications: |
11:15 -11:30 |
|
3.0 |
Discussion: Issues and Barriers to Successful Submissions and Recommendations for Change (Facilitator - Whitney Berta) |
11:30 - 2:45 |
|
3.1 |
RFA Objectives (e.g., appropriateness, understanding) | 11:30 - 12:30 |
|
LUNCH |
12:30 - 1:15 | |
|
3.2 |
Qualifications/Skills of Researchers and Partnerships Required by RFA | 1:15 - 1:45 |
|
3.3 |
Dissemination of RFA/ Application Process/ Review Process | 1:45 - 2:15 |
|
3.4 |
Other Issues/Barriers |
2:15 - 2:45 |
|
BREAK |
2:45 - 3:00 | |
|
4.0 |
Summary of Recommendations for a New RFA (Raisa Deber) | 3:00 - 3:25 |
|
5.0 |
Wrap-Up (Raisa Deber) | 3:25 - 3:30 |
| ONTARIO | |
| ABELSON, Julia Assistant Professor |
Clinical Epidemiology and Biostatistics, McMaster University, Faculty of Health Sciences |
| ARWEILER, Delphine Post-doc |
Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| BARANEK, Patricia, PhD | Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| BERTA, Whitney, PhD Assistant Professor (Facilitator) |
Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| COYTE, Peter, PhD Professor |
Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| DEBER, Raisa, PhD Professor |
Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| GURD, Dr. Geoffrey A/Director |
Research Management and Dissemination Division, Applied Research and Analysis Directorate, Information, Analysis and Connectivity Branch, Health Canada |
| LEMIEUX-CHARLES, Louise, PhD Associate Professor |
Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto |
| NAUENBERG, Eric, PhD | Ontario Ministry of Health and Long Term Care, Toronto, ON |
| O'ROUKE, Michele Associate |
Strategic Initiatives, IHSPR |
| SHORTT, Dr. Samuel | Department of Community Health and Epidemiology, Queen's University |
| WODCHIS, Walter, MA, MAE, PhD Research Scientist | Toronto Rehab, Toronto, ON |
| BRITISH COLUMBIA | |
| ANIS, Dr. Aslam |
St. Paul's Hospital Center for Health Evaluation and Outcome Sciences, Vancouver, BC |
| EVANS, Dr. Robert (Bob) |
Centre for Health Services & Policy Research, Dir Population Health CIAR University of British Columbia |
| WATSON, Diane, PhD Assistant Director |
Canadian Institute of Health Research, Institute of Health Services and Policy Research University Marketplace |
| ALBERTA | |
| JACOBS, Philip Professor/Director |
Department of Public Health Sciences, Faculty of Medicine and Dentistry, University of Alberta |
| WILSON, Dr. Donna |
Faculty of Nursing, University of Alberta |
| QUEBEC | |
| BERLINGUET, Dr. Marc Consultant |
Régie de l'Assurance maladie du Québec (RAMQ), Ministry of Health, Quebec City, Quebec |
| CONTANDRIOPOULOS, Andre Pierre Professor |
Département d'administration de la santé, Université de Montréal |
| FOREST, Dr. Pierre-Gerlier | Universite Laval, Departement de science politique/CAPP |
| NOVA SCOTIA | |
| NESTMAN, Larry Professor |
School of Health Services Administration, Faculty of Health Professions, Dalhousie University |
| NEWFOUNDLAND | |
| NEVILLE, Doreen Associate Professor |
Health Care Policy and Delivery, Division of Community Health, Faculty of Medicine, Memorial University of Newfoundland |
