News 58 of 118

In Conversation: Carol Weiss and Evert Lindquist on Policymaking and Research

Evaluators are studying how to make sure research reaches the policymakers who can most use it. (IDRC Photo: Y. Beaulieu)

2003-09-11

It seems logical to suggest that public policies would be better constructed if policymakers had access to the best research. Yet all too often, researchers and policymakers inhabit separate spheres — the work of the two does not connect. For example, Harvard University professor Carol Weiss tells a story arising from her experience of being asked by the United States government in the 1960s to evaluate a program of the “War on Poverty.” After completing a three-volume evaluation of the program, the Harvard educator heard nothing back from the government. It was as if her work had fallen into a void. After that experience, part of her research into evaluation, organizational decision-making, and research methods has dealt with this very question of how researchers can have an impact on the policy-making process.

Professor Weiss and University of Victoria political scientist Evert Lindquist came to the International Development Research Centre (IDRC) for a series of workshops on the link between research and public policy. [See related sidebar: Carol Weiss on IDRC’s study of the public policy impact of its projects] Professor Weiss has written 11 books and over 100 articles on evaluation and public policy research and ran Harvard’s postdoctoral program on evaluation. Professor Lindquist is director of the University of Victoria’s School of Public Administration and has written extensively on the machinery of government and policy-making, policy communities and networks, and the role of think tanks. Reports magazine spoke with both of them at IDRC’s Ottawa headquarters.

What kind of role — or roles — can researchers have in relation to public policy?

Carol Weiss: They can provide information, ideas, and advocacy. Researchers provide data, which is important. They bring to the forefront data that documents existing conditions and that indicates whether we are going in the right direction or the wrong direction. Information like: crime has gone up or crime has gone down. Researchers can also provide ideas, generalizations: We’ve looked at all the studies, and these are the generalizations that we think are true. Or they can provide advocacy. One of my colleagues has been studying the intersection between race and education for 30 years and he has very strong evidence and very strong opinions about what should be done to improve education for minority students. He’s championing that cause, and he has a lot of evidence to support his position. He’s a very active advocate, and that may have a direct influence on policy.

For how long have researchers been trying to influence public policy?

Carol Weiss: In the latter part of the 19th century and the early part of 20th century scholars were talking about things like pauperism, slums, and crime — all the relatively new, unfortunate consequences of modernization and industrialization. At the time, they were sometimes talking directly about policy, more often for social understanding — trying to enhance the public debate on these issues.

How long has this effort to influence policy been the subject of academic study?

Evert Lindquist: Academics have thought about the impact of research almost since the turn of the last century. Interest gathered with the ascendancy of the social sciences, and it grew during World War II and immediately thereafter. And as Carol Weiss has pointed out, justification for the War on Poverty in the United States during the 1960s was provided by social scientists doing research on poverty. It is not surprising that social scientists began to wonder about how well that advice worked — and about whether the policies motivated by that research were successful. When academics started looking at this in the 1970s, they found that the social science research often wasn’t used, or that the policies supposedly predicated on the research really didn’t work.

The original premise was that research gets produced, delivers insight, and feeds into decision-making calculations and policy. When this came into question, people started asking: Is the root of the problem that the focus of research was inappropriate? Did a good brokerage function exist between the people asking for research and those who were producing it? Or are there different channels of influence? What emerged was an understanding that what’s important is not just whether policymakers get exposed to and absorb information — but the realization that research moves into a conflictual and dynamic process, with many contending ideas. It's a rare time when findings, new concepts, and new theories have an immediate impact. Instead, they enter into this cauldron where people eventually change worldviews of what the problems are and what reasonable solutions might be. By the late 1970s early 1980s, it was understood that research was more likely to have an indirect impact; that a social dynamic was at work.

My work zeroed in on the fact that there are a multitude of researchers and decision-makers in every policy sector, along with a constellation of other actors — interest groups, think tanks, and consultants. Literature on policy networks and policy communities emerged in the late 1980s and early 1990s. Now we take for granted that, in any community or policy sector, there is a complex array of actors advocating different clusters of ideas.

Despite all the academic study of the subject, isn’t there still a random element to the way research influences policy? It seems that bureaucracies are conservative and ignore new policy options — researchers have to be lucky enough to have their work noticed during periods of transition or upheaval.

Evert Lindquist: I'll answer that by referring to another body of literature that emerged in the late 1980s and 1990s on agenda setting, which examined why decisions get made, or not taken, at certain points in time. This literature was inspired by garbage can models of decision-making from the early 1970s that focused on decision-making opportunities or policy windows. Without external influences on policy sectors, there is an innate bias in the policy-making processes. Unless a system has a reason to change, it will not have great receptivity to new ideas that challenge its underpinnings. Examples of such external impacts and policy windows include changes in governments or dramatic developments in other policy areas. For instance, think about the war in Iraq and how that’s affecting Canada’s international trade policies.

Given the need for policy windows, champions, and for ideas to percolate through society, what would be a realistic expectation for how frequently research can influence policy?

Carol Weiss: Well, it depends on what you mean by influence. If you mean that research causes a specific piece of legislation to get passed, that’s not very common. Although in the Ukraine they managed to pass at least three major pieces of environmental legislation on the basis of research IDRC has supported. But by and large you don't see so much direct influence on policy — at least not immediately. What happens more often is that a research project contributes to what I have called “enlightenment.” It punctures old myths, offers new perspectives, and changes the priority of issues. Research also helps to build capability. This is a longer-term influence where the research capabilities of researchers, or institutions, or whole countries are increased. Then in some cases the public gets involved in the decision-making, or the local authorities get involved and those are all core things in and of themselves.

Professor Lindquist, are the policy-making processes you’ve described equally applicable in developing countries as in the countries of North America and Europe?

Evert Lindquist: I think the short answer is “yes.” But the context will be different. In developed countries, policy sectors will encompass many highly capable public, private, and nonprofit organizations, whereas in some developing countries you might be looking at villages, or clans, or families, or fewer people and less capable organizations in the decision-making process because there are fewer resources in the system. But that doesn’t mean complexities are not at play in the policy development process in developing countries: there are contending ideas about what’s right or wrong; there are intermittent windows of opportunity that come along; there are people and organizations in civil society who challenge existing policy regimes; and there is a role for researchers who inform the decision-making process. So I think that these ideas — though grown for comprehending a somewhat different terrain — can be used to subtlety capture the different kinds of complexity associated with Southern jurisdictions and communities.

Stephen Dale is a freelance writer based in Ottawa.

For more information:

Prof. Carol H. Weiss, Professor of Education, Harvard Graduate School of Education, Gutman 467, Cambridge, Massachusetts 02138, United States; Phone: (617) 495-4144; Email: carol_weiss@harvard.edu

Prof. Evert A. Lindquist, Director of the School of Public Administration, University of Victoria, 3800 Finnerty Road, Victoria, British Columbia, Canada V8P 5C2; Phone: (250) 721-8084; Fax: (250) 721-8849; Email: evert@uvic.ca

Evaluation Unit, IDRC, 250 Albert Street, PO Box 8500, Ottawa, Ontario, Canada K1G 3H9; Phone: (613) 236-6163 ext. 2350; Fax: (613) 563-0815; Email: evaluation@idrc.ca

Sidebar

Carol Weiss on IDRC’s study of the public policy impact of its projects

A report being produced by IDRC is examining how, whether, and under what conditions IDRC-supported research projects have played a role in the formulation of public policy. The study is looking at a wide range of projects in different geographic locations. The findings of this report were discussed in a two-day series of meetings and workshops at IDRC headquarters in Ottawa.

How common is it to undertake studies to determine the extent to which projects have had a positive impact on public policy?

I don’t know any other study that compares so many different projects in so many places, and analyzes them using a centrally designed set of questions. So these cases become comparable on many dimensions. The scale is also unique: there are 25 cases. And the studies were done very quickly — in less than two years. So it represents the use of a lot of talent and resources.

What do you think was accomplished by doing this comparative study? What’s the benefit of looking back on these projects, and discussing their relative accomplishments?

You can think of [this study] in three different ways. You can think of it as journalism. You can think of it as organizational history. Or you can think of it as social science. And these are all very reputable, reasonable, logical ways of working.

If you think of it as journalism, they are nuanced accounts of what has gone on, what IDRC does. These accounts are presented to the staff and management, and people discuss them at workshops all over the world. And each person can take lessons from them and apply those to their own work. This is what it's all about: you derive some lessons from the experience.

Or you can think of it as history, in the sense that there is a corporate consensus about what this study means about IDRC and how it works and how it should work. And that becomes incorporated into the narrative of the organization, into the culture of the organization. “This is who we are and this is what we do.” This culture then becomes very pervasive and new people come in, and they absorb this as part of the culture of the organization, and it has staying power beyond the individual. It is organizational in the sense that this is now a consensual, permeated narrative.

Or you can think of the study as social science. If it’s social science, then you are trying to arrive at some generalized knowledge. You are trying to analyze the data very specifically, for example to say: What are the conditions under which projects have more or less policy influence? I think the plan is to go ahead and analyze the data, taking into account the provocative ideas that people have offered and suggested, and proceed with a very systematic empirical analysis. Now, another thing that social science can do is to go up a further level of abstraction, to contribute to theory. If you are contributing to social science theory, you are going up from the specifics of the cases to some social science concepts that talk in terms of things like power, conflict, incentives, and social norms. That would be another outgrowth of viewing this as social science.

When there are similarities between projects, isn’t it possible that they are randomly, coincidentally related to the specifics of each project, rather than to some general principle? Isn’t it difficult to draw general conclusions from such a wide variety of studies?

It is — especially when you are doing this internationally. If you are doing this on, say, education in the United States — where there’s a lot in common across places, where you have similar structures, rules, budgets, and similar testing — it's much easier. But here, where you are talking about mining in Peru, and fisheries in Asia, or trade between countries — it’s a real challenge to try to generalize from that.

But several people at these meetings have already taken that step. One person remarked, after these two and a half days of discussion, that the confluence of five things seemed to be essential to successful influence on policy. The first is a clear intent to influence policy at the outset. Then you need a partnership where the local partners have authority to set the direction, and take the leadership, and have the administrative capabilities to do that. You also have to have champions on both the IDRC side and the partner side. Another key factor is that there are good personal relationships based on respect and trust. The fifth requirement is that IDRC is persistent and hangs on to what it is committed to. And that was one person’s hypothesis on how the analysis could proceed, and those ideas will be tested to see if those conditions consequently show up in projects that had a lot of policy influence.

Top of Page

News 58 of 118