The Rise of Randomized Experiments in Development Economics

Dani Rodrik has an interesting new paper reflecting on the methodological gap that exists in development economics between the micro and macro approaches. In the last few years, randomized experiments (RE) have become for some economists (especially those at the MIT Poverty Action Lab) the only source of “admissible” evidence in development (at least it constitutes hard evidence as opposed to soft).

Austrian economists should have a lot to say about the rise of RE in development economics. On the positive side, it is true that rather than having some pretence of knowledge, economists and policy makers are better off admitting some level of ignorance as to what may work in development aid. However RE can also be seen as the symptom of two problems that have plagued mainstream economics for a long time.

First is the idea that the methodology of economics must be the same as that of the “hard” sciences. RE was developed originally in biology (especially in bacteriology and virology, if I am not mistaken) and its transfer to economics is what scientism is all about.

Second, while it is a good idea to maintain a humble attitude towards the social process, as F. A. Hayek explained, this does not mean throwing the baby out with the bath water—which is exactly what RE contenders do. Being suspicious of any claims to ex-ante knowledge as to what works in economics amounts to denying the existence of regularities (i.e. laws) in social phenomena that economics can establish. In the extreme, the idea that demand curves slope downwards is of no importance to RE contenders.

It is true that policy implementation is not always an easy exercise. Admitting that one doesn’t always have good knowledge as to whether policy X is going to deliver the expected result is a huge improvement over the intellectual hubris that often characterizes development policy. However, RE contenders see only half of the story. Like Hayek (but without saying so), they emphasize the importance of the context in determining the success of a policy, but they deny that whether a given policy will work also depends upon things we can know ex-ante such as: the state of the de facto institutions, the impact on relative prices, whether the policy enables gains from trade to be captured and exploited, etc. This knowledge is not always available without actually going into the field. But a thorough knowledge of the context doesn’t imply that RE is the only way to get evidence of the effects of a policy (quite the opposite in fact) and, even more importantly, it doesn’t mean that the laws of economics should not be considered as guides in the application of policy.

It is also the case that RE is limited in what it can tell us. RE can only be applied to a few areas and certainly not to the big questions such as monetary policy, etc. Moreover, perfect randomization of subjects is not always feasible; prior knowledge (on the part of subjects) can bias the results; the design of the experiment can itself be biased, etc. In other words, randomization in itself is no guarantee of anything.

Also, the way the RE is imagined in the first place is itself, not random. Researchers are confronted to a problem (e.g. what to do reduce the risk of catching malaria) and the solutions they imagine and test with RE are themselves potentially beneficial. In other words, they imagine solutions based on prior theories they possess about how the world works (e.g. bed nets are more useful at stopping mosquitoes than, say, a glass of water on a table). RE contenders pretend to do research without any priors, this is not true. And open-minded pragmatic approach devoid of theoretical priors does not exist; it’s an illusion that RE contenders maintain. And Rodrik and his co-authors (in their work on country-specific growth strategy) are victim of the same illusion.

At the end of the day, RE can be seen as a form of historicism, as it removes economic laws from the picture and asserts that only pragmatism in some given place and time can tell us what works and what doesn’t for this given place and time (what is called “internal validity” in the RE literature). (And this is in spite of the fact that it requires prior theory in order to be designed.) In other words, applying RE perfectly means that no policy can ever be generalized since it is all context-dependent (a problem known as “external validity”). One understands why the approach that promotes field-work research and the use of economics to generalize its results is not credible among new development economists. Not only field work research is soft evidence, but generalization is quasi-impossible.

Now, does this mean that the experimental approach (especially RE) in economics should be rejected altogether? Well it all depends on the way it is being used. RE can help solve some technical questions in the field of development policy that cannot be easily solved on the basis of theory alone (Rodrik uses the example of insecticide-treated bed nets in the paper). In so far as learning from the ground is concerned, more than one approach may be necessary: this is what triangulation is about. The point Rodrik makes is that RE cannot be seen as the unique source of “evidence.” I broadly agree with his view, but he seriously underestimates in my view the need for theory to understand social phenomena.

Comments

[Roger Koppl]

I think you make a good point, Frederic. The MIT group has done some great stuff, including a paper I love: "Are Emily and Greg More Employable than Lakisha and Jamal?" So I am pretty much a fan. But that doesn't mean we give up on other sources of scientific knowledge or forget old problems such as theory dependence. Randomized trials in medicine have some shocking problems as revealed in a 2007 study by Berger, Matthews, and Grosch: http://smm.sagepub.com/cgi/content/abstract/0962280207080639v1

Another cautionary study comes from John P. A. Ioannidis:
http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=1182327&blobtype=pdf
That one was blogged on Marginal Revolution by Alex Taborrok on 2 Sept. 2005.

As you say, we have to triangulate.

[Rafe]

A lot of early RE methodology was developed in Ag Science to estimate the response to fertilizers. Interesting to note that Pete has recently appropriated (pirated?) the image of the farmer or gardener (in place of the engineer) to capture the ecological approach that is required to grow things like food, fibre, peace and freedom.

A quick scan of the Rodrik paper raises the question, why no reference to the massive literature on the diffusion of innovations? (Another field pioneered in Ag Science)http://www.amazon.com/review/RH2X7ZS99BQ8E/ref=cm_cr_rdp_perm

Some of his comment suggested that developers often go into new situations "cold" with minimal expectations to guide their activities. That makes it really hard for research to be cumulative.

[Rafe]

Because entrepreneurs are the innovators par excellence, the context and spread of innovations may be of interest to Austrians.

For the Table of Contents of the classic work on diffusion of innovations http://www.amazon.com/gp/reader/0029266718/ref=sib_dp_pt#reader-link