Can Randomized Experiments Capture Counter-factuals in Policy Evaluations?

|Peter Boettke|

On my flight to Indianapolis today I listened (and watched) a video-podcast of Esther Duflo speaking to MIT alumni about the work of the Poverty Action Lab.  I downloaded the talk last night from I-Tunes U, which if you haven't done so you should visit.  Here is a link to a Technology Review article discussing Professor Duflo's work that makes many of the same points.

Anyway, during her talk she stressed how difficult it is to examine the problem of poverty alleviation and development in policy evaluation.  As she says the question she wants to ask is difficult to measure (did that child learn because they had that textbook?; what didn't the child learn because they didn't use that textbook?; etc.)  The interesting questions are counter-factuals.  But her pioneering methodology in development economics is designed to get at these questions.

I am intrigued by these new methods of empirical economics -- experiments, computer simulations, natural experiments, etc.  What do you think from a methodological perspective of these new methods?  Do they accomplish what they suggest they do?

Comments

[Jim Rose]

The usual suspects for natural experiments are countries with the same cultures, peoples and similar resources who are suddenly partitioned for arbitrary reasons.

These would be East and West Germany, North and South Korea and Taiwan and China. In each case, the socialist solution was left for dead by capitalism.

In the early 1960s, Joan Robinson got into a dispute with a colleague about the great economic success of Korea. It was a confusing debate until a listener realised that Robinson was talking about North Korea and the other about the South.

Soon after, there was a coup. The new South Korean dictator knew nothing of economics, but according to Tullock, he knew he was surrounded by the old dictator’s cronies so he fired them all. Before a new lot of cronies got properly in place, the economy boomed.

Taiwan and Cuba are also natural experiments. Both are threatened with invasion and trade sanctions by large bellicose superpower neighbours. Cuba is a dump. Taiwan changed from a rural backwater to a rich country.

East Germany was the best run of the socialist economies, but as Kennedy said at the Brandenburg Gate in his greatest speech “Freedom has many difficulties and democracy is not perfect, but we have never had to put a wall up to keep our people in”.

Joan Robinson developed the greatest ever put down of the socialist solution. She pointed out that in 1848, the battle cry of the communist manifesto had some relevance – rise up ye workers, rise up, for you have nothing to lose but your chains! She said in 1942 that this battle cry would have to be amended to 'rise up ye workers, rise up, for you have nothing to lose but your suburban home and your motor car!' Is that a natural experiment too?

I am not too sure how Mises would view natural experiments. His view of theory and history is that the correctness of a theory can be ascertained without the aid of experience. Natural experiments make no sense without a correct theory to tell us what and how to measure and what to make of our discoveries.

[Student]

Randomized experiments are a great step forward, but they don't solve all the problems. Heckman actually talked about this in his Nobel interview from a few years ago. There are still non-experimental corrections that may be needed to analyze data generated from experiments given that many implementation issues can arise in the field.

Furthermore, only average *effects* of particular programs can be recovered from the data. For evaluating counterfactuals -- i.e. performing simulations -- writing down models and recovering *primitives* is necessary. It seems to me that future empirical economists will increasingly use this relatively clean source of variation we can get from experiments to aid in identification of structural models. While this sounds like old-school Cowles Commission kind of stuff and may induce a knee-jerk reaction by you to reject it, you shouldn't. The newer structural researchers are not looking to advance a command-and-control agenda. Perhaps the greatest contribution they have made collectively is showing how *difficult* it is to obtain useful quantities from the data. Reduced-form methods are generally not enough if one is interested in more than the sign of some coefficient. So if one believes empirics matter for policy, then some constraints on policy are implied by their insights....

If you're truly interested in learning more, Heckman is the way to go. He's not just a technician. He cares about the history of the field and philosophical issues of causality.

[Rafe Champion]

William Easterl's Aidwatch blog has a category on evaluation and trials, including RTCs.
http://aidwatchers.com/category/aid-evaluation/

Apperently there is huge debate in aid circles about the value of RTCs.

[Troy Camplin]

Any data would of course involve more than a single subject. You have to study how something affects serveral in order to see and understand the patterns that emerge. At the same time, one does have to deal with individual differences. Fortunately individuals are made up of behaviors shared with others, so one can take a network approach and still do this as science.

[Roger Koppl]

Let a hundred flowers bloom.

If I may channel Sir Karl, we cannot predict what new knowledge these techniques will help develop, so we have to just let each researcher gamble on his or her chosen techniques and see how it all unfolds.

Maybe that's all one can say. But maybe a few general points are relevant as well. These new(ish) methods can do some things and not others, so it is "really" a question of how much they can do. It is hard to see how they could eliminate the need for praxeology, even if they might suggest some helpful tweaks to it. If we all agree that we cannot get along with a "physical language" and so on, if we all agree on methodological dualism along more or less Hayekian lines, then we will make use of our understanding of individual choice to imaginatively reconstruct the more complex overall order unintendedly generated by individual choices. We are pretty much stuck with the "basic economic reasoning" Bruce Caldwell has so rightly defended. Thus, it would seem realistically that these techniques won't make much change in "basic economic reasoning." But I don't see any particular reason some of them might not make big changes in everything outside basic economic reasoning. If we are more disciplined about choosing between plausible empirical stories, we may get more convergence on the consequences of different policies, as Duflo's work illustrates.

[Jared Barton]

"Do they accomplish what they suggest they do?"

If by "they" you mean randomized experiments (especially randomized field experiments), then the answer is "yes". And for a more nuanced version of "yes" addressing counterfactuals specifically, it seems that Student has already said more and said it better than I would have.

[It seems to me that natural experiments, in contrast to the designed kind, are a lot like using instrumental variable regression techniques: sometimes you might find a really good natural shock or IV, but often you have something a little iffy that's very sensitive to the specification. But perhaps that, too, is better than nothing at all...]

[fundamentalist]

While controlled experiments are interesting, it seems to me that the results are trivial due to the nature of controlled experiments. And I suspect that the interest in them is due to a desire to be more like the natural sciences and use inductive reasoning to build theories. I don't think that will be possible partly because the results tend to be trivial and I think it's misguided.

[Pedro]

I watched Duflo's TED talk and found it very interesting. As a person interested in the Austrian school of economics, I was intrigued (I'm not an economist).

Most of what she says makes sense. Superficially, her approach seems to contradict the views of Hayek and Mises on the epistemology of economics. However, I don't think this is the case.

In the first place, I don't think the result of the immunization-lentils experiment contradicts economic theory, like the article says. It's perfectly natural for people to react to an additional incentive. That the reaction was bigger than expected doesn't contradict economics at all. At any rate, it doesn't contradict Austrian economics, because economic theory cannot make exact quantitative predictions. Also, people's reactions depend on their objectives. Therefore, the strength of their reaction to the lentils incentive depends on how much they value lentils. I wasn't surprised at all by the result of the experiment.

In my view, these experiments are valuable, because their results can enable people to determine the best ways to relieve poverty. That's great. But no economic theory can be deduced from them. Their results are not generalizable. They apply only to the context in which they were performed. E.g, while giving lentils to people in India may yield the best results in immunizing as many children as possible, that might not be true in other places. I'm not saying this is a drawback of these experiments. It's simply something that is out of their scope. I think the purpose of these experiments is not to develop general economic theories, but to determine, in specific contexts, which poverty relief measures work best.

That's why I think this doesn't contradict Hayek's and Mises' views on epistemology. In my opinion, performing these experiments is not less valid from their point of view than recording athlete statistics in order to manage a sports team. In my view, what Duflo is doing doesn't qualify as economics in the sense to which Mises and Hayek were referring when they wrote about epistemology.

Also, I think these experiments have nothing to do with government policy. They don't say anything about whether governments should send relief to Africa or not. They just say: if you have x dollars and you want to help people in place y so that problem z is relieved, the best approach is p. For that, these technique seems to be appropriate. For building general economic theories, no.

[Jim Rose]

The 1942 essay by Mises on the social sciences and the natural sciences is instructive on the role of experiments in economics. Experiments are still complex phenomena all open to many interpretations based on various theoretical standpoints.

Consider the examples I gave of natural experiments such as East and West Germany, North and South Korea, Taiwan and China, and Taiwan and Cuba.

The total failure of the socialist solution in these countries, for example, could be explained by Trotskyite writings.

East Germany, North Korea and Cuba are degenerate workers states; manifestations of state capitalism lorded over by a new class of ex-comrades corrupted by power. A purer form of socialism that was run by a different left-wing sect all with finer motivations and greater insight to those who were in charge could have made all the differences!?

Public choice explanations of special interest capture and the soviet state as a manifestation of mercantilism as per Boettke, Anderson and Tollison come to mind too.

The failure of the socialist solution and the potential of amended versions could be even better understood if the left had developed its own constitutional political economy.

Just as there seem to be no left-wing Austrian economists, are their any left wing constitutional political economists? The closest would be Jon Elster and John Roemer? There may be others.

My point is that the meaning of controlled and randomised experiments depends on your priors and the state of development of economic and political theory. They can validate or throw doubt on what you believe, depending on what you believe, and how quickly you can account for anomalies, as long as there are not too many anomalies.

[Student]

Jim Rose, why are your only examples "natural experiment" examples? Unless you provide examples from "field experiments," it seems your comment only applies to the former. Perhaps related to your comment, the experimentalists are generally interested in micro-level questions, and not the macro-level (e.g. country-level) questions. A big reason, obviously, is feasibility. Furthermore, my impression is that there is much less to interpret in field experiments as they are implemented right now, and the matters that require interpretation are on a much more micro level. I don't think Jim Rose's comment has much pertinence to the field experiments the Poverty Lab and similar organizations/researchers perform. I don't think that your comments would stand given the professional debates over what these experimental papers can tell us. One benefit of the experimental "revolution" is that alternate theories can be accounted for in the experimental design, or another experiment can be designed to account for the shortcomings or new issues brought up by a previous experiment.

I'm with Jared Barton on the natural experiments/IV thing. The problems with naive IV are well disseminated now (the structural guys pointed this out a while ago though the rest of the profession has absorbed some of their dissatisfaction only recently), but sometimes some evidence is better than no evidence at all, especially for those of us who would like to have more than introspection to discipline our reasoning. If anything, there is wisdom in knowing just how bad our estimates may be, as I alluded to in my earlier comment.

[Student]

Tyler's post on this is informative as well: http://www.marginalrevolution.com/marginalrevolution/2010/05/how-good-are-rcts.html

[Peter Boettke]

student --- thanks for the pointer to Tyler's discussion. I very much like the Acemoglu paper that Tyler links to as well. I plan on spending the first 3 weeks of my comparative political economy course next spring discussing these issues. Also Leeson, Coyne and I have a paper on what we call comparative historical political economy which has been sitting at the Cambridge Journal for over a year due to nagging issues and I think we might rethink some of our arguments to frame it in light of these discussion and place our narrative style comparative work in juxtaposition to either Duflo on the one hand and Acemoglu on the other. But that is just off the top of my head this morning.

[gucci shoes]

The prizes of lifestyle are with the finish of every journey, not close the starting; I will persist until finally I succeed.

[topills.com review]

That has not been stable for the last several decades - it has steadily declined and it has recently gone negative.