Question 17

Do these debates on RCTs not raise the question of models of behavior and theories of change – and how do the authors view this?

Lant: Yes, indeed. Indeed, I very much doubt the “theory of change” that a key constraint to development progress is the production of “rigorous” estimates of causal impacts of specific interventions. One example. In 1999 Deon Filmer and I published a paper “What do education production functions really show” that demonstrated that, unlike the prediction of “normative as positive” producer theory the marginal product per dollar of inputs into education were not equalized and that some inputs (particularly those that entered teacher utility functions directly) were over-used by orders of magnitude relative to inputs (like chalk or books) that did not. We suggested the best way to understand this was that politicians who controlled education budgets were not “making mistakes” or were not making these choices because there were not credible estimates of causal impacts but because they really weren’t allocating resources to maximize education outcomes at all, but were balancing that with political interests. Hence, if one were concerned with raising learning outcomes one had to mostly worry about the politics (both electoral and organizational within governments) of education.

Moreover, we argued that making ‘recommendations” to “policy makers” on the “normative as positive” model that “here are highly cost effective inputs” that, since you are maximizing learning you should adopt was just dumb and clearly at odds with the evidence (see “Is the economics of education useless, or worse”). But along came the randomistas with their bright shiny object of a method that answers narrow disciplinary questions and improves on evidence that was already mostly good enough about many questions and focused on better estimates of causal impacts using still the self-refuting “normative as positive” theory of change that what policy makers lack is “good evidence” and characterizing their choices as “mistakes” that can be fixed with a study that is persuasive. So now we have wasted 20 years. The new estimates reproduce exactly the findings of the estimates of 20 years ago: e.g. there is lots of heterogeneity across countries, most “teacher utility raising” expenditures are not cost effective, there are actions that are orders of magnitude more cost-effective than the existing expenditures but which are not adopted (and, even when the evidence is there they cannot be, e.g. contract teachers in Kenya (Bold et al).

So, it is not like we development academics and practitioners have “learned” what is really needed to be known to improve education systems from the RCT experience: We knew fire was hot. All we really learned is that academics are willing to place themselves and what is their comparative advantage (the generation of “rigorous” evidence) in a privileged position, even when the evidence is obviously against that.

Editors: This is right, and this is what we mention in the introduction:

The opposing views surrounding RCTs are also based on different notions of development, poverty and, more broadly, politics, seen as a conception of the world in which we live and which we endeavour to attain. Is the world an aggregate of individuals seeking to be independent or is it a complex system made up of dialectics, multiple interactions, retroactions and systemic effects between social beings who are interdependent and wish to remain so? Should we see the “causes of poverty as a lack or want of relevant variables or as an active process of impoverishment or perpetuation of poverty” (Shaffer 2015: 154)? A “want-based” understanding of the causation of poverty calls for policies of “difference-making” wants (to cope with deficits in health, education, nutrition, water/sanitation, credit, and so forth); and understanding the impacts of such policies requires a counterfactual to be able to isolate the difference and attribute the impact to the policy in question. By contrast, a conception of the causation of poverty in terms of processes and social relations calls for macroeconomic and structural policies (exchange rate, capital control policies, social protection measures, and so forth); and understanding the impact of these measures requires a “mechanism-based approach” that explores the diversity and complexity of the causal processes that generate the impact (Shaffer 2015).