Question 5

How appropriate is the use of RCTs in the context of LICs (or LMICs), given the practical challenges that accompany their execution in low-resource, low-capacity and often fragile institutional and societal contexts? What considerations are then most important?

Lant: I am writing a new paper about RCTs and the question of external validity. Here is a passage from that paper:

In a similar vein one could argue that the field of economics to which RCTs are least likely to contribute is development economics. RCTs seem most likely to contribute to effective policy and program design in countries that are: (a) large, so that the cost of the RCT can be amortized over large programs, (b) sufficiently homogenous, so that results from one or a few study sites are likely to be reasonably applicable (that is, have at least a fighting chance at “external validity”), (c) have high governmental capability, so that whatever program designs are discovered to be effective can be replicated with fidelity, (d) with reasonably high average/median income, so that the key constraints to policy/program effectiveness are likely to be program design and not public or private sector budget constraints, (e) reasonably high average/median income so that redistribution is a plausible channel for improvement (in many developing countries if redistribution produced exactly equal incomes for everyone then everyone would still be in poverty by moderate global poverty lines). This sounds a lot like the USA, maybe some larger countries of Europe (UK, Germany, France), Japan. It doesn’t sound at all like the most pressing cases of development, the “fragile” states like Haiti or South Sudan or Somalia or Afghanistan.

There has never been any evidence the “knowledge” of the type RCTs generate is a key constraint to more effective development. To me it is much more likely that “state capability” (or an organizational capability for implementation type) and political interest/will to address the problem are more likely constraints.

Here is a very recent paper (and blog on that paper) that did an RCT of a scaled transplantation of a “school improvement plan” process used in the UK. It shows the program was implemented in a “paper’ way but had zero impact on anything real. So I have been arguing for a “capability first” approach.

Editors: In addition to Lant’s arguments, there is the ethical argument, which arises even more acutely in contexts where there is a strong asymmetry of power between researchers and populations, which is obviously the case in these countries. As we write in the introduction:

Ethical safeguards are all the more necessary in the Southern countries. Firstly, not informing participants (informed consent principle), if not deliberately misinforming human subjects to ensure a clean identification strategy, is at odds with the principle of ownership promoted by the development policies. Secondly, participants are generally vulnerable individuals, both economically (poor) and politically (voiceless), on whom it is easier to impose the trial, if not deliberately mislead. This asymmetry is especially strong in that the surveys are more often than not tantamount to lifesize laboratory games supervised by young students and research assistants from Northern universities. We also need to look into the choice of these populations, especially when testing a behavioural hypothesis or a theory put forward by certain RCT proponents (Banerjee and Duflo 2011; see Morduch, Chapter 3). Save advancing that the poor in Southern countries have specific rationality, the arguments of lower cost and less capacity to refuse to take part (a recurring problem with RCTs in Northern countries) due to a lack of knowledge of their rights and lopsided balances of power (including with respect to the experimenters) appear to be credible explanations (Patnaik, Interviews, this volume; see also Teele 2014), as has already been observed in the “offshoring” of medical clinical trials (Petryna 2007). Without going so far as to call for a “moratorium on experimentation” in the South (Hoffmann 2020), the issue should be at least addressed in priority.