Question 10

Coordinator sub-sub-post 1 Comment

The following were the results from the polls during this book launch event. How do they compare with earlier launch events? What are the “correct” responses?

Lant: I think the right answer for Poll 2 is “very little” of what is most important can be evaluated via RCTs, as what is most important are system and structural transformations and RCTs can pretty much only be “powered up” for individualizable “treatments.” Then, I think the answer is also “very little” because even in the domains that are amenable to RCTs we already see there is too little “external validity” to the studies are not adding up to anything.

With regard to Poll 3, for COVID-19 the time scale of RCTs (outside of the medical field, like vaccines) is too long to make a difference to a short-term crisis.

Editors: We agree with Lant that the right answer for Poll 2 is “very little”. As we write in the introduction to the volume:

In the terms of reference for a study commissioned on the subject, a group of DFID managers estimated that less than 5 percent of development interventions are suitable for RCTs (DFID 2012). Although this figure is not to be taken literally, there is no doubt that experimental methods are not suitable to evaluate the impacts of the vast majority of development policies.

We did this poll twice so it is impossible to draw any conclusions. But it is interesting to observe the same trend: a certain distrust of RCTs (probably linked to the type of audience attracted by the debate), but then when it comes to the COVID-19 pandemic, trust returns, which is contradictory. Not only is there no reason for RCTs to do better in a pandemic, but there is even good reason to believe that they will do worse, for several reasons that we briefly outlined in a Working Paper that can be found here. Here is the excerpt:

The magnitude of the shock is a game changer. The problematic transposition of local results obtained in normal times has no chance to hold in times of pandemic, where behaviors are upset. The transmission of the disease does not depend only on individual but also collective behavior, synonymous with generalized externalities the method is unable to capture. RCTs serve a knowledge generation strategy where, rather than tackling complex questions as a whole, they are into a series of small questions, the only ones the method is able to address. This micro, “small step” approach is not suited to the need for macro responses, imperative scaling up, national or regional levels, with systematic general equilibrium effects. These are massive emergency measures of partial unemployment, cash transfers, subsidies at decentralized administration, guarantees and lines of credit …, facing fiscal, monetary and budgetary policies to finance them; mobilizing all the resources (human and financial) of the administration. These kinds of interventions cannot be evaluated through RCTs. In addition, the urgency of implementing measures compared to the long time required to implement the method, as well as the ethical questions posed by the exclusion of populations drawn at random for the purposes of randomization are all additional reasons for ruling out experiments. Governments around the world have made no mistake about it.
While for the development of a vaccine, clinical trials have multiplied, to our knowledge no public policy in the field of nonpharmaceutical interventions and even less in the economic and social field has been based on the results of RCTs conducted so far or in progress. Even in the medical and epidemiological field, as soon as we look at interventions involving human behavior, the clinical methodshows its limits. A systematic review used by WHO of randomized experiments concluded that masks were ineffective for non-caregivers, even pointing to their negative effects (Jefferson et al., 2020). At the same time, another systematic review published by The Lancet, which relied mainly on observational studies, concluded that masks were highly effective for the general population (Chu et al., 2020). Does this mean that we should have given up on masks, or more judiciously applied the precautionary principle, while waiting for scientific studies to settle the question?
The ineffectiveness of social experiments does not mean that the randomistas have given up, on the contrary. Reiterating old studies, RCT proponents offer tips for improving insurance enrolment or online learning. They embark on massive campaigns (25 million people targeted in West Bengal) to test different incentive techniques to respect social distancing measures (Banerjee et al., 2020). One of these techniques is to stage a “celebrity” (in this case Abijit Banerjee himself). They suggest that governments invest massively in two areas, that of cash transfers and digital to effectively distribute these transfers to target populations. This is not unnecessary.
But on the one hand, many others also argue for this type of interventions and the method of randomizedtrials does not give them any particular legitimacy. On the other hand, here again, the structural dimension of these measures, both in their large-scale implementation and in their short, medium and long-term effects, remains intact. Yet it is essential. As indicated previously, what fiscal, monetary and budgetary policies are making it possible to finance these monetary transfer policies? What is the technical and above all the regulatory infrastructure guaranteeing a democratic use of digital finance? While it can deliver money faster to vulnerable populations, it can also lead to control and invasion of privacy. More than ever, RCTs must be put in their rightful place: one method among others without exceptional status, possibly to be combined with others, and ill-suited to respond to the challenge caused by the pandemic. Even though critical voices are heard, many academic and other circles have yet to realize their limits. RCTs, boosted by the Nobel Prize award and the redoubled efforts of their promoters to extol its merits, continue to drain millions of euros, often to the detriment of other approaches that are at least as rigorous and useful.
This contradictory sentiment (distrust of RCTs but confidence in the ability of RCTs to address the poverty problems associated with the pandemic) probably raises something more global: the need for all of us to believe in a better future and in miraculous solutions. And the strength of randomistas is that they promise miracles, capable of blinding even the most knowledgeable people.