Anthony Zacharzewski
Director, the Democratic Society
How do you pick relevant evidence for your policy?
Here’s an older post from the LSE, which struck a chord with something I’m mulling over at the moment. What is evidence-based policy making without some selection of the evidence you use. By extension – not in the LSE article – are some open policy making contributions more relevant than others?
[Evidence-based policy starts] by distinguishing what is true from what is relevant. If God provided an encyclopaedia of all the propositions about the world that are true, whatever that might mean, it would not help until you had decided which of these facts is relevant to your decision. The fact that it worked there is not straightforwardly relevant to the conclusion that it will work here. And that is not just to do with a fact having been established by an RCT.
Any method – whether RCTs or econometrics or more generally drawing on our shared stock of facts which we accept as true from science, or experience, or from just using our eyes – which establishes a fact does no more than that. It does not answer the next question, why do we believe that fact is relevant evidence for believing that it will work here?
This insight – that relevance matters as much as truth – is not merely theoretical. There are too many examples where doing here what worked there, failed because if a failure to see what was relevant to success beyond the fact that the intervention had worked somewhere else.
Small class sizes worked to raise reading scores in Tennessee, but the same policy failed in California. As often, with hindsight the explanation is simple. If, as they did in California, you introduce smaller classes on a large scale very quickly, and of course you have to have lots more teachers, you end up with a lower average quality of teacher, because you have to recruit less experienced or well trained people. And of course you have to have many more class rooms, so you take space for reading classes from other activities which themselves contribute to the flourishing of pupils and hence to their reading ability. These and other changes mean that the potential of lower classes never gets realised. Even if you are confident that smaller class sizes played a determining causal role in Tennessee, you have to think whether it can do so in California. And that means thinking about what helping factors are needed and whether they will be present. The true fact that it worked in Tennessee does not of itself tell you how to think about that. It does not tell you what helping factors have to be present to allow smaller classes to play their causal role. Let alone how you find out whether those factors will be present.
One Response to How do you pick relevant evidence for your policy?
Pingback: How do you pick relevant evidence for your policy? | The Democratic Society