I’ve spent a lot of time arguing that government should be more evidence based, and that wherever possible, we should do randomised trials to find out which policy intervention works best. We often have no idea whether the things we do in government actually work or not, and achieve their stated goals. This is a disaster.
So, with my grown up hat on, here’s a Cabinet Office paper I co-wrote with some government people on exactly this topic. We explain why randomised trials of policy are so powerful; we explain exactly how to do them; and we explain how to identify a meaningful policy question that can be explored cheaply in a good quality trial.
We also show that policy people need to have a little humility, and accept that they don’t necessarily know if their great new idea really will achieve its stated objectives. We do this using examples of policies which should have been great in principle, but turned out to be actively harmful when they were finally tested.
Finally, we address – and demolish – the spurious objections that people often raise against doing trials of policy (like: “surely it’s unfair to withold a new intervention from half the people in your trial?”).
Trials are used widely in medicine, in business, in international development, and even in web design. The barriers to using them in UK policy are more cultural than practical, and this document will hopefully be a small part of a bigger battle to get better evidence into government.
The paper also describes, for the first time, several fun examples of trials that have been conducted in UK government over just the past year, reporting both positive and negative findings. These trials all test small, modest changes in policy – and ones that are ideologically uncontroversial – because this is the best way to get trials adopted more widely.
What’s more, they’ve all been run by a small group of very smart people running out of the Cabinet Office, who have quietly set up what is effectively a randomised trials unit in government. There are quite a few people in the civil service who seem to be on board for all this, so it will be interesting to see if the idea catches on.
Anyway, I think (I hope!) that the paper is readable and straightforward, like the Ladybird Book of Randomised Policy Trials, and I really hope you’ll enjoy reading it. It’s a good primer on basic research methods, and on how to do a trial properly in any domain, with clear examples taken from the real world of medicine, business, teaching, job centres, web design, and more. The people I wrote it with are a mix of supersmart civil servant policy wonks and academics.
To be clear: this is a long read, and there’s a ton of material in these 30 pages. It’s free to download here:
(I read PDFs like this with iAnnotate on iPad, or more commonly on my Kindle: email the file to your Kindle’s email address as an attachment with “convert” in the subject line and PDFs come out very nicely).
Sorry to be absent, by the way: I’ve been working. There’s some fun stuff coming after the summer, until then you can find me on twitter, on my scrappy other blog, here’s my TED talk if you’re procrastinating, but most importantly of all: read our paper, and tell your policy wonk friends about it.