However, their use in this field is limited by ethical and practical concerns (How do you justify allocating an income maximisation intervention to only some of those who need it? How do you double or in many cases even single blind a social policy? It can be difficult to agree what the outcomes of interest are, and then to develop means of measuring these effectively etc etc) In many cases, researchers have agreed to adopt a next best approach, by using before and after designs of a single cohort (ie: just the intervention group.

Anyway, don’t have access to any relevant links just now, but I know that the Campbell Collaboration have been involed in RCTs in a number of areas of soical policy (crime and education for example).

http://ann.sagepub.com/content/vol589/issue1/

I’ve gotta admit I study politics and work as a research assistant for an international economist and I’ve never come across a RCT in my day to day studies / work. There’s also a general attitude amongst academics that they’re “impossible” or nearly so in political science, and woe betide the academic or student who drops the word “experiment” into a paper. Political scientists don’t experiment – they compare different policy options.

As for “evidence based policy making” this usually seems to consist of commissioning a thinktank or academic to do a quick and dirty secondary literature review over the course of a couple of weeks, then releasing it with a misleading press statement if it conforms vaguely to the decision you’ve already taken. If it doesn’t it might surface in some form in a institutes working paper collection.

Ah Politics!

http://www.badscience.net/wp-content/uploads/2007/07/170.pdf

can you post the link to the special issue?

Using Random Allocation to Evaluate Social Interventions: Three Recent U.K. Examples

I’ve sent you a PDF.

that’s great, i’m always keen to see interesting new stuff, can you give us 3 recent large, well conducted RCTs on social policy from the UK?

i have spent a lot of time browsing campbell, in fact i referenced it above, here

http://www.badscience.net/?p=462#comment-14889

its interesting, but its stuffed full of feasibility discussions, protocols for possible studies etc, there are very few RCTs, they are generally poorly performed and very small. the reality is there is a lack of culture of science in social policy, govt, the judiciary, etc. this means that people dont routinely think of trials research in the same way that eg docs and nurses do.

However there is still a need to do better research. In particular we need to get better at setting out and measuring outcomes from social policies.

The real ‘problem’ is not just of ‘ da meeja’, but of democracy and the need for public policy decisions to be legitimate and accountable. Criminal justice is a good case, as mentioned above, where there there are conflicting objectives, other than maximising the Quality-Adjusted Life Years of a clinical trial.

Public policy theorists sometimes try to frame these conflicts using public value theory.

To turn the tables on medical research, are there lessons that you could learn from social research? For instance, do you take economic and social costs into account when considering the risk-benefit ratio for new interventions? Do you think about what patients and the public value in medicine?

Anyway, here is the Savings Gateway pilot mentioned at ippr that used random digit dialling to recruit participants. I haven’t read it but the authors Ipsos Mori and Institute for Fiscal Studies have good reputations in the public policy field.

http://www.hm-treasury.gov.uk/media/7/0/savings_gateway_evaluation_report.pdf

http://www.guardian.co.uk/science/story/0,,2130777,00.html

Now, I have to admit it’s not exactly clear to me what’s happening to the Science and technology committee, but there’s surely not enough evidence in policy making as it is.

And surely there’s the rub, at least in part.

In medicine, we tend to agree what outcomes are important – death, is pretty important and we’ve developed some ways of measuring thing like quality of life and disability adjusted life years. We mostly know to criticise drug companies for choosing outcomes they know they can reach (which of you are interested in you LDL-C at 6 weeks after starting your statin, when you could have one that actually stops you dying as early as you might?)

The Daily Mail would presumably like an outcome of “whatever would punish those crims the most” inviting the prospect of a fun questionnaire survey (“How would you rate the enjoyment of you time in jail? 1. Hugely enjoyable – would love to come back to 5. Hated it – let me out of here”). Most of us reading this, it seems, would like an outcome of “not going back to prison”, “getting a job” or something.

I’m not saying that getting at these outcomes is impossible, but that it probably needs a lot of discussion and thinking about.

Well, yes and no. I don’t know if you were thinking of this, but I’m reminded strongly of Robert Martinson’s 1974 review of American offender rehabilitation programmes, “What works?”. Martinson concluded that there was no single approach which had been shown to work consistently – which meant that, if the government wanted him to recommend a rehabilitation package that could be rolled out nationally, he couldn’t oblige. This was misread as “nothing works” (the research Martinson summarised actually suggested that different things worked for different people in different situations; as Will points out, this has its own problems, but at least it suggests somewhere to start from). What’s worse, many people drew the conclusion that if nothing [in the way of rehabilitation] works, the only answer was to imprison more people for longer. In fact Martinson had taken it as read that prison didn’t work; he actually commended some forms of rehabilitation on the grounds that, although they didn’t reform ex-offenders any more effectively than prison, they didn’t do any worse – and they cost a lot less.

So, no, most social and criminal policy measures don’t work for everyone all the time – which is a problem when the Home Office is running a conveyor-belt of new measures designed to be rolled out to everyone, all the time, starting yesterday. But I’d be wary of saying that most measures *don’t work* – if only because of who might be listening.