Saturday November 11, 2006
I don’t know if you’ve ever tried using the Freedom of Information Act: it’s an excellent trouble making tool, and you do feel quite James Bond, but the act has its flaws. One being that if you ask for too much, as one lone, obsessive, disproportionately pedantic science columnist, they turn you down on grounds of cost. Quite spuriously and unfairly, to my mind. So now I’m offering a kind of skills swap: I’ll teach you all how to do an FoI request (it’s easy) if you help me get a bunch of data.
Contain your boredom and exhaustion, because it’s the Durham fish oil pill people again, but think of this more as an experiment in what happens when you just won’t let go. You’ll remember that this story bore the cardinal hallmark of quackery: they claimed they had proven that expensive Equazen fish oil pills improved school performance in various trials, and were eager to sing about this in the media, but when I approached to ask about the science, with a loaded postgraduate medical qualification in my back pocket, they shut up shop and fled. Dr Madeleine Portwood, eager to appear on Channel 4 promoting her “trials” and bamboozling non-science journalists, did not return my calls, or my emails, and the press office – bless them – weren’t much help. The durhamtrial.org “results” website is a superficially plausible but ultimately uninformative sham.
So I made an FoI request to get the data, and they refused to give me anything. The refusal, I ought to say, took them the full statutory 4 weeks (nice!). And why are they turning me down? “It is estimated that it would take 30 hours to fully respond to this second request, which would cost £750 when calculated using the statutory rate of £25 per hour.”
This is a joke: all I asked for, essentially, was the basic information you’d find in any write up of any scientific experiment: the trial methods in sufficient detail to decide if they were a “fair test” of the treatment, and the results in full. Oh, and a few leaflets. You can witness the full, formal anality of my FoI request online: I just used the CONSORT guidelines as a template, since they’re the gold standard for writing up any trial. If Durham and Dr Portwood really are in a position to bang on about their positive results, they should have this information at the tips of their fingers.
But no. And not only did they turn me down, they also – rather censoriously to my mind – pointed out that any cleverdick “multiple small applications” shenanigans from me would be totted up and rejected. And this is why, gentle reader, I need your help.
Some of it was bland stuff: copies of blank consent forms, leaflets for parents, the eligibility criteria for participants, the trial locations, that stuff. Some of it is more nerdy. “Nullius In Verba”, as it says on the Royal Society’s headed notepaper: “on the word of no-one”. That’s what science is about. It’s not that I think they’re lying, it’s that the devil is in the detail. Science isn’t about telling a journalist you’ve proved something in your brilliant trials, and then seeing your name in the paper, it’s about methods and results.
Did they deviate from the protocol during the studies, I asked, for example, and if so, why? How were the studies blinded? Bread and butter stuff. Were there any analyses they did, but which weren’t reported because they were, well, negative? As we all know, the more different analyses you do on your data – it’s called “data dredging” in the trade – the more likely you are to stumble on a positive result, simply by chance. Likewise, I wanted the protocols for studies that are not yet completed, because it’s always a nice idea to know the protocol before people start their study, otherwise they might change it halfway through, which can stop the trial from being a fair test.
This is bog standard science. That’s the information I wanted. It’s what I still want, and I intend to get it, even if I have to break in to Durham Town Hall and take hostages. Before we get that far, I need your help, and there are full instructions for your FoI requests below. Durham Council, Madeleine Portwood, et al, just so that you know: I very simply will not let this lie.
How To Make Your Freedom Of Information Act Request:
The FoI Act is an excellent way to observe whether public figures and bodies are behaving in a generally appropriate and open fashion. The mechanics of making a request are unspectacular. Most public bodies will accept an informal email. Peter Dinsdale is the Freedom of Information / Data Protection Coordinator at Durham Council, and he has said to me in the past that he would accept FoI requests by email. His email address email@example.com and he is very nice and helpful. Put “Freedom of Information Request” in the subject line. If they change their mind and insist on you using their tedious FoI form then you can find that here: they have 20 working days to reply to your request (or not), and if you're feeling ubergeeky you can read more about FoI here.
The trick with requests is to be very specific about the information you require. I lifted the list of what I wanted, essentially, from the CONSORT guidelines: they are an internationally respected set of standards for the information that should be presented for a clinical trial, and they represent the basic things you would need to know about a study to assess its quality and results. There is more on the CONSORT project and its history here, and the CONSORT checklist is here.
Pasted below is a copy of the request I sent to Peter Dinsdale which was rejected, on the grounds that it was asking for too much. I suggest that each person requests just 2 or 3 sections each, going through them in order to avoid confusion, and then posts what they requested, and ideally the full email request they sent, in the comments below, so everyone can see how far we've got and what is left to be requested. Remember, for each request, to say that you want the information for each and every one of the many different trials or studies into omega-3 fish oils that have been performed in children by Durham employees in the past 5 years. God knows how many there really are.
Good luck, have fun, and don’t forget to post the replies you get from Durham in the coming weeks. Don't worry about doubling up with what other people have requested, it's better to be belt and braces. And naturally if you prefer you can also just email me your requests and responses without posting them below. Thanks again and good luck!
Dear Mr Dinsdale
many thanks for your assistance with my FoI request, which I made two weeks ago.
As I have unexpectedly encountered difficulty exploring the details of the trials and studies performed by Durham I would like to make a further FoI request.
As you may be aware, Durham Council's representatives (primarily Madeleine Portwood but also Dave Ford and the press office) have been reporting to the media since 2002 that they have performed numerous studies on the efficacy on omega-3 fatty acids on various outcomes in children and adults.
To the best of my knowledge these have been performed in pre-school children, primary school children, and secondary school children in Durham, students in Durham and Edinburgh, schools in Middlesborough, schools in Sunderland, and possibly elsewhere. As per my previous FoI request I would be grateful for a comprehensive list of studies both completed and ongoing, regardless of whether they had positive or negative results, including studies that were not completed.
For each of these studies I would like details of who was involved, what intervention was given, what was measured, what the results were, etc, further details are below. The positive results of these studies have been discussed extensively in the public domain and the media by Durham staff (going back several years) but I have been unable to obtain details that would enable me to assess these claims. Generally when researchers go to the media to promote positive results of completed studies, since the studies have been completed, and the researchers have been able to appraise the results, they are willing and able to explain the technical aspects of those studies.
The specific details of what I need to know – which are the bare minimum any scientist would expect in order to appraise a study – are as follows below. I would be grateful if you could give me this for each of the studies separately.
I'd be grateful if you could also confirm receipt of this email.
Dr Ben Goldacre
119 Farringdon Road
Fax: 020 7117 3593
For studies that have been performed and discussed:
Eligibility criteria for participants and the settings and locations where the data were collected.
Precise details of the interventions for each group and how and when they were actually administered.
Specific objectives and hypotheses.
Clearly defined primary and secondary outcome measures and, when applicable, any methods used to enhance the quality of measurements (e.g., multiple observations, training of assessors).
What the sample size was, how sample size was determined and, when applicable, explanation of any interim analyses and stopping rules.
Were the participants randomly allocated to different treatments? Method used to generate the random allocation sequence, including details of any restrictions (e.g., blocking, stratification)
Method used to implement the random allocation sequence (e.g., numbered containers or central telephone), clarifying whether the sequence was concealed until interventions were assigned.
Who generated the allocation sequence, who enrolled participants, and who assigned participants to their groups.
Whether or not participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment.
How the success of blinding was evaluated.
Statistical methods used to compare groups for primary outcome(s); Methods for additional analyses, such as subgroup analyses and adjusted analyses.
Flow of participants through each stage: specifically, for each group what were the numbers of participants randomly assigned, receiving the intended treatment, completing the study protocol, and analyzed for the primary outcome.
Describe protocol deviations from study as planned, together with reasons.
Dates defining the periods of recruitment and follow-up.
Baseline demographic and clinical characteristics of each group. Numbers analyzed
Number of participants (denominator) in each group included in each analysis and whether the analysis was by "intention-to-treat". Could you please state the results in absolute numbers when feasible (e.g., 10/20, not 50%).
Outcomes and estimation
For each primary and secondary outcome, a summary of results for each group, and the estimated effect size and its precision (e.g., including a 95% confidence interval).
Address multiplicity by reporting any other analyses performed, including subgroup analyses and adjusted analyses, indicating which of those were pre-specified and which were exploratory.
All important adverse events or side effects in each intervention group.
Interpretation of the results, taking into account study hypotheses, sources of potential bias or imprecision and the dangers associated with multiplicity of analyses and outcomes.
Generalizability (external validity) of the trial findings.
General interpretation of the results in the context of current evidence.
For studies not yet completed
I would be grateful if you could let me know the protocol for studies that are in progress or not yet completed.
Material relating to each study
I would be grateful if you could give me, for each study, the participant information leaflet, any material given to parents/teachers regarding their participation in the study, and any paperwork relating to how consent to participate was obtained, eg blank consent forms.
And if that all sounds like too much effort…
… then I'll also be appealing their rejection, and pursuing other avenues behind the scenes, so do drop an email to firstname.lastname@example.org and let him know that you support this attempt to obtain adequate supporting scientific information for scientific claims made directly to the media. Copy me in if you like, email@example.com. This circular time consuming farce and secondary formal communication is quite absurd.
It really would have been much easier if the "lead researcher" Dr Madeleine Portwood – to whom you are directed with all technical questions – had simply been polite enough to even return a phone call or email. Meanwhile we got this kind of rather unpleasant activity from someone saying they were not related to Madeleine Portwood. Absolutely, stunningly bizarre. I hate this kind of thing really. Oh well. And did you ever think, if fish oil is so important, how come vegetarians aren't stupid?