Ben Goldacre
The Guardian
Saturday 30 May 2009
Obviously we distrust the media on science: they rewrite commercial press releases from dodgy organisations as if they were health news, they lionise mavericks with poor evidence, and worse. But journalists will often say: what about those scientists with their press releases? Surely we should do something about them, running about, confusing us with their wild ideas?
Now you may be inclined to think that a journalist should be capable of doing more than simply reading, and then rewriting, a press release: but we must accept that these are troubled times. Through our purchasing behaviour – and I assume someone cleverer than me measures these things competently – we have communicated to newspapers that we want them to be large and cheap, more than we want them to be adequately researched.
So in this imperfect world, it would be useful to know what’s in academic press releases, since these are the people of whom we are entitled to have the highest expectations. A paper in the Annals of Internal Medicine this month shows very clearly that we have been failed.
Researchers at Dartmouth Medical School in New Hampshire took one year’s worth of press releases from 20 medical research centres, a mixture of the most eminent universities and the most humble, as measured by their US News & World Report ranking. These centres each put out around one press release a week, so 200 were selected at random and analysed in detail.
Half of them covered research done in humans, and as an early clue to their quality, 23% didn’t bother to mention the number of participants – it’s hard to imagine anything more basic – and 34% failed to quantify their results. But what kinds of study were covered?
In medical research we talk about the “hierarchies of evidence”, ranked by quality and type. Systematic reviews of randomised trials are considered to be the most reliable: because they ensure that your conclusions are based on all of the information, rather than just some of it; and because randomised trials – when conducted properly – are the least vulnerable to bias, and so the “most fair tests”.
After these, there are observational studies (“people who choose to eat vegetables live longer”) which are more prone to bias, but may be easier to do. Then there are individual case reports. And then, finally, because medical academics like to think they’re funny, right at the bottom of the hierarchy you will find something called “expert opinion”.
In this study, among the press releases covering human research, only 17% promoted the studies with the strongest designs, either randomised trials or meta-analyses. 40% were on the most limited studies: ones without a control group, small samples of less than 30 participants, studies looking “surrogate primary outcomes” (a blood cholesterol level rather than something concrete like a heart attack, for example), and so on.
That’s not necessarily a problem. Research is always a matter of compromise over what is practical, or affordable: it would be nice to randomise every single patient, everywhere, whenever there is any uncertainty over which is the best treatment for their condition, and perfectly follow their progress, but that would be quite a piece of work. It would be nice to randomise everyone in the country to different lifestyle choices at birth, to see which had the most significant impact on their health, so that in 70 years time we would have a comprehensive story on the best way to live, but it’s not administratively realistic, and it’s hard enough to get people recruited and cooperating in a brief 3 week study, let alone lifelong change.
So people conduct imperfect research, knowing that it is the best we can do with the resources available, knowing that the results must be interpreted with caution and caveats. This isn’t “bad science”, because the studies themselves are – we assume – well conducted, and faithfully described in their publications. The errors come at the level of interpretation, where people fail to acknowledge the limitations of the evidence.
That failure is a crime, but is it limited to quacks and hacks? No, and that is the key finding of this new paper. 58% – more than half – of all press releases from this representative sample of academic institutions lacked the relevant cautions and caveats about the methods used, and the results reported. I would like journalists to be experts in their field – and I don’t think they could be bluffed as easily by a politician and a sports personality as they are by a science press release – but make no mistake, this is a war on all fronts.
Bob Ward said,
May 30, 2009 at 9:01 am
Good work, Ben.
davehone said,
May 30, 2009 at 9:16 am
Hi Dr Ben, I just wonder how much of this is down to the researcher and how much of this is the PR part of universities etc.
I have had plenty of my press releases mangled by the university press offices with (what I thought) were pertinent points and caveats taken out and new things added in (often without my knowledge / permission). The alternative is to try and do things independently, but there I found it hard to communicate with the media directly (as a researcher I hardly have everyone’s e-mail addresses or phone number and nor was my press release necessarily written in a media-friendly manner) and it took up a huge amount of time (and nor did it please the press office).
It seems that the media themselves, the universities and the researchers need to come together on this to make sure the right point gets across and is reported in the right way. I do my best (which is of course perhaps not good enough) but find it intercepted and changed when out of my control, yet lack the time / experience to fix it, and frankly am not helped by the press officers who are theoretically there to help promote my work, nor the media who want to publish it.
Your general point is well made, and of course I am sure there are people putting out poor press releases, or not being careful with what they write or say, but even ‘good’ ones are probably getting caught in the gears as they try to get their points across.
matthewthomas said,
May 30, 2009 at 9:55 am
Another problem I have found is that the distinction between clinical research and basic research is blurred, because clinical stories can sound more like a breakthrough.
A (basic science) study I was involved with was published in a good journal and we decided to inform our funding body and the university, because we wanted to be able to tell anyone interested what a scientist does all day with their money. Our study was in mice, which we clearly pointed out, and acknowledged the important difficulties in translating mouse work to humans. Although could have some future clinical implications if properly followed up – we proposed that future work could examine whether our protein was a viable drug target – we were careful to make it clear to the university PR office that it was not a clinical study, that we didn’t have a drug, and we were unlikely to try to make one ourselves. Almost immediately we had apparently predicted a drug within ten years. Within days we had apparently had a drug that prevented heart attacks. It seemed that both PR office (to an extent) and news outlets believed that no-one would be interested with a science story, so it had to be a clinical success story.
The longer this continues, the worse that public understanding of us basic scientists will get.
nigel edwards said,
May 30, 2009 at 10:28 am
My specialty is health policy not scientific research. The idea of a press office that is able to send out a press release that is not agreed by the originator of the research is alarming. I have an excellent press office but they always check the finished product with me first. There is an obligation on me to ensure that what is being said really reflects what I want said and that opportunities for misunderstanding is spotted and removed.I rely on them to advise me on hazards like this, not to get me in a paper at any price. The reason for this is that my reputation as a reliable source is too important to be ruined by a ‘good line’ which is not right. This is also true for researchers.
Press office’s are probably being measured by the number of mentions they get, not by the accuracy of what they say. This is dangerous, not lesat for the long term reputation of the organisations they represent. This looks like a management problem about the objectives and operating rules of the press office and universities need to fix it.
Michael Grayer said,
May 30, 2009 at 11:11 am
One thing that appears to be absent (at least from my quick skim-read of the article) is a comparison of the “eminent” and “humble” groups of universities. While the results here are worrying regardless of which university’s press office distorts the findings, I would be slightly less worried if it turned out that it was the lower ranked universities that provided the greatest distortion.
Thinking about it, though, it’s entirely plausible that it could be the other way round as well. Which would be rather more worrying.
It may also be interesting to see whether universities who ranked near the middle of the table provided comparable results.
Synchronium said,
May 30, 2009 at 12:17 pm
Wow, that’s a surprising figure. It would be good if they did another study 5 years down the line to see if there’s any trend.
SharonC said,
May 30, 2009 at 1:04 pm
I agree with davehone.
I don’t see why we would think that University PR offices would be filled with people better able to write suitable descriptions of science than journalists. Ok they are nearer to the actual scientists, but then academics are but an email away to
journalists too.
The last time I had an opportunity to write a press-release type of article, the university PR person offered to write the article, but given his previous comments on the subject I could see he hadn’t grasped the subject properly, so I wrote the article myself with the help of colleagues.
To my amazement it did end up in a newspaper with minimal editing, such that misrepresentation did not occur. I put that down to lots of time spent writing the article in the first place to a high standard to minimise the editing that the paper would want to do.
HolfordWatch said,
May 31, 2009 at 8:42 am
The excellent Language Log discusses this: Study: hacks often bamboozled by flacks.
Sili said,
May 31, 2009 at 5:11 pm
For a while there I was trying to understand why you’d stooped to plagiarism.
Then I realised that I the sentence I recognised was from the bit that Liberman quoted on the Log yesterday.
I hate my brain and the revisionism that it engages in.
Robert Carnegie said,
June 1, 2009 at 1:28 am
The experiment is probably do-able to an extent with historic press releases, if you’re looking for a trend, for effects of different types of university funding, staffer funding, maybe access to publication – is there substantial unpublished-at-the-time science getting covered in press releases?
Then again, nearly 20 years ago I was getting asked by the head of department whether I could see my way to getting something published and score departmental points. We were the faculty of law, department of accounting, and I was the computer technician – we’d been donated a computer laboratory a while before. Briefly I considered writing up something on on-screen print preview technique (which I was quite proud of)… So that pressure to produce isn’t new like the Internet.
Filias Cupio said,
June 1, 2009 at 3:05 am
This article in “The Register” should be of interest to anyone who is reading this.
“Blog homeopathy horror hammers hippy herbalists”
(“Graun abandons Q&A after subject hides under bed”)
www.theregister.co.uk/2009/05/29/graun_neals_yard_storm/
Ben gets a mention.
anotherfakeid said,
June 1, 2009 at 7:36 pm
It’s only going to get worse. Science is being removed from the list of core school subjects to be replaced by ICT (computers and I don’t mean the science behind computers either). Press releases and press reporting will continue to appall you all. Imagine Sheriff Carter from “A Town Called Eureka” writing a press release.
getinspired said,
June 2, 2009 at 10:53 am
anotherfakeid is right
www.tes.co.uk/article.aspx?storycode=6012613/
Well, nothing is definite but it’s certainly not an unlikely prospect. It
getinspired said,
June 2, 2009 at 11:11 am
anotherfakeid is right
www.tes.co.uk/article.aspx?storycode=6012613/
Well, nothing is definite but it’s certainly not an unlikely prospect. It’s crazy, I still know people that thing heavier things fall faster! – how can you come out of high school thinking that? I dread to think what effect taking science out from the core subjects will have. Good news for quacks and charlatans I guess…
(@Ben: Sorry to nitpick but I think you forgot to put in your tag in this post :/ – unless you meant to have the whole article show on the home page – in which case, ignore this comment)
getinspired said,
June 2, 2009 at 11:15 am
ahhh major browser catastrophe!
Link should have been
www.tes.co.uk/article.aspx?storycode=6012613
pipedreambomb said,
June 3, 2009 at 11:52 am
How about if an independent scientist or panel gave each piece of research a grade from A+ to F for Research Quality, based on a tabulized version of the previously informal hierarchy you mention.
It’d be relatively quick and simple to get a rating – 5 points for a test group over 1000 people, 10 points double-blind placebo, etc – and most importantly, it’d be very easy for journalists to grasp so they can quickly judge the merits of a press release.
notmyopinion said,
June 7, 2009 at 1:40 am
anotherfakeid is NOT QUITE right.
According to getinspired’s link, science may be dropped from the PRIMARY curriculum. Pupils over 11 will still study science – just like in the “olden days” of the late 20th century.
irv said,
June 8, 2009 at 9:55 pm
I used to work at a newspaper (in IT) and saw quite a few reporters and editors quit or get laid off, only to get higher paying jobs as PR types shortly thereafter. There is apparently a lot of overlap in the skills required for the two jobs.
With that in mind, expecting different performance from one than the other seems irrational.
Possibly it would help if someone created a handbook for science reporters or PR people, to help them navigate some of the pitfalls that weren’t covered in their college classes in writing.
Whether or not anyone would actually use such a handbook is an open question.
Ian B said,
June 9, 2009 at 1:01 am
But should not the question surely be, “why are scientists sending out press releases at all?”
What is the intention of a press release? Research is supposed, is it not, to be incremental, published in a journal to be considered by other researchers in the context of the body of knowledge in the field as a whole. So why is academic research being announced in this piecemeal way to the public? That’s not science, is it? It’s PR. Why are academics indulging themselves, then, in propaganda?
Let us be honest here. Science has an ideal as a quest for truth, performed by selfless toilers. But the reason science has become science by press release is that many scientists having axes to grind, and are releasing their results in this manner via the press precisely because both the press and the public do not understand the subtleties of interpretation of studies and the fine points of statistics. If a researcher is for instance an anti-fat person crusader, he can release to the press the statement that (e.g.) “30% of stillbirths are to overweight women”, creating the impression that being fat is a risk to the child, having not included in the press release the fact that 30% of women are defined medically as overweight.
It is ludicrous for such researchers to argue that it’s the fault of university PR offices, or the journalists, or the public, for mangling or not understanding the research. If researchers really cared about clarity, they would not author the press releases in the first place, or would issue retractions. But there are enormous vested interests in the scientific community, as in all communities, and there is more to be gained by bandwagonning scares, and worst of all is the simple fact that many of these researchers, particularly in hotbutton issues like tobacco control, obesity, alcohol and so on are just, let’s be frank, fanatical nutballs relentlessly pushing various prohibitonist agendas.
If we want an end to bad press releases, we must have an end to press releases, period. There is no scientific reason for the readers of the News Of The World to be breathlessly told that people who eat 1mg less salt per day have a 3% lower risk of a heart attack. Is there?
Teapot said,
June 10, 2009 at 12:07 pm
Unfortunately British universities have become heavily corporatised and bureaucratised over the last 20 years, and one of the worst effects has been the ever-increasing obsession, particularly induced by the RAE, with getting (grant) money, as against doing anything useful with it. I suspect the PR is all part of this, with the hope that getting your name in the papers helps you, your department and your university get more cash in the future.
heavens said,
June 15, 2009 at 11:15 pm
Davehone (and others): Have you considered addressing your PR office on the issue directly, perhaps using this paper? Next time, try sending them a list — yes, a plain, simple, bullet-point, “my brain has been eaten by PowerPoint” list — all of the relevant key factors, like:
* NUMBER OF PARTICIPANTS: 36 people
* STYLE OF TRIAL: observational
* LENGTH OF TRIAL: three weeks
and then point out that (because these studies are increasingly common) the university will be ridiculed if all of these details are not included somewhere in the press release and it gets evaluated. Naturally, you’re just trying to help them by providing all of the information that they’ll obviously need in a handy package.
Fear of embarrassment is a powerful motivator.
If you can get your entire department to use a similar approach, then you might get them used to the fact that this checklist should exist, thus changing your departments habits.
(Hmm… Behavioral psych study, anyone? The “intervention” could be a one-hour meeting with the PR folks about a change to their press release approval process, and a note to the geeks about not getting a press release sent out if they don’t provide all the details. Compare “experimental” group against a random sample of press releases from a similar university.)
quiact said,
June 19, 2009 at 5:08 am
Those who design and create releases for the press, which is the print media, are designed to contain information of sufficient importance or interest to the public, historically speaking. This is what U.S. citizens are led to believe about what we may read.
Instead, those who design and release written information to the press are often sponsors of the print media who will issue the press release.
Such sponsors often instruct such media outlets with mandated authoritarian nuances such as the press release that they created will not be altered in any way by the print media that agrees to release the press created by the sponsor of the media outlet.
Of course, the sponsor and creator of such a press release create such written words in order to promote the sponsor’s company, as well as its products. By doing so, they are allowed the freedom to embellish if not fabricate what may be annotated on the release they issue to the press that has now been bought by them, the corporate sponsor.
These well- constructed statements are meticulously composed and customized before they are issued to targeted editors and contacts at mass media publication locations.
The sponsor also has been known to direct the location and time of the release of their press creation that, upon direction from the sponsor, is completely un-reviewed by such a media source.
As this is done, the mass media outlets are again instructed on how to present their completed statements by who are often corporate sponsors. Furthermore, the media is given instructions once more that what has been written by their sponsor shall remain as it exists.
As a result of this collusion, press releases are presently a form of public relations often utilized for those companies who create what is supposed to be an attempt to express their products as being newsworthy to the readers.
Press releases, historically, have been created and released to inform the readers by adding insight and related information for them regarding a particular topic that was typically complete and balanced. At least, that was the intent.
Today, they seem to be more or less an annotative commercial with press compositions generated by corporations in particular, so it seems.
Unfortunately, and presently, press releases are often embellished, biased, and incomplete with deliberate intent in order to benefit the creator of these documents, who again develop them solely to increase awareness and usage of their products that they promote with their business, which they want to be viewed as favorable and with a positive image to the public.
One could suggest that the mass media who receives these press statements from certain corporations are transformed into acute front groups who perhaps coercively offer third party legitimacy for the content of the press release as they release this information to their readers.
The often notable if not intentional, flaws at times are numerous within such press releases that reflect reckless disregard with informing readers in such a way, who are the American public. Citizens typically believe that what they are reading from a respected media source is both honest and complete.
An example is an anonymous press release posted on the Medical News Today website (www.medicalnewstoday.com) that is dated in March of 2006. The title: “Cymbalta Safely and Effectively Treats core anxiety symptoms associated with generalized anxiety disorder.”
Cymbalta, by the way, is a psychoactive drug often utilized for human affective disorders.
Clearly, this title itself includes words associated with relief or elation, which are subjective and not objective elements which would clearly be more appropriate- with a health care press release in particular.
The first paragraph of this press release repeats the results mentioned in the title of this article, but also states Cymbalta offers relief of painful symptoms associated with anxiety, as well as improved functional impairment- also claimed to be associated with anxiety in this press release.
These conclusions are speculative at best, as these inferences appear to be unexamined by others regarding the benefits claimed to exist with Cymbalta as illustrated in this press release.
Cymbalta was not approved by the FDA for anxiety or any of the symptoms associated with this condition at the time of this press release. In fact, Cymbalta was not filed with the FDA for this speculated new indication for anxiety that was desired by Eli Lilly until May of 2006.
By definition, this press release may possibly be off-label promotion as well as misbranding of Cymbalta that was performed overtly in this manner of the press release, one may speculate.
As one continues to read this press release, testimonials were intentionally created and inserted into this press release that illustrated results they hope are impactful to the reader regarding Cymbalta.
This testimonial was from the lead author, who expanded the claims made initially with utilizing various medical terms, which was followed by this person’s passionate optimism about the great potential of Cymbalta based on this remarkable study.
This study, by the way, was to be addressed in further detail at a National Anxiety meeting some weeks after this press release was announced to the public on this website.
The second testimonial was Eli Lilly’s Medical Advisor expressing his elation about what the lead author just stated, followed by how much he was encouraged by these results that will benefit so many others that have these debilitating medical conditions.
Of course, profit forecasts and desired market growth and expansion regarding Cymbalta remarkably were not stated in this press release.
What is not included in this particular press release were any clear statements regarding the disadvantages and adverse if not toxic events associated those who take Cymbalta.
Reactions from Cymbalta users include discontinuation syndrome at times, when the user stops taking this medication, which I understand can be quite devastating for the one experiencing this syndrome.
Furthermore acts of suicide and suicidal ideation have been frequently associated with those who take Cymbalta as well. There has been a lack of efficacy suggestions by others who have taken Cymbalta.
Basically, anything that may be considered negative aspects about this drug were not annotated in this particular press release as it should have been for fair balance that is or should be a primary standard in the pharmaceutical industry and the professions involving journalism.
Acquired from Wikipedia:
According to The Elements of Journalism, a book by Bill Kovach and Tom Rosenstiel, there are nine elements of journalism [1]. In order for a journalist to fulfill their duty of providing the people with the information they need to be free and self-governing. They must follow these guidelines:
1. Journalism’s first obligation is to the truth.
2. Its first loyalty is to the citizens.
3. Its essence is discipline of verification.
4. Its practitioners must maintain an independence from those they cover.
5. It must serve as an independent monitor of power.
6. It must provide a forum for public criticism and compromise.
7. It must strive to make the significant interesting, and relevant.
8. It must keep the news comprehensive and proportional.
9. Its practitioners must be allowed to exercise their personal conscience.
In the April 2007 edition of the book [2], they have added one additional element, the rights and responsibilities of citizens to make it a total of ten elements of journalism.
The staff involved with the release and publication of such press releases as this one was annotated and described should perhaps be more informed on what not to accept and what to present regarding these issues addressed. Then they may meet the requirements and obligations of what they provide the people.
As with any reporting by the media, objectivity and thorough completeness of the topic discussed in a press release is a necessary requirement with any publishing that is potentially exposed to so many other readers- with issues related to the restoration of their health in particular:
“The public has a lot at stake, and the media has a responsibility always to be aware of the source of information and the conflicts those sources might have when they report the results of clinical research. People who have financial stake in the results of clinical research can well be biased in the way research is conducted, in the way they report it, and what they say about it when interviewed by the media.”
– Arnold Relman, former editor in chief of the New England Journal of Medicine
Dan Abshear
Vivienne said,
June 20, 2009 at 7:24 am
I do quite a bit of PR copywriter, including press releases on scientific topics.
I’m shocked that some university PRs don’t have press releases signed off by the lead academic on the study. This is basic practice in PR.
I do copywriting on scientific and technical topics for companies, and the scientist ALWAYS has full editorial control over what I’ve written.
I’m normally glad they do because, despite having a science PhD, I’m not arrogant enough to believe I can write about an unknown field without errors.
wayscj said,
November 21, 2009 at 6:02 am
ed hardy ed hardy
ed hardy clothing ed hardy clothing
ed hardy shop ed hardy shop
christian audigier christian audigier
ed hardy cheap ed hardy cheap
ed hardy outlet ed hardy outlet
ed hardy sale ed hardy sale
ed hardy store ed hardy store
ed hardy mens ed hardy mens
ed hardy womens ed hardy womens
ed hardy kids ed hardy kids ed hardy kids e
joey89924 said,
November 22, 2012 at 1:17 am
Science is being removed from the list of core school subjects to be replaced by ICT (computers and I don’t mean the science behind computers either).
HV9910