golem.ph.utexas.edu/category/2006/09/a_plea_to_save_new_scientist.html
www.sunclipse.org/?p=296
golem.ph.utexas.edu/category/2007/09/the_virtues_of_american_scient.html

The reason seems to be that they often run feature stories on research reported about in the arxiv preprint server, which not always will make it, at the end of the day, through peer review.

Since then I’ve found it hard to place too much faith in things reported at conferences. Not that conferences are a bad thing (expenses paid jollies are all the holidays some scientists get). But only a small proportion of posters and talks given at a conference are ever turned into published articles.

1. They let scientists know what the current hot topics are.

2. They give socially challenged scientists things to talk to each other about at conferences, helping them get to know each other.

3. They provide an advert for the speaker’s work, like a movie trailer. Last time I presented something at a conference, several people expressed an interest, and I gave them a draft of the paper, containing most of the details that will be in the published version. This arrangement benefits everybody.

But look at what happens when stories appear in the media claiming that, for example, MMR is dangerous: if the work has not been published, then the story can’t be refuted, even if it later turns out to be rubbish. But, by then, the damage has been done. A few more parents fail to have their kinds vaccinated. The measles rate goes up.

I think that if you argue that that conference presentations shouldn’t be reported until the paper comes out, you also have to argue that scientists should refrain from presenting at conferences until their paper is published.

After all, even if no reporters are present, a 15-minute conference presentation in advance of a published, peer-reviewed paper leaves even a professional audience without the full details they need to scrutinize the research adequately.

So it seems to me that to be consistent, you’d have to argue that about 99% of the presentations at about 99% of the scientific and medical conferences should never be delivered, or that all presentations should be lengthy recitations containing at least as much detail as in the published paper.

Do you really want to argue that?

I wouldn’t say a crime reporter should wait until a trial is completed before reporting it, but it might be beneficial to wait until it’s started, because that’s when the evidence begins to receive scrutiny. Until then it’s just speculation. The results in a scientific conference presentation or press release can’t receive that level of scrutiny, because the details are not known until the paper comes out.

When a scientific finding is announced publicly, at an open scientific meeting, a press conference, or publication in a peer-reviewed journal, that event is news.

I think it’s unreasonable to ask science journalists to wait to some time in the perhaps distant future, after the full paper is written, peer reviewed, rewritten, published, and fully available, to report the news.

Crime reporters aren’t asked to wait to report a crime until after the accused’s trial is completed and the transcript is available for full scrutiny.

Political reporters aren’t asked to wait to report on legislation until the final version of the law is passed and signed.

On the other hand, it’s not unreasonable to insist that journalists report scientific news as accurately as limited time and space allow. For example, they should quote impartial experts who can put the research into context and apply the necessary caveats.

But that’s just the problem – the articles are often NOT written accurately. Look at how the person writing the subheader for Ben’s THES article got the message entirely wrong. There are loads of examples on badscience of journalists getting the story completely wrong. If the full scientific paper is available at the time that the story appears in the media, then any misreporting can be detected and followed up immediately. This can’t happen if the paper slips out quietly a few months later, after everyone has stopped talking about it.

And what about cases where the story is reported accurately, but the science is dodgy? If the full scientific paper is available at the time that the story appears in the media, then the science can be subjected to full scrutiny at that time. The thing that really convinced me that it was safe for my daughter to have MMR was reading Andrew Wakefield’s original paper, and seeing how unconvincing his arguments were. If scientific papers are available at the time that a story appears in the media, it protects the public from bad reporting, AND bad science.

In that spirit, conference coverage could be described as “the first draft of science.” Indeed, that’s the motto of the International Medical News Group (a division of Elsevier), which publishes more than a dozen trade papers for specialist physicians (i.e. Family Practice News, Clinical Neurology News, etc.)

You could argue that these publications are targeted to highly educated physicians, who presumably understand the tentative nature of scientific findings. But why hide these stories from the general public, assuming that the articles are accurately written (a big assumption, I know) and that they don’t imply that the latest finding is the last word on the subject.

that bloke ioannidis’ think pieces and other research on this are v interesting, nice brief summary here, more in eg ioannidis’ plos medicine article:

online.wsj.com/article/SB118972683557627104.html

An interesting analysis, although as a journalist I don’t entirely agree that embargoes and the Inglefinger rule are good things.

But I have a question: If you believe that journalists should only report on peer-reviewed published papers that are available to everyone, would you then ban journalists from covering medical conferences?

Virtually none of the presentations at conferences are peer-reviewed, and except for societies that publish un-peer-reviewed brief abstracts, the papers presented are not widely accessible.

re: blaming the “scientific community”. there will always be cranks and rightly so, sometimes they have excellent ideas, i would be horrified at any kind of central gagging system and i cannot imagine for the life of me what such a thing could possibly look like, or how it would work. it is no surprise there are odd cranks. what is wrong is that they are reported as if they represent a mainstream plausible opinion, as if their views represent a credible survey of the literature.

How much difference does availability of the full paper make to what the press write and the public hear? Look at what Melanie Philips wrote about the Cochrane report on MMR.

Do GPs really have the time to look up the original research when someone comes in waving a press clipping?

And a comment:

It is easy to blame the press but the scientific community needs to get its own house in order as well. It was Wakefield who decided to make misleading accouncements about MMR. It was the Uni of Utah plus Fleischman and Pons who did the same for cold fusion. The Ingelfinger rule didn’t stop either of them.

Certainly by me, at any rate.

Maybe there’s a good explanation for this that hasn’t occured to me, but why doesn’t medical science have an equivalent of arXiv? A comprehensive open access pre-pub archive would be helpful, right?

speaking of which i should follow this up:

www.badscience.net/?p=159

i vaguely remember i promised to do it annually.