‘Anti-religionist Dawkins, the best-known English dissenter since Darwin, is the merciless demander of provable fact.’

http://www.dailymail.co.uk/news/article-1070535/QUENTIN-LETTS-The-50-people-wrecked-Britain–2.html

Oh no! How dare he! Don’t people see that demanding provable fact just gets in the way of selling papers.

Does Mr Letts think that provable fact wrecks Britain? Perhaps he would like us all dead from beubonic plague through lack of anti-biotics. He’s actually playing a clever game. By claiming that the demand for provable fact is damaging he gets to have his cake and eat it or more applicably scare people out of MMR protection for their children and then dodge any responsibility for it. Wow! I wish I could live in his twisted world. But sadly, an ability to think critically got to me.

http://www.cochrane.org/reviews/en/mr000011.html

The result is that the systematic reviews that inform, for example, NHS purchasing policy, are often heavy on published interim analyses with positive results for surrogate outcomes (disease or progression-free survival) and light on mature trials with equivocal results.

This is a particular issue where the disease has a long natural history. For instance, recurrence of successfully treated breast cancer can happen over twenty years or more. So making policy on the basis of an one year interim analysis (as happened with the NHS, Herceptin and the HERA trial) rather than waiting to see what happens at, say, five years, might be seen by some as jumping the gun. Doubly so, if there’s another trial out there making the same comparison but with negative results which is difficult to find because it was only ever presented in a poster at a conference.

http://www.pharmac.govt.nz/2008/05/23/Accepted%20Authors%20Manuscript%2008CMT0415MetcalfeTrastuzumabDMc.pdf

If you want a really stunning example of how wrong things can go, especially when early analyses involving small numbers of events are involved, check out Wheatley and Clayton’s paper on the AML-12 trial. Early positive results in favour of the experimental intervention turned to harmful results later on.

http://www.ncbi.nlm.nih.gov/pubmed/12559643

In the eyes of the industry, the pr companies, patients and publications who lobby on their behalf, an interim analysis with negative results is just ‘immature’; an interim analysis with positive results is proof that the NHS should be buying their drugs.

“Then they went to Pubmed.gov, the searchable database containing almost all the proper medical journals out there.”

Mm. I’m an NHS librarian, and like many of my colleagues I tend to tell doctors that to be fairly sure of retrieving the majority of articles of interest, they need to search at least EMBASE as well as MEDLINE (MEDLINE is, mostly, the database that Pubmed is querying). See e.g. http://www.cfpc.ca/cfp/2005/Jun/vol51-jun-research-1.asp. So I’d be surprised if a Pubmed search retrieved every trial that was published.

OK, so the Pubmed search was a sensible back-up for their other measure of publication, the publication record in the trials registry (though this is not clear in their abstract). But while we’re in the business of niggling at methodology…

Trials were counted as “published” in Pubmed only if the trial registration number was found in the search. It’s easy to hypothesise publications which missed this number off; for example, journal articles which haven’t yet been fully indexed in MEDLINE (which can take months after publication) tend to have just a basic “in process” record, which might not hold the trial registration number.

There’s also the issue of time from registration to publication. The trials register began in 1999. Many cancer trials go on for five years plus (I don’t have a reference for that, it’s an unsubstantiated hunch based on the protocols that come through my local research committee). Chances are a number of completed trials finished fairly recently and just haven’t been published yet – it can take a fair old time, even if you’re swotty about writing up quickly & you’ve got a paper which makes a good story.

I don’t actually disagree for a moment with the conclusions drawn in this paper or with Ben’s comments. Good meta-analyses often do funnel plots to look for publication bias and they can be magnificently skewed. It is a fantastically important issue. But if this paper had been picked up in the mainstream media, could it have been a slightly over-sensationalised story about sensationalism?

“Oh,” he said. “Nobody’s ever been able to replicate his work.”

I’ve shelved it for now, but I still don’t know what I’ll do in the end. Perhaps we’ll be the first to replicate it? I’ve talked to the original researcher (now working for a different institution with a different target), and he’s confident that we can do it. I can’t find anyone that will admit to having tried to replicate the work, but the fact is that the study was published 15 years ago, and people move. It sure would have been nice if people would put a short note in J Neg Results (or equivalent): Tried this, didn’t work, maybe it’s us, maybe it’s not. First-year chemistry students can write a report like this; why can’t we?

In physics, it is unlikely that anyone will come to harm as a result of the results of experiments not being published, but I can’t help but think it would prevent a huge waste of taxpayer money if all research which produced negative results was published.

For example, you use a particular mathematical treatment to go through a difficult, nonlinear problem. In the course of working through the problem, you discover that it violates one of your original assumptions. You think, “damn, I’ve spent 6 months on that, and there was no way of spotting the violation until I’d worked through it”, yet, unfortunately, most journals wouldn’t touch a write-up of this failed treatment with a bargepole.

After three years you are sent a research proposal from another country to review which asks for a hundred thousand pounds to use exactly the same theoretical treatment that you discovered was invalid.

My point is that scientific papers are not just used to amass and interpret evidence, but can also be used to establish whether future research directions are worth pursuing. It is so difficult to decide how to use public money to fund science, I feel like the people making the decisions could do with all the help they can get.

I think that, these days, most people working outside of Pharma company marketing departments might acknowledge, in their heart of hearts, that negative and equivocal results should be published. Many use Journal editor intolerance of boring studies as an excuse not to(and need to be referred to Joe Dunckley’s post above).

I think that more people would deny that failed studies should be published – and maybe a campaign is needed for failed studies, in the same way as we needed one for studies showing negative and equivocal results. They’re unlikely to give much evidence about efficacy, but they might give a lot of data about safety, compliance and clinician/patient confidence.

You can not force someone to put their negative results into a journal-publishable form.

You can certainly force them to publicly report their results on a registered clinical trial, by ensuring that anyone involved may no longer participate in any other clinical trials until such reporting has been performed (if it is delinquent).

“ClinicalTrials.gov currently has 61,995 trials with locations in 157 countries.” (Wow. That is a lot more than the 44,232 listed at the time of the cited analysis in September 2007.) If registration with ClinicalTrials has become the mandatory hoop, perhaps it is important to the public to simply ensure that trial designers finish their jump by reporting their results, published or not, in the same database?

I’m missing something here… and am too lazy to read the primary papers…

You are simply assuming that just because the media report does not detail it then no controls ocurred. I note you are still ignoring that since it is a meta analysis they are not responsible for reporting the methods, only checking that the studies in the analysis meet the required criteria for good science. This is a Cochrane Report. If you denigrate this one then you have to denigrate all Cochrane reports since the criteria are the same.

Do you really want to go there?

Researchers led by Ying Cheong, from the reproductive medicine unit at the University of Southampton and the city’s Princess Anne hospital, concluded that ‘acupuncture around the time of embryo transfer achieves a higher live birth rate of 35 per cent compared with 22 per cent without active acupuncture’.

The ‘proper’ control would have been sticking the pins in not-quite-the right-place. But, of course, the ‘researchers’ would then have known who was control vs treatment.

Women undergoing IVF are highly vulnerable to this sort of nonsense, and it seems rather unfair that potentially large numbers of them will be ‘pin-cushions’ in the name of placebo.

“you can’t say it’s unacceptable to publish wonky science just because some morons will overinterpret it”

Not sure ‘morons’ is a good word in this context. You’re describing people deliberately skewing the message in their favour (i.e. smart), or people believing such distortions.

A lack of morals or education (or both) doesn’t make someone a moron.

Ben chose to highlight that 27% of all, among which nearly 79% published positive results, highest of all.) You further intimated that this can impact prescribing decisions.

Nearly 82% of the 2,028 trials included in Ramsey and Scoggins’ analysis were Phase I or II, neither of which should necessarily influence prescribers’ behaviours, as it is much less likely that these trials will include compounds that have marketing authorisation. Frustratingly, the article did not include a cross-tabulation of Sponsor type by Phase, which would be interesting to look at.

In my view, I don’t believe private-sector, for-profit companies are obliged to publish in a PubMed-citeable reference the results of all trials including those where an investigational compound has failed and the trial does not progress, as this would represent an additional cost burden on a pursuit where they will not have a commercial return. (That’s not to say I don’t believe that all clinical trials should actually be reported and shared in a free and easy-to-access way.)

I’m curious to know what your view is and whether you feel this is something you might raise in a future column.

Note that getting women trying to get pregnant to ‘take’ anything will be inherently problematic so acupuncture is perhaps a good way to achieve stress relief. Probably massage would have the same effect. Though it might be a bit too hands on for some people so less acceptable.

http://www.guardian.co.uk/science/2008/sep/21/medicalresearch.health

Did the author of this ‘most interesting’ piece think to ask your opinion of it? Or is it all true (i think the lack of a proper control group as opposed to no treatment seems to be required…).

I think it’s time you sent Mr Campbell the sort of email that would make Giles Coren blush…

It is a shame that Bettween the public and the scientists there is an ineffecient middle man – the media.

What there needs to be is more regulation of the media to stop them printing incorrect facts, rather than not publish certain research as I previously stated.

A result obtained/not obtained
For “a result obtained”:
Promising/Unclear/Unpromising
An academic contact for further information.

Some thing like that would avoid the deadweight cost of publishing a lot of useless information in academic form; give the principal steer from each trial easily and quickly; and let those who need to track back to work that is relevant to them.

My employer funds environmental research on the proviso that all resulting data is lodged with one of its nominated data centres. Our data holdings are freely available to the scientific and general public, and subject to charge for commercial users, but with measures in place to protect the intellectual property of the data provider.

We hold a pretty big stick, in that future funding may be withheld from scientists who fail to comply with data submission. I wonder how we would police the submission of data paid for with independent money..?