What we’re talking about is combining both the positive result trials and the negative result trials into a database that would provide a better way of cross checking for previous investigations into what ever subject.

even if only the authors, paper or trial name and key words were entered into this database it would be an excellent research tool.

To give an example. Say i’ve done a trial, written it up etc but that the trial was a negative result. I may well chose not to publish to the general science community but just pass my results along to those paying or who had to be reported to. What i could then do in this instance is to enter the result into the hypothetical database.

If somebody then requested the results based upon their search i could direct them to the study results or send along the abstract etc.

As for your statement that the results of a study of this kind not being advantageous, i’d have to disagree.

The study in question was not simple exposure to pollen. What was done was an extract of the pollen was made and administered to the patients on a daily basis over a period of 18 months. This was done through a solution held under the tongue for 60 seconds then swallowed. unfortunatly i was only able to do 6 months of this trial due to sever injuries and i don;t know what the end results were.

Personal experience seems to indicate a good reduction in symptoms for around 4-5 years from my partial treatment (anecdotal i know). However this study would have produced data showing either a positive result (which may indicate a good delivery method to further researchers and encourage further trials with other pollen types) or a negative sugesting the opposite.

Either way, more knowledge = good, less = bad

There is a database of experimental results; it’s called pubmed.com

The kind of information you desire is contained within the normal scientific literature; you go to pubmed, type in your keywords (ragweed AND allergy) and you get links to the relevant publications. Release of the ‘raw data’ from this kind of study would not be terribly beneficial to the scientific community because the interpretation of such work is very straightforward; you expose the patient, measure the response (i.e. bronchial constriction/wheal and flare/subjective itchyness scores or whatever). There’s not much new data that can be gleened from such a database (as oppossed to reanalysis of old data).

On the other hand, more technical studies (i.e. recording the behaviour of parts of the brain in response to a stimulus or whatever) contain vast amounts of information, normally as a result of simultaneous recordings of various parameters (e.g. heart rate + blood pressure + nerve activity). The results from a particular study may focus on one subset of the data (i.e. what happens to nerve activity in the 30s after you do proceedure A), when such a subset only represents perhaps 10% of the whole dataset. In these instances, there may be other researchers interested in what the relationship between one parameter and another at basal levels (e.g. is nerve activity modulated by heart rate?). Public access to such results should be encouraged and could be greatly beneficial.

I’d be willing to bet that much less than 50% of the money that gets into research is wasted on double efforts; I know this because I’m a researcher and I’ve had to run the grant gauntlet. Research that tries to reproduce previous results is simply not funded. There are more important things to fund, and not enough money. The people that decide who the cash goes are experts in the field, recruited by the grant-funding bodies for said expertise. Only about 10-15% of applications are successful, and the people that write the successful grants are the people with the track record of high impact novel findings (with the odd exception). That’s not to say that nobody tries to replicate previous findings – existing models are frequently used to develop new techniques, and sometimes contentious findings need to be reviewed by other laboratories before they’re accepted as fact. This is definitely a minority however.

ie Say i have the idea of checking if controlled ragweed pollen extract exposure would actually reduce allergy symptoms. I can then go away and check the database using those as keywords.

If this is done i would find that someone has actually done this but using grass pollen as opposed to ragweed. (i know… i was one of the guinea pigs )

I could then study their results to see how their study was designed and hence design mine to cover any gaps i might feel warrented attention or persue their results in another direction.

This way research would actually complement prior studies and build upon results rather than being another stand alone project.

Think how this would apply in the world of cancer research where you have lord knows how many individual charities supporting large numbers of studies into various cancer types. I’d be willing to bet that 50% of the money that actually gets into research is wasted on double efforts.

It’s still at the developmental stage, and I haven’t got it to work for me, but the principle is sound; often (particularly in neurophysiology) the difficult part is actually getting the data. There are many times in which it would be relatively simple to extract useful information from experiments that examined something else.

If you’re interested, it’s at neurodatabase.org (I read about it in Nature Neuroscience).

Pip pip

The current system of science needs publications, and publications in good papers, to justify research grants, with no small pressure involved. Nor is it a trivial job to write a paper. To produce one that had minimal impact in the Journal Of Near-Irrelevance is not a good use of the effort that paper writing can involve. There’s also a certain amount of ego: no one likes to admit they went through 20 different things to find something which didn’t work. It’s close to saying: “Sorry, I’m s**t, and here’s a record of my failure.”

It’s not like the empirical truth is not emerging. It’s more like what you see is the refined truth rather than the crude mass of data. And I agree science would be better if negative results could be readily presented; it would probably save lots of waste, and give you far more to show that you have actually spent the last 3 years working. But under the current system, don’t hold your breath.

But in the real world, politics is not easy. It is actually much more difficult than rocket science. Scientists often make the mistake of thinking that because science is technically difficult, all other problems are relatively easy. With a doctorate in physics I used to make that mistake myself. It made me feel ever so clever to assume that people who couldn’t solve political problems were stupid jerks. Then I started trying to work on such problems and found they really aren’t at all easy.

I enjoy the “bad science” column, and I think some corrective to the sloppy use of science in advertising and journalism is very much needed. However, I have a sneaking worry that the impact on non-scientists may be negative. It probably makes them feel they truly are lame, and that’s not a nice feeling.

Can we support people who want to think straight without making them feel lame?

Public Library of Science

jun.lemonie.net/JUnChem2001.html

Articles such as “Great Results: How to Achieve Them With Statistics.”

I even managed to get something published….

In a healthy system, twenty people do a study, and only one of them finds that the variable in question is significant with p>0.05. That one publishes, then the other nineteen publish articles building upon the pioneering work of the guy who got lucky, then a metastudy is written finding that the preponderance of evidence indicates against rejecting the null. Since you’re no stranger to the metastudies, I imagine you could give a few examples of a story like this yourself.

It’s a slow and potentially inefficient system, and the positive result always gets to go first, and people who don’t know how to work the “show works that cited this article” button on their lit database think that the positive result is unrefuted, but in the long run it seems to generally work OK.

very, very good point. from my own (perhaps limited) experience ‘negative results’ often arise because of some biological reason, i.e. compound/protein/molecule X doesn’t have a chance in hell of treating condition Y because substance Z stops it from doing so, something that maybe laboratory A has already found out but lab B spends 10k and six months finding out for itself. if lab A publishes it’s findings, even tho it’s kinda negative (i.e. X does NOT treat Y), B wouldn’t have gone to the effort.

this brings us onto whether there are two types of negative results – Type I, where for instance a drug/treatment/process has no therapeutic effect where it was expected to do so (like my convoluted example,above ), and Type II, where instead of/as well as a therapeutic effect, the drug/treatment/whatever produces negative consequences and/or deleterious side effects. currently journals do publish Type II negative results, because they are useful in warning others of unwanted consequences to potential drugs etc.

Type I needs more attention as most of us here seem to be saying, perhaps in the kind of journals that specifically set out to pubilsh negative (or rather ‘not positive) results – like those in Bob O’H's list)

Obviously if a new technique / drug / approach is devised then re trials could be worth it.

True. But speaking specifically about negative results, I’m not sure how much of a problem a lack of peer review is.

More generally, one way around the permanency/peer review is to have a central site where all researchers submit stuff and to allow comments to be sent. This would allow researchers to discuss issues about the research in the same place it is published. Some sort of Wiki set up may be useful. In my mind this would be a good place for poeple to make points reviewers may have missed – I’m sure we’ve all read papers with glaring holes reviewers have missed…

# listed on CV’s,

# considered in academic job applications, and

# counted, perhaps specially and separately, in RAE’s.