Wednesday, April 27, 2011

Publishing negative results

Recently, I found my way over to the All Results Journal (now available in Biology, Nanotechnology, Physics, and Chemistry flavors, though issues are only available for Chemistry and Biology). In their first editorial, the editors of All. Res. J. Chem. explain that:

We are pleased to introduce you to The All Results Journals: Chem (All Res. J. Chem.). A very particular journal, as it publishes fully indexed chemical articles and reviews that challenge current models, tenets and dogmas. This journal represents the first open access source for chemical research concerning negative results and will be a valuable resource for researchers all over the world, including those who are already experts and those entering the field.

The All Results Journals: Chem immediate goal is to provide scientists with responsible and balanced information in order to avoid unproductive synthetic routes, improve experimental designs and economical decisions. Many journals skew towards only publishing “positive” data; that is, data that successfully proves a hypothesis. The All Results Journals: Chem is the home for negative or “secondary” data: experimental documentation of hypotheses that turn out not to be true, or other experiments that do not lead to an advance of a specific hypothesis but are nevertheless a true rendering of that experiment. For example, if a researcher sets up an organic reaction and a variety of molecules do not react in exactly those conditions, it would be very useful for other researchers to know this (to avoid time and wasting money).

The journal editors are apparently targeting three types of difficult to publish data: experiments that fail, experiments that are incomplete or inconclusive, and experiments that disagree with current mainstream scientific understanding. I am not sure I really see a value here, though maybe I am just getting old and crotchety.

Experiments that fail. When I was a student, I thought publishing failed experiments would be a very valuable resource. With more scientific experience, I realize that many groups will wonder whether it was the experiment or the execution that failed, especially for negative (or inconclusive) results that go against scientific intuition or prior experience on related systems. Publishing this sort of failed experiment MAY help, but may not.

Experiments that are incomplete or inconclusive. Personally, I don't think incomplete or inconclusive experiments really should be published. Yes, there is lots of data languishing in notebooks, but that is at least in part due to lack of motivation and/or time to write the (probably minor impact) results up. Incomplete experiments can be misleading, and inconclusive experiments either need rethinking, better design, better instrumentation, or more data. Both of these situations are likely caused by funding running out or someone who was working on the problem leaving the lab. This leads straight back into lack of time/motivation to write.

Experiments that disagree with current mainstream scientific understanding. I am not sure we need a new journal for this. Shouldn't current journals be eager to publish results that conflict with current understanding (and therefore advance science)? I know that truly new, truly innovative stuff sometimes has a hard time getting published, but still extraordinary claims should require extraordinary evidence, shouldn't they?

I totally agree that the literature is biased towards positive results. But many negative results are currently available (and more and more frequently searchable) in Masters and PhD theses. It will be a real shame when dissertations become introductory and concluding chapters wrapped around published papers (as is happening more and more). So maybe we do need something new? With the barrier to Web publication low, I certainly admire the editors for giving it a try!

Now the All Results journals may end up being an excellent resource for scientists moving into new areas or pushing the edges, but I have my doubts. The cynic in me thinks that it will mostly be a CV-padding tool for future academics.


Chris said...

Of the three aims, the only one I really see value in is the "negative results" option. It could help a lot of people who might otherwise duplicate the negative results. This would save money and time (which is money) for all involved...

Anonymous said...

Is this new journal peer-reviewed? At long last, we may have a proper use for that backhanded non-compliment which it can be so tempting to put into reviews: "This paper fills a much-needed gap in the literature."

Hermitage said...

At least in my area, no. 1 is a nonstarter because while three labs may claim to do triple backflips and a handstand there will be another four that say that it's bullshit and they're measuring it wrong. I firmly disagree with the idea of publishing inconclusive results, makes no sense to me. That just leaves no. 3. I suppose 'Journal of Paradigm-Violating Studies' has a certain ring to it ^^.

prodigal academic said...

Hermitage sums up my thoughts on a journal of negative results--many groups just won't believe it and will attempt the experiment anyway.

Theoretically, publications are peer reviewed. So refer away!

Thanks for the laugh--I would totally read the Journal of Paradigm-Violating Studies, but I think the issues will be short (if it is peer reviewed for real :-)

Anonymous said...

I've been reading your blog for a little while now and find it quite interesting, just haven't commented yet. I was just curious as to why you think it's a shame that more theses and dissertations are following the published-papers-surrounded-by-an-intro-and-conclusion model. What drawbacks do you see to this way of doing things?

prodigal academic said...

The main reason I dislike the "published papers surrounded by an intro and conclusion" model is actually because of the bias in the literature against negative results. By negative results, I mean things like drug X had no effect on condition Y. A significant (but unknown) percentage of experiments end this way, but because it is really difficult to get such a result published unless it is crammed into a paper about something else, these experiments can end up repeated multiple times (a huge waste of time and money). In a traditional dissertation, this sort of experiment is easy to include.

I don't have a hobby horse for the "good ol'days"--in fact, my own dissertation contained some published paper chapters. Mine was more of a hybrid, and also contained chapters of unpublished (and unpublishable) work. I think there is something to the argument that a dissertation should represent as much of the PhD work as possible, and not just the publishable chunks.

Anonymous said...

Makes sense. Thanks for the explanation!

Lewis said...

nice post! I had a look at the website and discover the blog ( they are running, very cool too!
Thanks for sharing it!