Last week several journals have published an agreement made on an National Insurance in Health (NIH) workshop in June 2014. It focus on preclinical trials, but allows a wider view on the development of the publication of research in general. Furthermore, large journals, like Science and Nature have accompanied this with further remarks on their view on the future of proper documentation of scientific research, which head into the direction I named “Open methods, open data, open models.” a while ago. In this post I would like to comment the agreement and some reactions from these major journals.
The agreement itself consists of five sections, and at least some of them are also applicable for other fields apart from the medical prime audience. The first point is about a rigorous statistical analysis. It bases on a measure Science has published a while ago, and asks for more rigorous control of the used statistical methods. As commented on this before, this can only the start of several procedures to be introduced within the scientific system. The second point addresses this and focus on reproducibility. They call it transparency in reporting, and is an essential part of future research publications. Parts of it are specific for medical research, but some of this, as “Standards” and “Statistics”, which asks for using of standardised methods and the full report of the used statistical parameters, are well applicable also in other research fields. The third point is data and material sharing. That I see this as a mandatory requirement for the publication of research in some years time I had written before. In fact it only asks directly of data publication, without any quality control requirement or the mandatory publication of raw data. It is therefore only a first step on this long road. The fourth element of the agreement is the availability of refutations within a journal. If this can be seen as an idea to institutionalise ‘after publication discussions’ again (after they are now outsourced to blogs and twitter), cannot be said. Nevertheless, argumentation on an official basis should always be possible. The last point is the consideration of the establishing of best practice guidelines. The examples are of cause very medical related, but standardisation in reporting of research is a key for future research success.
Both, Science and Nature, have reacted to this with editorials. Science focused on reproducibility of research. The focus therein is set on the documentation for the reader and the information for the author of the stated principles above. As this focus wholly on data and not on models, this is of cause not sufficient for other sciences.
Nature used some more room for this topic. The editorial is very similar to the one above, so we can neglect it here. More important are the additional articles. The first article describe the agreement as a success for metascience, which looks at the science of science (so should I call myself from now on a metascientist? ;)). It asks for more openness on the used data and methods. Nevertheless, it doubts that reproducing every study is useful and prefers that by making the procedures more transparent a simpler understanding of the executed research. I personally think that he has a point here, but also have the opinion that we have to invest more in the replication of science in general. The other article I like to address here is named “Code share“. Therein a publication of the computer programs is lobbied. It does not ask for a requirement of code sharing, but simply making it visible to others that the code is available from other sources, when it is available online. I will look forward to the consequences this will have in future, as it is a non-mandatory requirement to publish it. The editor will in future decide, whether code sharing is a requirement for an article in the nature journals, and I fear that this will not bring much success. But it is worth a try. Nevertheless, it also asks for guidelines on this within the communities, so I hope the quality assurance of code will be addressed. And I think, it will be even more important now, when other publishers should follow, to teach the next generation a proper way of coding.
More comments on this topic can be found in the current issue of Nature geoscience, which I might comment in a future post.
All in all, it is an important step forward, that one discipline starts to formalise and standardise procedures for research documentation. But as stated several times in the past here, a lot has still to follow and especially other disciplines needs to adress these problems.
* I will check the availability of articles outside the paywall later and ammend these information within the article.