During this week discussions on twitter and on the blogs focused on a discussion paper by James Hansen et al in Earth System Dynamics Discussions. The paper forms part of a legal case in the US and basically states that the current and expected warming over the next decades is unprecedented since the last interglacial. In this context the Guardian has run an article on the paper. While they state that the science is not yet peer-reviewed, the authors have run a series of interviews and comments, which is usually happening only when a paper is actually published. As usual I refrain in this blog from commenting on climate politics, but as I have written my PhD on scientific publication processes I would like to focus here on the implications of the media scrutiny within the discussion paper phase of a scientific publication.
To understand the problematic of this action it is important to explain what a discussion paper actually is. Discussion journals came around a few years ago. They form part of an open-peer-review process, in which a manuscript is published before a peer-review took place. On this anyone is allowed to comment and two or more invited reviewers perform the actual reviews by publishing their reports openly (but anonymous) alongside the other comments. It is important to understand, that the discussion papers, even when they seem to be published in a (discussion) journal do not form part of the accepted published science. They are comparable with a manuscript under review and are actually in a process I have described in Quadt et al. 2012 as quality assurance. This step lie between the actual analysis and the peer review and it is suppose to enhance the quality of the paper. It has to be seen independent of the peer review, as this process incorporates a judgement on the scientific validity of the presented science on whether it is worth to be published in the specified journal.
So far this process is useful, it might deliver better paper, it helps in the transparency of the process and still the peer review process happens classically. Consequently, the resulting paper can be accepted formally as a scientific publication as it is established for centuries (even when the actual establishment of peer review (so the requirement to perform the process prior to an accepted scientific product) is just a hundred years old). But what happens now when this process is under massive public scrutiny, potentially under the influence of political interest groups? I will argue in the following that a non-standardised influence of the quality assurance on the peer review process can lead to manipulated science and so an endangering of the valid peer-review status of a resulting paper.
Why is it so important that the two processes, quality assurance and peer review are separated? As I had developed the terminology in context of data publication, it might be the best to illustrate it with data. So assume we have a dataset under review and additionally a quality assurance like quality evaluation is performed on it. Myself have developed schemes, which argued for the application of standardised tests in the QA process, also to help the reviewers in their tasks. But what happens if when the tests are manipulated? What if instead of the standardised testing, outsiders with personal interest change the parameters of the tests without the reviewer noticing it? Can a reviewer in this case perform a good quality assessment required for the review report? Probably not. Same is true within the discussion paper phase. A reviewer expect that the other comments are written by fellow scientists in the best interest. This is the assumption, which makes the process a standardised one. As soon as the comments do not reflect anymore this assumption the process becomes tricky. Reviewers might be guided by flawed comments, and the additional pressure due to the press reports can endanger the independence of the review reports (what happens if a reviewer does not reject a paper, because he/she feels moraly obliged by the press reports to accept it (or the other way round)). This is comparable to have a jury in a court case under public spotlight and before the verdict the jurors get the current tabloid press into their hand, including a rich comment section where everyone can communicate their believes. Would we trust the judgement?
Therefore, the danger is too hure that the final decision is flawed, even when I trust that reviewers act in the best interest of science. Media scrutiny has no place within the discussion paper process, it endangers science and poses the big question, whether the peer review process at this point should be concluded at all. In the past years we had to make new experiences with new forms of publication processes, but most, at least all accepted, are basing on classic principles. A flawed discussion paper phase, as it is possible in this case, breaks with this view. Manipulation is always possible, be it within peer review or any other step of the creation process of a scientific publication, but when it is encountered, usually the paper get retracted immediately. Here the manipulation of the process is not yet proven, but made possible by the actions of the media and those commenting publicly on the discussion paper. I am sure the journal and editor, as they have already announced in the discussion phase, will try their best to keep the discussion phase within limits. Anyway, there are guideleines for the discussion paper phase to minimise the risks and they should be followed:
- do not cite unpublished work (and discussion papers are unpublished and uncitable work) as an important evidence
- refrain from reporting on the process as long as the peer-review process is not yet finished
- as a scientist do not comment on the content of the manuscript of the review apart from the comment section of the discussion paper
- as author refrain from doing PR for the paper apart from generally inviting an audience to participate in discussion phase
Without these simple rules, the open peer-review is at the risk to be flawed and this might lead to manipulated science.