A basic point of the new paper is the introduction of quality evaluation. But what does this mean and why do I think it is important? Well, for the first question I have to talk a little bit about the background. The common words we use together with quality are assurance and control. Depending on their definition, they are focussing to make the product or the processes, which lead to the product, better. Since the products we are talking about is data, both are focussing to deliver better datasets.
Nevertheless, in peer review we are handling now a different stage, since we are now in the phase, in which we want to quantify the quality. To do this, some points have to be made clear. First is the fact that quality is subjective. Especially, when we think about the peer review process, it is important to keep in mind that this is not an objective process. The quality of the publication entity is defined by the opinion of the reviewers and editor and has therefore inevitably a personal touch. Of cause the same is true for data peer review.
Another point is that the quality assurance can be divided into two phases, which were described in Quadt et al (2012). The first is the technical quality. This part can be relatively well standardised and the controls (quality checks) in this field are well-developed within the data centres. The other part is the scientific quality assurance, which looks at the content of the data. This second part is more complicate to handle and especially with large datasets not simple to achieve. Nevertheless, a dataset with a good quality has to fulfil criteria in both parts and so some tools have to be developed to assist this process.
The proposed tool in this case is the quality evaluation. This process estimates the quality and helps with this the reviewer to give a good statement. Furthermore, it allows more transparency for the process, since all components of this process are reproducible, when the input parameters are known. The optional transparency is important, since it build up trust. Like for traditional publications, where in critical cases the journal is able to publish the review reports, also here the basis of the decision can be reproduced.
As a basis the idea is to create from the opinion of a reviewer a standardised form, which is then tested on the data and delivers a result, which indicate, under the given reviewers opinion, the quality of the data. For this at first some quality checks have to be defined. This helps to automate the process and as long as standardised tests are used it helps also to reproduce the results. The only prerequisite of the tests used are that they deliver a probabilistic result (so indicate quality with a number between 0 and 1). Additionally to the tests, the parameters have to be defined. Afterwards the reviewer should parameterise his expectation, so what does a given test result means in terms of quality. At last a weighting of the test is needed.
By using the given algorithm from the paper it is then possible to create with this one number. It is, when the ranges of the variables are well-chosen, a number between 0 and 1 and show the percentage of quality under the given input parameter. That one number is critical, is self-explaining, especially when the background is not known. It is therefore important, that when those numbers would be published, the input parameters of the tests, the priors and weighting are published as well.
With this relatively simple method it is possible to evaluate and measure the quality. When a second reviewer thinks that the chosen settings are not good enough, s/he could modify them or introduce new tests to show his/her way of testing the quality of the data. All in all it can be used as a powerful tool for detection of problems within complex datasets. This and more will be part of the following posts on the background of this paper.