For everyone working on data analysis in climatological science, using references is essential. These references, representing some form of truth, is often the target, which models have to reach. Verification (or in non-meteorological science validation) methodologies evaluate the results against the references and dependent on the methodology deliver good results when the model is near to it, matches its variability or is close in other statistical parameters. The power of these references in these analysis and defining our knowledge about the world is immense and so it is essential that it really has something to do with things we see in front of our windows.
Last month Wendy Parker published a paper named “Reanalyses and Observations: What’s the Difference” and looked at the references from a more philosophical point of view. She listed four points, which critically looked at the connection between references and observations and in this post I would like to take a look at them.
In general I agree with her points, but I would like to look at it in a bit more detail. The point-list start with the well-known and already in this blog discussed problem that models (and with it reanalyses) and observations are not really different. Both are basing on assumptions and pure observations, without some kind of model, do not exist. The paper rules it as insignificant difference and I think that this is certainly correct. Many people underestimate the flaws of observations and consequently see them as a great reference. Unfortunately, observations have limits to give information about the truth and this leads to the situation, that the full uncertainties of observations within an application can be much larger than the given measurement uncertainties.
The second point covers the fact that reanalyses include some kind of forecasts within their creation process, while real-world observation does not. This is certainly a complicate issue, as the paper is not very clear what is meant with it. Yes the data assimilation procedures require some form of forecasts, as the observations should fulfil the physical constraints given by the used model equations. Anyway, I would also agree here, that this is not really an issue. The open question is, whether there are possibilities to reduce the influence on forecasts within the reanalyses.
Third point is the trouble of an ill-posed inverse problem, which the reanalsses try to solve. I wouldn’t fully agree here with the paper, as I think observations and reanalyses got similar issues here. While it is true that reanalyses try to infer from observations with some form of inverse modelling, observations do similar things. This is of course true for satellites, but also for simpler observational types. Nevertheless, the aim of the observation is to infer onto the truth more directly than reanalyses (which use observations as a step in between), so yes there is a difference between the two types of references, but I do not think that it is this simple. But I would agree, the existing differences are not significant.
The final point is the less well understood uncertainties of reanalyses compared to observations. First of all this is true, reanalysis have hardly ever an uncertainty attached. This is a major problem, which is tried to be handled in the next generations of reanalyses. But I think the problem is bigger. The observations have uncertainties attached, when they get used in the data assimilation of the reanalysis. But the uncertainty due to inhomogeneities of the measurement procedures is often not reflected in the measurement uncertainties. So especially for long-term reanalysis, which run into the trouble of low amount of input data (well, honestly in ocean reanalysis this is true even today), even uncertainty estimates basing on the observations and the model uncertainty might be too small. Overconfidence with references is a major issue, and attaching uncertainties to reanalyses can only be the first step. The paper highlights this problem by showing an example that uncertainties do not really add up. And yes, this final point is the most important.
As such, I think the paper is important, as it summarises parts of the issues of our references in climate science quite nicely. It also shows, that a philosophical approach is important in science and I hope it gets more common to use this form of view in the climate literature.