Thursday, March 23, 2006

Comment on Frame et al

As I hinted recently, I've got a few things to say about Frame et al: "Constraining climate forecasts: The role of prior assumptions", GRL, 32(L09702) (F05). In fact, jules and I submitted them as a comment to GRL a couple of weeks ago.

I'll start off with the bits I like: F05 give a nice demonstration of how the choice of prior can have a significant influence on the resulting pdf in a probabilistic estimation, especially when the observational evidence is quite weak. Furthermore, markedly different results can be generated by different, but apparently natural and plausible, ways of describing initial ignorance. Therefore the same evidence could be used to draw quite different conclusions due to what are ultimately fairly arbitrary choices. It's obviously an important point that doesn't appear to have been adequately considered in at least some previous work.

If they'd stopped there, I'd have had nothing to complain about. But now on to the bits I don't like so much. Firstly, they present what they see as the solution to this problem - they assert that we should choose the prior to be uniform in the variable which we are trying to estimate (ie uniform in climate sensitivity, if we are wishing to estimate this). This, in their words, "resolves" the "arbitrariness" and allows them to generate what they describe as "objectively determined" estimates. "Objective" is a dangerous word to use here, as the probability cannot be objective in the sense of a frequentist probability - what they presumably mean is merely that they are providing an automated rule that removes this element of choice from the procedure (or perhaps, imposing their own subjective judgement in place of anyone else's). But that's not the biggest problem I see in their suggestion. Where their method really falls down is that it generates results which are not self-consistent. As we demonstrate in our comment, their method will generate the pair of results P(X>3)=2.3% and P(X4>34)=7.8% from a single observation of X=2+-0.5. Under any standard definition of Bayesian probability, P must be a function, which (again by definition) means it must be single-valued. But X > 3 and X4 > 34 are precisely the same proposition (there's no sleight-of-hand with negative values here: X is positive definite, and I could equally have used X3 or X5). Therefore their P cannot be a probability at all!

There are some generalisations of probability (Dempster-Schafer theory) in which probabilities are defined as taking a range of values. Elmar Kriegler is the only person I know who's gone any distance down this path within climate science. Arguably, this provides a better framework in situations of deep uncertainty, but handling these issues correctly is far from trivial (note that a uniform prior in X does not actually represent a state of true "ignorance", but rather the specific belief that 10 < X < 20 is ten times as likely as 2.5< X < 3.5, for example). It is not at all clear to us that F05 have provided adequate theoretical justification and underpinnings for what is in fact a rather drastic challenge to the standard view of Bayesian probability, and they certainly haven't (IMO) drawn sufficient attention to the radical implications of their work.

The other complaint we have about their paper is in their description of the sort of problems that we are all attempting to answer. They say:
Unless they are warned otherwise, users will expect and answer to the question "what does this study tell me about X, given no knowledge of X before the study was performed?"
(and they use "no knowledge" to justify their choice of a uniform prior). In context, this could reasonably be re-written as (A) "what would our estimate of climate sensitivity be, if we had no data and knowledge other than that directly considered by this study?"

However, it seems clear to us that what users really want to know is (B) "what is our estimate of climate sensitivity, using all of our data and knowledge?"

The answer to question A will necessarily have greater uncertainty than the answer to question B. If someone wants to generate an estimate of climate sensitivity, they should use all of the data, either by explicitly considering it, or by the use of a prior which encapsulates (as accurately as possible) the information which the study doesn't directly look at! This is precisely the issue that our recent paperaddresses, so perhaps it is a bit harsh to pick on F05 in this respect (rather than the numerous other papers that have apparently mixed up the two questions in a similar way). On the other hand, these guys are specifically presenting a theoretical analysis of probabilistic estimation, together with recommendations as to how we should all go about it in future (rather than just having a go at producing an estimate themselves), so it's surely more important that they get it right. We certainly don't think that their opinions should be accepted by default, without some meaningful debate over the issues.

Inevitably, given the space constraints of a 2 page comment, it is hard to get the points across clearly without running the risk of appearing overly hostile. That's life, and I'm sure they have thick enough skins to cope. Indeed, depending how they reply, our comment might end up in the bin anyway - unlike most papers, where I only have to convince some neutral referees and can therefore be pretty confident of publication, in this instance there is (at least potentially) an opponent who will try their best to point out weaknesses in our case.

Both these comments, and the contents of our recent paper, are summarised on our poster for the EGU in a couple of weeks. It will be interesting to see how they go down. Unfortunately I'll not be there, so jules will have to face the angry horde by herself :-)

6 comments:

Anonymous said...

Re the quote from Frame et al

Unless they are warned otherwise, users will expect an answer to the question "what does this study tell me about X, given no knowledge of X before the study was performed?"

How could one do a study of X with *no prior knowledge* of X? To me, the one of the most conceptually appealing characteristics of Bayesian statistics is that it formalises the idea that there is always prior knowledge.

Anonymous said...

We're writing a reply, and are supposed to have it done by next week. Should be in roughly on time.

The Angry Horde will probably be diverting its attention to supping pints in Vienna's excellent bars, so I don't imagine Jules will have to contend with very much.

Dave Frame

Anonymous said...

>However, it seems clear to us that what users really want to know is (B) "what is our estimate of climate sensitivity, using all of our data and knowledge?"

I would have thought that what users really want to know is (C) What are the values/probability distribution of (and difference between) our estimate of climate sensitivity using all of our data and knowledge including this study compared to our estimate using all of our data and knowledge excluding this study.

That way, everyone has to apply Annan & Hargreaves technique :)

And the chances of a London Metro style interpretation of the climateprediction.net initial results paper instead of my interpretation wiki would be reduced.

crandles

James Annan said...

Anon,

Agreed.

Chris,

In principle maybe, but given that many analyses re-examine data that have already been used in one form or another, it may be hard to disentangle what exactly the prior is...if you assume that we already agreee that sensitivity is "likely" to be in 1.5-4.5, then the vast majority of "observationally constrained" papers appear to have told us essentially nothing about sensitivity!

Anonymous said...

Is it possible to read the Frame et al reply?

crandles

James Annan said...

Not until they write it :-)

Of course it's up to them how public they want to make it anyway. I understand Dave at least is at the EGU in Vienna, which may cause some delay.