Not even wrong?

Quantifying uncertainty in economic numbers, past, present and future.


Which way are we going? (with thanks to Andrew Boag)

The chart below shows a set of forecasts for spending on working age disability benefits. They didn’t go well.

Graphic by The Institute for Fiscal Studies.

I call charts like these ‘cat-o-nine-tails’: they flail at the future.

A quick look into my stash of graphical curiosities turns up this one in similar style – a few years old now, but a belter.

It shows the projected number of births in the UK. Again, projections fail all over the place (let’s not get into an argument about the difference between projections and forecasts – we use them all to guide our expectations).

It’s true that once or twice the dotted lines actually came close to the outrun – at least for a while. But hear anyone celebrating those moments, and ask if it was luck or judgement. One forecast shown in the first of these charts – by the UK’s Office for Budget Responsibility (OBR) – said in 2013/14 that spending on working-age disability benefits by 2017/18 would be about £7bn. The outrun was closer to £11bn. That’s a lot wrong in not long. Things that were generally expected to go down, went up, massively.

Governments need to plan, and so they’ve no choice but to produce numbers, and the OBR does that for them with as little political interference as you could wish. I’ve no doubt about its independence – or competence, actually. And in fairness we should add that this was an unusually bad case.

‘… for the easiest of open goals against the human pretension to know what’s going on, study forecasting…’

The Hidden Half, page 220.

The question is whether an extreme case of wrongness teaches us anything. ‘Well, of course, we know this – all forecasts can be wrong,’ say forecasters, in that way of having their cake and eating it by making an abstract declaration that we hear so often we don’t really hear it at all. But it would be useful to be reminded quite how wrong.

The lesson, then, is not so much that the numbers can be wrong – that’s inevitable to some degree – it’s how we alert people to the size of the possible errors when these include the occasional howler. What’s the best way of representing this possibility?

How late are the trains?

A standard way is to talk about the average forecasting error, or average revision to the data. So it’s commonly said about GDP data for example that the mean absolute revision of quarterly GDP growth figures is 0.1 to 0.2 percentage points, a small-sounding number which makes them appear pretty accurate.

But think of catching your train to work in the morning: how late it is on average (say 5 minutes), might not make much difference, and doesn’t tell you what you really want to know (that about once a week it’s 25 minutes late and the rest of the time it’s ok). Knowing the latter might change which train you catch, the former might not.

So if you’re inclined to dismiss this particular set of numbers about in-work disability benefits as unusually bad, the first question to ask would be: ‘how unusual?’ Maybe the numbers are usually a lot more accurate than that. But then, the train is not usually 25 minutes late. So some measure of the fact that one time in five it’s a lot late – a vague sense of the distribution of errors, in other words – might be a useful corrective to the kind of over-confidence that otherwise develops if we depend on averages, which tend to flatten out the errors.

When it’s the extremes of error that cause the biggest problems, using communication that disguises them doesn’t sound to me like the best way to do it. In the book, I talk about the example of GDP revision, which over the long-term is uncomfortably big uncomfortably often, even though the average revision might not frighten anyone. This average revision – typically of 0.1 to 0.2 percentage points on quarterly growth figures – sounds harmless enough… until you know that quarterly growth itself is often only about 0.3 per cent-ish these days, and that there are revisions of 0.4 or more in about 40% of quarters. In other words, big revisions come along often.

Are we frank enough at the outset about the true extent of the inevitable uncertainties in all data? Do we properly capture the potential for waywardness?

Because the practical problem is not how wrong the data or the forecast tends to be on average, it’s how wrong this set of data could be that I’m looking at right now. How do you know if it’s averagely wrong, or extravagantly wrong? You don’t. And for a sense of that, it can’t be said often enough: our numbers, forecast or otherwise, can be and often are, simply miles out.

Should we say on the next forecast of working-age disability benefits that most forecasts for it have been wrong not merely in degree but in direction?