Earlier this month the BBC ended its weather forecasting contract with the Met Office and changed over to a new provider - commercial operators Meteo Group. People hate change, and not every aspect of the switchover has been welcomed. But what interested me most were these three 'improvements'...
• More locations: The BBC has added thousands of new locations, including many international, to our database
• New 14-day forecast: Users can view 14 days of hourly forecast data for UK locations and major international cities
• New forecast features: Added data fields like ‘chance of rain’ and ‘feels like’ temperature
So I thought I'd give the new 14-day forecast a proper test drive.
I started 14 days ago, on 1st February.
Every day since then I've checked the online weather forecast for 14th February, and made a note of what was predicted.
I've been using the weather forecast for Bow (which has different figures to Stepney, Poplar, Stratford and the generic London page). I have a lot more data than I'll be showing you here. And I can confirm that the concept of a 14-day forecast is definitely suspect.
It turns out that 14th February was an excellent day to have picked. Yesterday started off sunny and dry. In the middle of the day a band of rain arrived, lingered all afternoon and became heavier in the evening. The day ended cloudy and wet. With all sorts of weather to predict, and a very specific arrival time for the band of rain, there was plenty for the new forecast to get right.
First of all, the overall summary. The summary for the actual weather yesterday was "light rain and breezy", and the temperature for most of the day was 6 or 7 degrees.
Here's what the BBC Weather website predicted for yesterday's weather on each of the preceding days.
1st Feb: Light rain and breezy (3-8°C) 2nd Feb: Light rain showers and breezy (3-8°C) 3rd Feb: Sunny intervals and breezy (4-9°C) 4th Feb: Light rain showers and breezy (4-9°C) 5th Feb: Light rain and breezy (4-9°C) 6th Feb: Light rain and breezy (5-10°C) 7th Feb: Light rain and breezy (5-9°C) 8th Feb: Light rain and breezy (5-10°C) 9th Feb: Light rain and breezy (5-9°C) 10th Feb: Light rain and breezy (7-10°C) 11th Feb: Light rain and breezy (7-9°C) 12th Feb: Light rain and breezy (7-8°C) 13th Feb: Light rain and breezy (6-7°C) 14th Feb:Light rain and breezy (6-7°C)
So that's pretty good. The forecast was right on 1st February, then wrongly optimistic on 3rd February (the only day when a 'sunny' symbol was displayed). But since 5th February the text summary for 14th February has been correct, which may be a coincidence, or is damned good going. The temperatures weren't quite so well targeted, and only in the last few days did the range close in on "cold, and not varying much". But overall, if you were planning an outdoor event, thumbs up.
But the most revolutionary feature is 14-day hourly forecasts, so let's dig into how they performed.
What these tabular graphics should have shown for 14th February was a dry morning and a wet afternoon, with the first rain arriving around noon. Here's what they actually predicted on the previous 13 days.
1st Feb: sunny intervals, then light rain from 2pm 2nd Feb: sunny intervals, with showers around 2pm 3rd Feb: sunny intervals all day 4th Feb: sunny intervals, with a shower at 1pm 5th Feb: light rain all day 6th Feb: light rain all day 7th Feb: light rain all day 8th Feb: sunny intervals, then showers from 1pm 9th Feb: light rain all day 10th Feb: light rain all day 11th Feb: cloudy, then light rain from 2pm 12th Feb: clouding over, then light rain from 2pm 13th Feb: clouding over, then light rain from 2pm 14th Feb: clouding over, then light rain from 12 noon
And that's less good. The general shape of yesterday's weather only became clear on 11th February, having been too pessimistic for the previous six days, and was way off on 3rd February. That 1st February forecast still looks good, but with so many other different predictions floating around, who was to know at the time?
What the forecast never got right was that the rain would arrive at noon. Indeed even at 10am yesterday morning the BBC website was still confidently predicting the first drops would fall at 2pm. If the timing of a band of rain is that hard to predict even two hours ahead, why is anyone bothering a fortnight earlier?
And this is where another innovation, the ‘chance of rain’, is supposed to help. Meteo Group's computers churn away to produce a probability estimate of rain at a particular point in the future, and these are also displayed on the BBC website up to 14 days in advance.
0% = definitely dry
100% = definitely wet
20% = "out of 100 situations with similar weather, it should rain on 20 of those, and not rain on 80" (so, probably dry)
Here's the ‘chance of rain’ for noon on February 14th, as predicted over the previous fortnight.
Being probabilities, it's never possible to say these were definitely wrong. But they're certainly not very good probabilities, with every single prediction suggesting it'd probably be dry at noon, whereas it fact it was just starting to be wet. The really bad forecast is that for 13th Feb, the day before, with a wildly over-optimistic 4%.
What's particularly intriguing is how the percentage for "rainfall at noon" changed over Valentine's Day morning.
14th Feb (9am): 9% 14th Feb (10am): 14% 14th Feb (11am): 81% 14th Feb (noon): 96%
If you'd checked the weather forecast at 10am, you'd have assumed it probably wasn't going to rain at noon. But by 11am a massive recalculation had taken place, and now it very probably was.
BBC Weather's FAQ page says "As MeteoGroup forecasts take advantage of hourly updates, which include real-time information from radar, satellite, and nearby weather station observations, you may notice the probabilities changing in the short-term (next 2-3 hours)." And change they do, indeed I can see it all over yesterday afternoon's figures, with percentages suddenly shooting up above 50% a few hours before the rainfall they're supposed to be predicting.
Another thing I've noticed is that the BBC's former forecasters, the Met Office, appear to calculate these percentages very differently. Here's their rainfall forecast published at 11am yesterday morning, with Meteo Group's forecast lined up underneath.
For a start, the Met Office only give values to the nearest 10%, which seems much more sensible than Meteo Group's spurious accuracy. More importantly, Meteo Group's percentage hits 81% at noon before settling into 30-60% for the afternoon, whereas the Met Office sticks to 10% until 4pm, then shoots up to nigh-certain at 6pm. The weather symbols are wildly different too. Why do these two predictions vary so much?
What seems to be happening is that the two forecasters have different opinions on what counts as rain. Yesterday's rain was only spits and spots from noon until around 4pm, indeed you might not even have counted it as proper precipitation. Then more showery rain continued until 6pm, at which point a much heavier band of rain arrived and continued for most of the evening. Meteo Group counted all of this as "single raindrop", with no hint of how wet it was going to be. But the Met Office differentiates a lot more, with 'spits and spots' not registering at all, and more relentless rain meriting a "double raindrop". I'd far rather have seen the top forecast than the bottom forecast if I'd been heading outside.
It seems the rainfall symbols in the BBC's online weather forecast don't mean the same as they did before, and you're a lot more likely to see a single raindrop when previously you'd have seen two, or none at all. Meteo Group appear to want us to forget intensity and learn to use their percentages instead, but when those percentages are often wildly inaccurate until a few hours beforehand, and most of the population doesn't understand numerical probability anyway, that doesn't seem very likely.
As for hourly weather forecasts 14 days in advance, these appear to be little more than a gimmick. I'm basing this on a single day's analysis, of course, which might be considered scant evidence. But when the BBC website is predicting that the whole of the last week of February will be "sunny intervals (3-9°C)", perhaps take that with an enormous pinch of salt.
