Thursday, December 29, 2016

How accurate are temperature forecasts for Dublin

I got a very positive reaction to my presentation about my weather forecast tracking application on Bluemix. Most people who saw this presentation asked which weather forecast service was best. Unfortunately I had to reply to them that I didn't know because I had only just built the app and never used it yet to do any detailed analysis. This post is an attempt start rectifying that error by looking at the accuracy of the various services for forecasting temperature in Dublin over the last month.

At the innovation showcase event a number of people spoke highly of  the Norwegian forecast service yr.no and claimed that it was the most accurate in their experience. Therefore, I decided to add this service into my data collection service (it is a testament to the flexible architecture that this was a relatively easy task) and wait until a month had passed before beginning analysis so that we had  enough data from them to compare.

I chose to initially analyse data from Dublin since this is where I live. I analysed temperature readings and forecasts for the period from 22nd of November to 21st of December 2016. I looked at the actual weather reported as well as the forecast for next day and the forecast for 3 days time. Some of the services offered forecasts for further into the future, but I chose to pick the 3 day forecast because this was supported by all services. If you want, you can see the actual data here.

The thing that most people commented upon at the innovation showcase was the fact that actual temperature data reported by each service did not match up exactly. When I analysed this, I noticed that some of this difference was down to data rounding. The temperature data reported by OpenWeatherMap  and DarkSky is given in celcius with 2 decimal places, in contrast WeatherUnderground and yr.no only give a single decimal place and both WeatherOnline and the BlueMix weather data are reported in whole numbers.

People asked me which service was actually correct, but the truth is that I have no way to tell. What I decided to do was to compute the average of the temperature reported by all services and then comparing the number reported by each service to this average. As you can see from the table below, the average error was actually quite small - not much more  than the average of 0.25 degrees which would be expected from rounding error. Interestingly, the yr.no service has the greatest level of difference from the average.

For assessing the forecasts I looked at the difference between the temperature that was forecast 1 or 3 days earlier and the actual temperature reported on the date in question. I also looked at the forecasts for one week ahead, although this could only be done for 4 of the 6 services since WeatherOnline and OpenWeatherMap don't supply 1 week forecasts.

The average level of error was slightly higher for 3 days in advance  (1.74 degrees) than it was for 1 day in advance (1.41 degrees) and higher again (1.95 degrees) for 7 day forecasts. The most accurate forecasts were from the BlueMix weather service across all 3 timespans. The second most accurate 1 daya forecasts were from the yr.no service although the 3 day and 7 forecasts from yr.no were definitely worse - the 7 day forecasts from yr.no were the worst of all services..


-->
ProviderDecimal PlacesAve ErrFcst1 ErrFcst3 ErrFcst7 Err
weatheronline00.391.201.37
openweathermap20.371.932.21
weatherunderground10.581.962.012.30
darksky20.521.591.791.30
BlueMix00.520.800.801.30
yr.no10.750.962.282.90
Average0.521.411.741.95

-->

Overall, I think the conclusion is that the temperature forecasts are actually quite close to the  actual values, but the details might vary if I looked at a longer time period and/or more cities.

In reality temperature forecasts are relatively easy since temperature doesn't vary widely from day to day in this part of the world. Next I will look at wind speed forecasts since these might be more variable and also because I have been told that they are closely watched by insurance companies since wind speed correlates with weather damage,



-->-->

2 comments:

  1. Hi Brian, I used to be a weather forecaster and a couple of points I would make regarding temperature are:
    1:Temperature can vary significantly over short distances. City center temperature is likely to be different to the airport for example, or any place near the water is going to be influenced by the water in onshore flow situation. So you would need to find out what stations are used to verify each forecast and compare only like with like.
    2: Measurements should also be made using quality approved equipment that is set up according to accepted standards. Equipment used by Met Eireann could be trusted while a private weather station may not be up to standard.

    ReplyDelete
  2. Thanks for the comment Conor - I thought I replied earlier but I must have forgotten to hit "publish".

    All of the data was retrieved by requesting information about the exact same lat/long coordinates. However, the services obviously only have a finite number of weather stations whose location would not exactly match what I queried about. So i guess the different locations of the weather stations used probably was a factor in the differences.

    The Weather Underground service publicly claims to be using weather data coming from amateur stations, but the others claim to be using professional data only.

    ReplyDelete