Norwegian forecast service yr.no and claimed that it was the most accurate in their experience. Therefore, I decided to add this service into my data collection service (it is a testament to the flexible architecture that this was a relatively easy task) and wait until a month had passed before beginning analysis so that we had enough data from them to compare.
I chose to initially analyse data from Dublin since this is where I live. I analysed temperature readings and forecasts for the period from 22nd of November to 21st of December 2016. I looked at the actual weather reported as well as the forecast for next day and the forecast for 3 days time. Some of the services offered forecasts for further into the future, but I chose to pick the 3 day forecast because this was supported by all services. If you want, you can see the actual data here.
The thing that most people commented upon at the innovation showcase was the fact that actual temperature data reported by each service did not match up exactly. When I analysed this, I noticed that some of this difference was down to data rounding. The temperature data reported by OpenWeatherMap and DarkSky is given in celcius with 2 decimal places, in contrast WeatherUnderground and yr.no only give a single decimal place and both WeatherOnline and the BlueMix weather data are reported in whole numbers.
People asked me which service was actually correct, but the truth is that I have no way to tell. What I decided to do was to compute the average of the temperature reported by all services and then comparing the number reported by each service to this average. As you can see from the table below, the average error was actually quite small - not much more than the average of 0.25 degrees which would be expected from rounding error. Interestingly, the yr.no service has the greatest level of difference from the average.
For assessing the forecasts I looked at the difference between the temperature that was forecast 1 or 3 days earlier and the actual temperature reported on the date in question. I also looked at the forecasts for one week ahead, although this could only be done for 4 of the 6 services since WeatherOnline and OpenWeatherMap don't supply 1 week forecasts.
The average level of error was slightly higher for 3 days in advance (1.74 degrees) than it was for 1 day in advance (1.41 degrees) and higher again (1.95 degrees) for 7 day forecasts. The most accurate forecasts were from the BlueMix weather service across all 3 timespans. The second most accurate 1 daya forecasts were from the yr.no service although the 3 day and 7 forecasts from yr.no were definitely worse - the 7 day forecasts from yr.no were the worst of all services..
|Provider||Decimal Places||Ave Err||Fcst1 Err||Fcst3 Err||Fcst7 Err|
Overall, I think the conclusion is that the temperature forecasts are actually quite close to the actual values, but the details might vary if I looked at a longer time period and/or more cities.
In reality temperature forecasts are relatively easy since temperature doesn't vary widely from day to day in this part of the world. Next I will look at wind speed forecasts since these might be more variable and also because I have been told that they are closely watched by insurance companies since wind speed correlates with weather damage,