Apr 11 2008
Skill in 2008 Atlantic Hurricane Forecast?
It’s been a slow climate news week, so today’s post will be about the weather. Every year in April, Dr. William Gray’s group at Colorado State University puts out an Atlantic hurricane forecast. The news media usually gobble this up like it’s actually news. I haven’t heard anything yet this year, but I think that’s because I’ve been watching less of the news on TV. This year is no different; they predict that there will be 15 named storms, 8 hurricanes, and 4 intense hurricanes.
On their website, they provide an archive of their forecasts from 2001-2008. The number of actual storms is also available, so it’s possible to assess the skill of their forecasts.
|
The figure shows that from 2000-2004, their forecasts underpredicted the number of named storms by a small amount. In 2005, the prediction was off by a factor of 2. The last 2 years, they have overpredicted the number of named storms. A similar pattern repeats for both the number of hurricanes forecast and intense hurricanes.
I’ve added the mean value of number of 2001-2007 storms, hurricanes, and intense hurricanes as a thick line; green, orange, and red, respectfully. The large dot is their 2008 prediction, and it is exactly the same as the mean for the last 7 years.
Something interesting appears when we plot the predicted values versus the actual values in a scatter plot. You would expect that if they forecast a higher than average hurricane season then the actual number of hurricanes would be higher than average. You would be wrong.
The three plots below show that over the last 7 years when the Gray group at Colorado State predicted a higher number of names storms, hurricanes, and intense hurricanes, in general the opposite occured.
|
|
|
All three plots show a negative slope. This is not the best way to assess whether their forecasts have skill. Instead, let the null forecast be the mean value of the number of named storms. We can then plot the absolute difference between the predicted value and the actual value for both the null forecast and the Colorado forecast.
|
Here the + symbol represents the absolute difference between the null forecast and the actual value of named storms. The circles represent the absolute difference between the Colorado forecast and the actual value. For 7 out of the 9 years, the null forecast was better than the Colorado forecast. During the 9 year period, the Colorado forecast had a total absolute difference of 38, and the null forecast had a total absolute difference of 25 storms.
This analysis of absolute difference between the number of named storms and the prediction is also the same for hurricanes and intense hurricanes (not shown). That is, the null forecast of predicting the mean value would produced a more accurate hurricane forecast.
Only years 2001-2007 were used because that is all the forecasts archived on the Colorado State website.
A better null forecast would have used the average of the previous X years instead of the same value. I.e, the prediction for 2000 would use the average from 1990-1999.
(No Ratings Yet)
Loading ...Related Posts:
16 Responses to “Skill in 2008 Atlantic Hurricane Forecast?”
To reduce spam, comments are automatically closed 30 days after the last comment. If you would like to comment on any closed thread, please use the contact form at the top of this page.
I have empathy for hurricane forecasters. it’s an inexact process at best. There are way too many variables to be very accurate. Too many last minute changes, like dust storms and volcanoes can effect the numbers. They should give up on this quarterly forecasts and go to one say in July and let it go at that …
How do they account for the tiny tims? This has become a sport of packing the numbers to give desired maximum fright results.
As I see it, all that these forecasts are doing is scaring people needlessly and driving up insurance costs.
Interestingly those who deny skill in climate modeling defend skill in hurricane prediction. In both cases we need to carefully define skill.
Oh yeah, using past experience for prediction is a good method until you move out of the range of the past climate, which is where Gray’s methods will break down. It is also a problem to using such methods to understand what is driving the system. OTOH, climate models may not be as good as say, neural networks, in detailed prediction (the problem of regional climate) but they give you a good picture of the drivers (what if experiments for example).
“Interestingly those who deny skill in climate modeling defend skill in hurricane prediction“.
Hi. Do you have a link to anything to support this? I’m struggling to think of anyone on the sceptical side ever saying anything like it.
RP Sr.
Bill Gray for that matter.
Interestingly, those who assert that Shakespeare’s works were written by Bacon almost always accept that London is south of Edinburgh. Scientists have enquired at length as to why, and recently are coming to the conclusion that it could be because the questions are independent.
Fred, both are models. Has no one told you that the party line is that models are evil?
Not all models are equally good or predictive. It depends on the model and on what is modelled. We can model auto engines rather well. This is no reason to think we can or cannot model long term climate. Whether we can predict hurricanes has no bearing on whether we can predict the long term climate. They are different things.
So, its quite reasonable to take one view of Gray’s ability to predict hurricanes and a different one of the IPCCs ability to predict climate. Just because you do not trust the IPCC models does not mean you should not trust Gray’s. Or the other way around. Atmoz, for instance, probably accepts the IPCC models as valid, but is here showing why Gray’s are not.
There is nothing surprising about this, any more than there is about people liking beef and not mutton. Different things are different.
No Fred, the attack has been on “models” as in “Evolution is only a theory”. You don’t read enough of the climate audit lit.
Thank you for your good work and all the many useful information on this blog. You are writing that “it has been a slow week” leaving me with the feeling that you´d prefer some action.
I think there are plenty of action every secound around the clock. You are making a great inpact on the climate crisis and the more we all do to solve the climate crisis the better it will turn out.
I have come to understand it is a great importance that we start telling each other what we want. What is it we realy want? I know that Albert Einstein wrote “Is this Universe a good Universe?”. I think he wrote those words to let everybody answers it by themselves.
The Universal Las of Attraction was known by Einstein as well as many other great people (Beethoven, Lincoln, etc). They all knew that we get exactly the things we ask for. Hence, let´s say to each other that we can solve the climate crisis together.
This is a clever vision. We must say what we want because this is what we will get.
Eli, I don’t know which Party you are talking about that has the line in question.
What I do know is that the accuracy of Gray’s models has no bearing whatever on the accuracy of the IPCCs models, and that reasonable people may take different views of the accuracy of either one without fear of inconsistency.
So if the point of the jibe is to suggest that it is inconsistent to think Gray right and the IPCC wrong in their models, no it is not. It may be mistaken, either about Gray or IPCC, or both, but it is not inconsistent.
There is one huge difference between Gray’s predictions and the IPCC: Gray provides predictions that can be demonstrated to be true or false within a reasonable time frame. The IPCC only provides predictions which cannot be falsified with anything less than a new ice age. The lack of falsifiability automatically makes the IPCC predictions less interesting or relevant.
At least we can compare the IPCC predictions somewhat around the time of the next El Nino. It won’t be perfect, but it will be indicative of how good the models are and it won’t be too long to wait before having to ‘act’, if needed.
Another modeling difference: Gray also only makes one set of predictions. Certain other folks will tout any GCM result that comes close to producing the “right” answer and say, “See? The models are right!”
Also, looking at Klotzbach and Gray’s 2007 summary, I see that they have no problem admitting when their forecasts were “not successful.”
Rabett, the attack has been on GCMs, not ALL models. Many of the attacks (including those on climate audit) come from people with experience in models (weather, climate, or otherwise) themselves. Logically, wouldn’t those be the folks worth listening to when it comes to the limitations of models?
Normally, I would jump on your case for spreading such misinformation. But considering the poor reviews you get from your students, it does not appear that much of what you say makes it through to others in the first place.
[...] a forecast can be accurate, but have no skill. I’ve discussed this before in relation to the seasonal hurricane forecasts having no skill when compared to climatology. Although, they continue to get press coverage every [...]