Atmoz

Obsession with Copper

I look a lot at Copper. The main reason is that I originally started collecting data at Copper and therefore have a lot more data there. I added other resorts later. When looking at statistics, it’s useful to look at a lot of data, and sometimes it is helpful to look at outlier cases. Last storm had a day at Copper that recorded 16″ of snow. That definitely an outlier.

copper_error_distribution_01_december_2014
In this post, I’m again going to be focusing on Copper. First, let’s look again at the forecast error distribution. This is produced by taking by subtracting the actual snow value from the forecast. Therefore, if the actual is higher than the forecast, it will produce a negative number, and vice versa.

The most obvious thing about the distributions to the right is that both the National Weather Service and Opensnow do a good job keeping their average error around zero. Snowforecast.com does an okay job and Forecast.io and Snow-forecast.com routinely under-forecast.

There’s more data for NWS and Opensnow since I originally only tracked those two sources.

There are some huge outliers, both positive and negative.

To learn more about the distribution, I ran a few simple statistics which are displayed in the second image to the right.

copper_error_stats_01_december_2014

If we look just at the mean of the error distribution, Opensnow has the best accuracy, having an average error of only 0.092″. That’s pretty good. The median value is 0.5″ though, so that means that the distribution is skewed. We see that the skewness is negative with a value of -1.622″. That means that distribution has a longer tail on the negative side.

All the sources have a negative distribution, though Opensnow’s is the largest. They also have the largest Kurtosis, meaning that the distribution is broader than the standard normal distribution. A standard normal distribution has a mean of zero, standard deviation of one, skewness of zero, and kurtosis of zero. Having large values of skewness and kurtosis mean that data have a bias and is indicative of poor forecasts.

NWS has smaller values for skewness and kurtosis than Opensnow, but a larger standard deviation. Trying to understand what this means in terms of answering the simple question “who has the best forecast?” is difficult. Having a small standard deviation is obviously good. But since the mean, standard deviation, skewness, and kurtosis are all related we can’t just say that Opensnow is the best because it has the smallest mean and standard deviation. If their distribution had a skewness of zero and kurtosis of zero then it would obviously be the best. But NWS is less skewed and has less shoulder.

One thing that can be unequivocally be said: Forecast.io and Snow-forecast.com are really bad at predicting snow. Snowforecast.com comes in a distant third to Opensnow and NWS. I would say that NWS and Opensnow do equally good jobs at predicting snow at Colorado resorts.

This is confirmed in the last two columns of the table. The Bust % is an admittedly arbitrary statistic of when I personally feel that a forecast is a bust. For example, if a forecast calls for 6″ of snow and it turns out there’s 16″, that’s a busted forecast. If a forecast calls for 2″ and it snow 3″, that’s a good forecast. We can see that NWS, Opensnow, and Snowforecast.com are pretty equal when it comes to busted forecasts. The t-test compares a distribution to the NWS as a null hypothesis. The only statistically significant difference at the 95% lever is Snow-forecast.com and that’s not a good thing since they’re obviously worse.

Posted in snowstats Tagged with:

Review of last week’s storm

[Correction: I was accidentally pulling data from snow-forecast.com that was in centimeters and not inches. Needless to say, this means there forecasts were actually the value below divided by 2.54. Or way wrong. They predicted zero out of 4 days correctly. Thanks to Joel Gratz for finding this error!]

There’s no snow in the forecast, so let’s look at the previous storm to compare and contrast forecasts.

Let’s start with Copper, because they received the highest 24 hour snow total. On the 25th, 16″ of snow fell on Copper. That’s a lot. Here’s a table that snows 24 hour snow forecasts along with the actual amount that fell. The Opensnow column is a range forecast in the format {low,high}.

DateNWSOpensnowSnowforecastForecast.ioSnow-forecastAvgActual
2014-11-226.49{4,8}81.8633.1*67
2014-11-237.09{3,6}145.2691.6*7.55
2014-11-245.28{4,8}106.5032.4*75
2014-11-255.9{2,6}65.585.1*616
Totals24.76{13,28}3819.21512.2*26.533

[* Snow-forecast values have been modified since originally published to their correct values. They were originally reported in centimeters. They are now in inches.]

Opensnow average forecast was 20.5″. As you can see, both the NWS and Opensnow under-predicted this storm by a large margin. But other sources did quite well. Snowforecast.com over-predicted snow amounts. Snow-forecast.com was the closest to actual and only missed the forecast by 2″ over 4 days. That’s pretty good. If we look at individual days, we can see that the most they were off was on the last day when Copper got 16″ of snow. They predicted 13″, which was the only forecast that day over 6″. So Snow-forecast was by far the winner with this storm. They accurately predicted 4 out of 4 days. [See correction at top of page!]

The NWS comes in with the second best overall forecast. They got 3 out of 4 forecasts: the first three days. They busted big on the last day, just like everyone else except Snow-forecast. My arbitrary cutoff for a “powder day” is 6″. They were the closest to predicting a powder day.

Coming in third is Opensnow who got 3 out of 4 forecasts as well. But on the critical fourth day, they predicted only 2-6″ of snow. So they busted this forecast.

Snowforecast.com actually didn’t do that well even though they predicted pretty close to the storm total. On two days, they predicted powder days when the actual snowfall was only 5″. And then on the actual powder day, they predicted only 6″. So they busted on 3 out of the 4 forecasts, but ended up getting lucky for the storm totals. Because they busted on three out of four days, Snowforecast.com comes in fourth for this storm.

I’m not sure why I track Forecast.io as their forecasts are always the worst.

I’m hoping that they help in creating a “best guess average”. That’s the “avg” column immediately to the left of the actual amount. Basically that value is a weighted average of all the other forecasts after accounting for any bias in the error distribution. For total amount, the “avg” came in second. It’s individual forecasts were a little off: 2-2.5″ on the second and third day. And on the fourth day, it barely predicted a powder day. I’m a little disappointed that it didn’t do better, but it’s just a simple algorithm, so I guess I can’t expect too much.

Posted in snowstats Tagged with: ,

Crested Butte Stats

I tweaked the way I calculate whether a particular forecast is a bust. I decided I was being a little too hard on the forecasters. For example, a forecast of 4″ with a reported snowfall of 6″ would have counted as a bust. Being off by 50% is still a lot, but for this particular example, it’s not a big deal.

crested_butte_error_distribution Crested Butte opens in 2 days! The table below represents the latest bust percentages at that resort.

Busted Forecasts at Crested Butte

NWS: 38%

Opensnow: 31%

Snowforecast: 30%

Forecast.io: 50%

Snow-forecast: 46%

Error distributions are shown to the right. Forecast.io and Snow-forecast.com are clearly the worst snow forecasters. They both have a high bust percentage and huge outliers. Snow-forecast.com is weird. They tend to either vastly over-predict or vastly-underpredict.

NWS tends to under-predict, but not by that much. Opensnow tends to over-predict, but not by that much. Snowforecast.com has busted forecasts at the same rate as NWS and Opensnow, but there a few under-prediction outliers.

Based on the data collected so far, Opensnow is the best snow forecaster at Crested Butte, but the NWS isn’t far behind.

Posted in snowstats Tagged with:

Happy Eldora Day!

Error distribution for Opensnow forecasts at Eldora

Error distribution for Opensnow forecasts at Eldora

Eldora Mountain Resort opened today. Which means that I can now take the bus to go boarding. Woot! In honor of that, I’m dedicating this post entirely to Eldora.

First, let’s look at the error distribution from the forecasts at Opensnow. It’s plainly clear that the distribution is not normal. There is a huge peak for errors between 0″ and 2.5″ and then the distribution falls of quickly (quicker on the negative side). The mean of the distribution is 2.248″ and the standard deviation is 2.760″. (If you really care, the skewness is 2.159″ and the kurtosis is 3.562″.) There are 4 days where the prediction was off by more than 7.5″.

Two of the error values are related, in that there was a large over-prediction one day and then a large under-prediction the next. That’s an artifact of my tracking since I only keep track of 24 hour snow forecasts. I’d like to keep track of 24, 48, and 72 hour forecasts, but that would involve extensive re-writing of code. However, if I did this it would be easier to verify storm totals. Right now, I really can’t do that since forecasts get updated every day. I’ll possibly add this when I have the time.

Error distribution for NWS forecasts at Eldora

Error distribution for NWS forecasts at Eldora

This is the NWS error distribution. There are again some obvious things that stick out. First, this distribution is heavily skewed to negative values, meaning that most of the time NWS is under-predicting at Eldora. However, if we look at the mean value it is 1.736″. So that one extreme over-prediction skews the mean to positive values. The standard deviation is 2.400″, which is less than Opensnow. I use standard deviation values as a proxy for how good overall the source is at predicting at a certain resort. The lower the value, the higher the skill. It’s not perfect, but it gives you an idea of how well forecasters are doing.

Eldora isn’t going to get snow today, but it’s a beautiful bluebird day out there. So I’m sure everyone that could make it is having a blast.

In site news, I’m still working on integrating the graphs into the resort pages. That won’t be finished until after Thanksgiving I don’t think. I simply don’t have the time to devote to it right now.

Until tomorrow!

Posted in snowstats Tagged with:

Turkey Pow

I’m stealing this headline from Opensnow because I like it.

I’m still in the process of adding a method to a WordPress plugin to display error histograms on the resort pages. It’s actually relatively easy to do in theory, but I can’t decide how to display the charts. I could display all the data at once. But I think that would be too crowded. I haven’t tested this out yet because I’m 99% sure it will be. Other solutions would be to initially display one histogram and have links to the others. Haven’t decided on this yet.

In other news, I decided to produce a best-estimate of the next 24 hour snow forecast based on the 5 sources I track. Tomorrow is kind of boring with only two resorts forecast to get over 1″. I haven’t started tracking these forecasts yet to see how they compare, but mathematically, they shouldn’t be worse. Which isn’t to say they’re going to be better. Most likely they are going to be statistically the same. Which is a bummer, but we all use the same models to issue forecasts.

Weather-wise, we’re in a bit of a slump. There’s a cold front coming through in a few days that will likely bring some snow to the state. Commenters on Opensnow seem to be stoked. It’s still too early in the season.

I’m still writing a “should-I-go-skiing-today” script. It’s obviously based on snow forecasts for the 24 hours as well as snowfall over the previous week. It’ll give more weight to recent snow. And obviously if it just snowed 10″ overnight, it’ll tell you to call in sick.

Posted in snowstats Tagged with:

Lies, Damned Lies, and… Snowstats?

I was involved in a discussion on Opensnow yesterday. It started with a comment from “gman” that said: “Colorado River Basin SWE at 95% of average as of right now (averaged from all SNOTEL sites). What a great start to this season!” That’s the whole post. Grammatically, there are two sentences. Since there are only two sentences in the post, the obvious implication is that the first sentence implies the second. I pointed out that 95% of average is below average. This is only strictly true if all values are non-negative though, which is the case for snowfall data. The lovely commenters at Opensnow didn’t like that much.

One reply stated that “averages don’t mean much, especially here in CO”. They then went on to compare this year to the last few. This is a great example of recentism. Recentism is the emphasis on recent events without a long-term, historical view, thereby inflating the importance of a topic that has received recent public attention. When we compare to only the last few years, we easily forget the longer-term past. That’s why long-term averages exist. SNOTEL uses the median from the period 1981-2010 as their baseline average. That’s a nice long period and it’s value clearly has meaning or else it wouldn’t be published.

Another commenter compared the 95% value to the value a few weeks ago, which was around 0%. This is recentism in the extreme! Yes, we had a nice snowstorm. Yes, there is a lot more snow on the ground now than a few weeks ago. That may mean that there is good skiing right now, but it doesn’t say anything about the total amount of snow that has fallen.

The original posted reminded me “don’t forget the + or – 5% accuracy typically associated with these types of measurements”. I’m not sure what that means. All values have an error associated with them. Just because a value is reported as 95% +/- 5% doesn’t mean that it’s 100%. It means it’s probably somewhere between 90% and 100% with 95% being the best guess.

My favorite responses were “Whats up debbie downer? Sheesh” and “You and Nathan need to hold hands, take a long wallk and talk about averages till the sun sets.”

copper_error_distributionThis is the error distribution of Opensnow forecasts. You can see that the errors are roughly normally distributed. There’s a peak near zero error and a longer tail on the negative side (under-prediction). But you can’t say that the distribution means nothing! There are many things that pop out immediately from this histogram.

  1. Opensnow over-predicts more than it under-predicts
  2. Under-predictions have larger errors than over-predictions
  3. Predictions are generally good; there aren’t many errors over 4″
  4. Most errors are between -1.5″ and  1.5″

What this histogram doesn’t show is how the forecast error varies with the actual snowfall amount. For tomorrow, I’ll try to create a scatterplot that shows this. What I expect to find is that as the actual snowfall increases there will be a corresponding increase in the forecast error.

 

Posted in snowstats Tagged with: ,

Copper storm totals 11/10-11/15

This last week has seen a lot of snow fall over the state. Almost all forecasts from all the sources I look at under-predicted. Not only did they under-predict storm totals, they under-predicted almost every single day! I’m going to look at Copper Mountain again today.

Every morning I check forecasts for the next 24 hour snow predictions. I do this because 24 hour predictions should be the most accurate. I would like to update in the evening, so I can know whether to head to the mountain the night before, but forecasts are updated in the morning. C’est la vie.

So let’s look at how the NWS and Opensnow did…

DateActualNWSOpensnow
2014-11-1051.33
2014-11-1120.510.5
2014-11-1281.892.5
2014-11-13117.286
2014-11-1462.324.5
2014-11-15105.988.5
Totals4219.2925

As you can see, there weren’t a lot of good snow forecasts there. Basically, what I think we’d all like to see is not so much perfect forecasts, but accurately predicting powder days. That is, if a powder day is predicted then we’re happy if it’s a powder day even if the total amount predicted is off by a few inches. Conversely, if a non-powder day is predicted then it better not be a powder day, because we’ll miss it!

For me personally, missing a good powder day is worse than having a predicted powder day but not receiving a lot of snow. Why is this? Because if a powder day is predicted, then I’m probably going to drive to the mountain. If it doesn’t snow, then at least I’m still boarding. If a powder day isn’t predicted, then I may or may not drive to the mountain, but probably not. And therefore, I’d miss a good snow day.

Definitions of what a good powder day is vary between people, but for me 42″ over 6 days qualifies. I actually didn’t get out because I had other work. I didn’t mind because it’s so early in the season. If it was January and the base was good, I would have likely skied all five days had I been up there. But would I have made the trip based on the forecasts? Likely not for the 10th through the 12th. Even with 7″ in the first two days, the next days forecast was only 1.89″ or 2.5″. That’s hardly enough for a good day. And I would have not been happy when 8″ fell!

The forecasts for the next day (7.28″ and 6″) would have seen me drive to the mountain. I would have been surprised to see so much new snow, but I would have already been there. So even though the forecasts were off by 4-5″, I still consider that a much better forecast than the previous day when the errors were similar, but they missed predicting a powder day.

One thing that sticks out is that every forecast was an under-prediction. What caused this bias? I mean, NWS under-predicted by over 50%! That’s absolutely terrible. Opensnow did a little better, but still every day was under-predicted.

It’s early winter, so maybe everyone is a little rusty from the summer. That’s my hypothesis anyway.

Posted in snowstats Tagged with:

Yesterday at Copper

Copper got hammered yesterday! That was expected, but they got a lot more snow than predicted. The National Weather Service predicted that Copper would receive 7.28″ of snow and Opensnow predicted 4-8″ (with an average of 6″, obviously). Other resorts got hammered as well: Breck got 10″, Keystone 9″, and Crested Butte 11″. But those resorts aren’t open (yet!), so it doesn’t really matter.

But Copper is open. They don’t have any runs open yet that are worth driving out there for, but it’s only a matter of time. So let’s look at what went wrong. Joel writes:

The over production of snow along I-70 is more mysterious. I think it has to do with what I talked about this morning, the cold air from yesterday trapped near the surface and forcing the moisture streaming in from the west to rise and create snow. Sometimes all the factors just come together perfectly.

Since I’ve never been an operational meteorologist, I’m not going to second guess anyone. However, it’s clear that something was wrong with the models. He goes on to say:

I really really really dislike being wrong about a forecast. That said, I’d much rather have realistic expectations, have most mountains meet those expectation, and have a few areas come in much higher that forecast. That’s much better than shooting too high and being disappointed.

Here’s where I can offer input. Here Joel is essentially saying that he tries to under-predict snowfall. That’s what it means if you are right on 90% of the time, but that other 10% you under-forecast. That’s a bias in the system. As scientists, we should be trying to eliminate all biases. So what we should be doing is trying to forecast the exact amount every single time. Then the over-forecasts will be canceled out by the under-forecasts.

Let’s look at how he actual does: Copper Mountain Statistics.

As of today, the statistics table is as follows:

Source
Days Under
Avg Under
Perfect
Days Over
Avg Over
NWS282.71″27261.99″
Open Snow163.31″25371.84″
Snowforecast162.38″2262.00″
Forecast.io113.32″572.34″

The table says that Opensnow predicted 78 non-null days (starting on January 1st last season, plus a few this season). A null forecast is where the forecast was for 0″ of snow and there was exactly 0″ of snow. Non-snow days are fairly easy to predict, so we toss those out so we don’t skew the results. Of those 78 forecasts that either predicted snow or there was actual snowfall, Opensnow only under-predicted 16 times, or 21% of the time. Conversely, they over-predicted 47% of the time!

The National Weather Service is equally split amonst over- and under-predictions.

Further, when Opensnow under-predicts, the amount that they are off by is a lot larger than when they over-predict. So when they under-predict, they are under-predict by a huge amount! If you look at the NWS numbers, their average under-prediction is larger than their average over-prediction, but they are much closer. The Opensnow numbers are skewed.

I have code that makes histograms of the forecast error. I might break that out tomorrow to see if we can put some better numbers out there.

Posted in copper, snowstats Tagged with: ,

Busted!

super bee cam

Copper Mountain Super Bee cam on November 3, 2014.

Yesterday saw a lot of snow for many of the resorts. Unfortunately, it wasn’t well forecast. Another bummer is that snocountry only reports data for open resorts. That mean we only have measured totals from Arapahoe basin, Copper Mountain, Keystone, and Loveland.

Let’s start with the good news. Both the NWS and Opensnow predicted there would be 5″ of snow at Abasin. 6″ were recorded. That’s about as good as can be expected given certain limitations. For instance, the NWS forecast is a point forecast for a certain grid box in their model. The model doesn’t know were the recording station is, and it probably doesn’t care.

Opensnow predicts using a range for the amount of snowfall. Since the NWS uses a single value (and other sources too) , I need to convert their range into a most likely value. I use the mean. Mathematically, this doesn’t make sense. There are a lot more low snowfall days than high snowfall days, so assuming a normal distribution (which is implicitly what I do when I take the mean) is wrong. However, I think it’s good enough for our purposes here. So fore a forecast of 4-6″ of snow, I use a value of 5″ as a best guess.

So both the NWS and Opensnow accurately predicted the amount of snow at Abasin. Let’s move a little ways away to Loveland. NWS predicted 6.1″ while Opensnow predicted 5″. The amount recorded at snocountry was 8″. Obviously NWS did a little better than Opensnow. by 1.1″.

When we head over to Keystone, things get much worse. Both NWS and Opensnow busted the forecast. NWS predicted 1″ and Opensnow 5″. They recorded 9″!

Copper saw NWS predict 2″ and Opensnow 3″ with a snow total of 8″.

I’m in the process of developing a Bust Index. Basically, I want an index where we can compare forecasts from different forecasters at the same resort as well as compare forecasts at different resorts. At the moment it’s still very crude and I don’t want to use it publicly yet, but it does seem to bring something to the

table. Let’s look at some specific numbers for NWS forecasts at Loveland over the last 3 days.

DatePredictedActualBust IndexBust or Not
2014-11-103.8270.83BUSTED!
2014-11-110.9143.4BUSTED!
2014-11-126.180.31Not busted

Looking at this table, it’s clear that the higher the Bust Index, the larger the difference between the forecast and the actual snowfall. A snowfall of 3.82″ would be good skiing or boarding, but wouldn’t be a great powder day. But 7″ of snow would likely be a good powder day (depending on what time the snow actually fell). Missing a 4″ day is vastly different than missing an 8″ day. The Bust Index on the 11th is too high though. If the weather prediction is 1″ of snow and it snows 4″, so what? A 4″ day isn’t going to be spectacular unless there have been 4 or 5 of them in a row.

I’d like to hear your thoughts. What makes a busted snow forecast?

Posted in snowstats

Snowstats 2014-2015

I’ve started the cron jobs for snowfall tracking! #snowstats

What does this mean? It means I’m reading for winter.

Posted in snowstats