Mar 24 2008
I Guess I Don’t Understand the Time of Observation Correction
Of all the adjustments that are made to the USHCN, the time of observation bias (or TOBS) should be the least controversial. If someone records the daily extreme temperatures at 4pm they will have slightly different numbers than if they recorded the temperature at 8am. But The nice thing about the TOBS adjustment is that it is supposedly just correcting for this bias. The USHCN contains the raw data and the tobs data, so we can compare them for a given surface station. The USHCN documentation describes the time of observations bias as follows:
Next, monthly temperature values were adjusted for the time-of-observation bias (Karl, et al. 1986; Vose et al., 2003). The Time of Observation Bias (TOB) arises when the 24-hour daily summary period at a station begins and ends at an hour other than local midnight. When the summary period ends at an hour other than midnight, monthly mean temperatures exhibit a systematic bias relative to the local midnight standard (Baker, 1975). In the U.S. Cooperative Observer Network, the ending hour of the 24-hour climatological day typically varies from station to station and can change at a given station during its period of record. The TOB-adjustment software uses an empirical model to estimate and adjust the monthly temperature values so that they more closely resemble values based on the local midnight summary period. The metadata archive is used to determine the time of observation for any given period in a station’s observational history. This adjustment is the first of several adjustments to the USHCN monthly temperature values.
That’s fine. I understand that reading the temperatures at different times can cause a slight bias in the data. I was fiddling around with the TOBS data and comparing it to the RAW data and noticed something odd. For the few cases that I’ve looked at, it appears that the TOBS adjustment is not and adjustment for the time of observation bias. Let’s have a looksee at one particular surface station. In this case, I chose the station at State University, MS.
|
Here we have the raw data for this station in black, and the tobs corrected data in red. In recent years there has been no correction to the raw data. The further back in time one goes, the more the tobs is corrected. I then plotted the difference between the tobs data and the raw data. The result was interesting.
|
We can see that there were two big jumps in 1960 and 1983. Therefore, when we go to the station metadata, we should expect to see the time of observation to change at those two times. And in fact, there are record changes in the metadata at exactly those two dates. The bad news is that I don’t see that the time of observation changed. I’ve included an excerpt of the MMS metadata below.
|
I’ve circled the time of observation for the temperature. In each case, the temperature is recorded at 0800, or 8am. There are records of change that occured at the same time as the TOBS correction, but they could not be from a change in the time of observation.
What is the TOBS correction actually correcting?
(No Ratings Yet)
Loading ...Related Posts:
30 Responses to “I Guess I Don’t Understand the Time of Observation Correction”
To reduce spam, comments are automatically closed 30 days after the last comment. If you would like to comment on any closed thread, please use the contact form at the top of this page.
DUDE. i LIKE YOUR STYLE. lEEP PUSHING FOR INSIGHT… WANT TO HAVE BABIES WIUTH YOU.
Would it be so hard to take questions like this and fire them off to some appropriate person at NOAA? Possibly they would know.
Nice work. It would be interesting to get a proper explanation from the source.
For the times, I’m not sure if a standard UTC deviation is used, otherwise the dates of change doesn’t seem to correlate with major changes in Daylight Savings Time and I also don’t think MS went through a time zone move.
nice work atmoz, tobs is oone of the major adjustemnts to the raw files. perhaps check the history file versus the online version, ?? I hate to think about auditing all these things by hand
There is an assumption that those TOBS meta-data circled in red above are correct. No way to know for sure, of course. But it’s easy to envision that observers could very easily pre-fill, or simply repeat, SOP-type information like those.
Maybe the observers weren’t aware that modifications to the data would be based on the values reported for TOBS. The system wasn’t originally put into place with Global Climate Change as an application objective.
Independent Verification of such meta-data will be next to impossible.
ps
Another example of being cooler in the past than we realized.
I have been wondering about TOBS ever since that unhappy day preusing the NOAA adjustment page. It’s their biggest adjustment.
Yes, a totally valid concept. But where was the due diligence?
So now, you’re telling us that TOBS . . . isn’t TOBS? Say, WHAT?
I will say this much for the NOAA: at least they let us in on the bottom line.
Although I was stunned to discover that their net UHI adjustment is a measly -0.1F! (I kid you not. ) And–according to SHAP–them surface stations just seem to be getting artificially cooler and cooler, so they MUST be adjusted warmer . . . but that is an outrage for another day.
“Yes, a totally valid concept.”
But applying modifications to measured data by use of an empirical correlation, imo, is less than a valid concept.
As suggested by Steve Bloom, I have emailed NOAA about this.
Nathan,
Here are what the origonal handwritten observers reports say about time of observation for some of the months before and after the changes:
June 1989 7:30 AM CST
Feb. 1989 7:30 AM CST
June 1983 7:30 AM CST
Feb. 1983 7:30 AM CST
Jan. 1980 8AM
Jan. 1975 6AM
Jan. 1970 CST (no time given)
Jan. 1962 5-P-C
Oct. 1960 5-P-C
I’m not sure about the quality control of any of the metadata. The station history file shows a change in temperature observation in Jun 1962 from 1700 to 0800. I don’t know if that’s local time or not. It appears that there was a move in 1983 but no change in time.
[Reply: I look at the station.history file in my post linked below.]
[...] my last post, I stated that I don’t understand the time of observation correction. This was because when I was looking at the metadata, I could find no evidence that the time of [...]
Atmoz, hopefully NOAA does not respond with something along the lines of “we have already published the necessary information to figure this out, go find it yourself”. Seems I have seen that sort of response before on another blog.
[Reply: As long as they point me to the relevant paper(s). But since I've sent the email, I've discovered that the MMS metadatabase does not contain the correct time of observation. So the main questions in my email are now almost moot.]
I apologize for the dumb question, but I have not followed the TOBS adjustment discussions closely. Does this adjustment tend to cool older records rather than warm them, as is seen in the graph above, or is the sign/magnitude of the adjustment on older records random?
atmoz, karls paper on tobs is online, if you cant find it, i’ll mail it to you
[Reply: I've skimmed it. But in my opinion, it's less than clear.]
Steven Mosher, can you post a link?
[Reply: It's behind a paywall, but A Model to Estimate the Time of Observation Bias Associated with Monthly Mean Maximum, Minimum and Mean Temperatures for the United States.]
John,
The sign of the change depends on whether the observation time was moved from morning to evening or from evening to morning. If the time of observation changes were random throughout the day, we would expect that there would be no net change caused by the TOBS adjustment. In reality, most station have been moving from an evening observation to a morning observation, which causes a slight negative bias, and thus a positive adjustment to subsequent years (usually seen as a negative adjustment to previous years). The effect of the changes can be seen in this graph.
Atmoz, You realize that Tobs is a giant can of worms. There is a guy over on CA named JerryB who seems to know a lot about it
and has done some work himself. He also seems to have some inkling of the inner workings at NOAA WRT to files and such. There was a TOBS thread there a while back, but it degenerated. I’ll post there and see if JerryB resurfaces and send him your way.
All that said, I think Karls paper could use an update as you note it’s somewhat Opaque.
Here Atmoz
http://www.john-daly.com/tob/TOBSUM.HTM
I’ve left a message for Jerry Brennan at CA. Maybe he can help, maybe not. Worth a try. Like I said he seems to know a lot about the ins and outs of the files and TOBS
Here’s something for JerryB or ATMOZ. Audit TOBS adjustments applied anywhere in the the range of years 1899 - 1930.
As Steve McIntyre has noted many a time, it is odd that all of these adjustments seem to have a positive slope over time. We have TOBS, UHI, station record combination, annual estimation when monthly records are missing - the list goes on.
[Reply: Not all of them do. But it makes sense that many of the TOBS would since more people are now taking the measurements in the morning as opposed to the evening.]
The TOBS adjustment should adjust the TIME axis, not the TEMPERATURE axis. Most of the early period, the thermometers recorded the correct Tmax and Tmin but occasionally they were assigned to the wrong day. I can see no valid reason why such enormous adjustment is made to Temp. Indeed, if one ignored the wrong date entirely and just averaged the available Tmax and Tmin, the result should be ok for long term trends.
There is a complication. What is day and what is night at the Poles? The “days” last 6 months and ther “nights” last 6 months, one way of looking at it. At the Equator, it’s clear cut. So there is a grey zone at some intermediate latitude where TOBs starts to become irrelevant. If I recall correctly, it was at such suspect latitudes that Karl did his modelling that has formed the basis of the adjustment ever sine. I think it’s nonsense and there are abundant Temp records to support this.
Atmoz,
It appears that you may be taking the SHAP/FILNET adjusted temperatures as if they were the TOB adjusted temperatures. The 3+ lines have the TOB adjusted temperatues, the 3A lines have the SHAP/FILNET adjusted temperatures.
[Reply: Good catch. I did make a mistake for this post. It was fixed before the next one though.]
Well, OK all is a bit strong. Most is fair.
john Goetz, if you need the Karl Tobs paper ask steveMc for my email and i’ll mail it to you
The correct TOBS-RAW:
RE: Geoff Sherringtonon 27 Mar 2008 at 2:24 am
Somewhere around the latitude of Dingwall Scotland. In high summer, it never really gets dark there. Anything north of that, and you would certainly be in the grey zone.
Yes, but do you get the point that the max-min thermometers in common use gave a correct maximum or a correct minimum and usually both, on any given day.
So the monthly average of Tmax, minus Tmin, divided by 2, should give a very close approximation to instrumental best ability for a montly average. The problem is made large by taking a daily average and trying to correct for right day-wrong day from an idealised set of lookup tables about how temperatures should behave.
I can see no way that the TOBS adjustment should induce so often a cooling of the early years. It is mathematical hocus pocus. And yes, I have read Karl about 3 times and still think it’s pea and thimble. Maybe I lived in the tropics for too long, where the song goes
“The tropical sum beat down on his bum
And melted his balls into butter”. (WWII vintage, that.)
Someone, please awaken me if I am missing a critical point.
Geoff.
[Reply: By taking the temperatures in the morning, as is currently done at most locations, you'll be over-counting the extreme low temperatures. They'll be counted for two days instead of one.]
Disagree with reply. Simplified assumptions here - the Tmax are all ok, being well away from thermometer reset times. Read then reset happens in the morning as you propose.
Case 1. If you reset in the morning after the preceding 24 hour min (and max) has been reached, you will record the correct min (and max), irrespective of whether the min is colder or warmer than the previous or next day. All answers are correct. No adjustment is needed.
Case 2. If you reset in the morning before the preceding 24 hour min has been reached, it will show on your next days’ record as being either correct (if the 2 days are identical weather wise, or too cold (if the next day had a warmer min) or not cold enough (if the next day had a colder min). On average, you are just getting a disconnect between day and temp. That’s why I wonder about the South Pole, where day has little meaning in this regard.
Please correct any misconceptions I might have expressed.
Either way, your faithful thermometer has recorded the right max and min every day for a month.
Geoff,
Whatever the Tmin, and at whatever time it occurs, if day 2 morning temperatures
are warmer than day 1 morning temperatures, the day 2 Tmin will be taken
from day 1. While it may be higher than the day 1 Tmin, it will be
lower than the day 2 morning low, thus the bias.
This will occur often at mid latitude locations.
Jerry,
Are you not assuming that temperatures vary according to some idealised sine-type curve? The thermometer is designed to read and record the highest and the lowest temperatures in a 24 hour period. If the coldest part of the day is at the time the thermometer is read and reset, then that is the true min. If it gets colder later on day 2, the min termp device will record this and correct in your mind for what you might have thought till then, was the morning min at reading.
In other words, if it is very cold at reading time, the min temp for day 1 might be only a second (so to speak) from the min temp for day 2. This is not an error, this is a measurement. The reset will not change anything because the device will revert to the same Tmin as before unless you take a few hours to reset instead of a few seconds.
If Tmin does not occur regularly at 4 am on a cyclic curve, that does not invalidate a Tmin at a 9 am reading/reset time, if it is indeed colder.
I still do not know how to correct for TOBS at the South Pole (where noon is set by convention, not by climate) or at points progressively away from the Poles, where day and night are often indistinguishable except with a clock.
TOBS has big problems. I keep saying this and I keep getting Karl quoted back. I can’t follow Karl’s logic. He seems to confuse reading with resetting.