Frickin' Freakonomics

That whole How Valid Are T.V. Weather Forecasts? story just sets my teeth on edge. It is another example of "gotcha" news stories about how scientists are idiots that can't get anything right.

For starters, the statistical methodology was not explained. How does J.D. Eggleston define "Degrees Missed"? You need to put a sign on the temperature deviance in order to rule out systematic bias. If there a consistent temperature bias, could he and his daughter have placed their backyard thermometer in a place that gets radiated heat from a rock or their house foundation? Was their house at a higher or lower elevation than the official elevation for their city? This matters because meteorologists use the predicted height at 850 mb and extrapolate downward to the official elevation.

If you want to know the value added by a forecast, you take the difference between the forecast and no forecast.

What does that mean? You can set the baseline as either persistence (what was the weather like yesterday?) or climatology (what was the average on this date over a long period of at least 50 years?). Then calculate the differences between the actual observed weather with the forecast and also against no forecast. The change in forecast skill (hopefully, a reduction in error) between the forecast and no forecast is the true value-added of the forecast.

Studies that handle statistics more carefully show that today's 15 day forecast is about as accurate as the 3 day forecasts made 30 years ago. This is primarily due to improvements in numerical weather prediction and the use of satellite data.

What methodology did they use for verifying rain forecasts? Did it rain in their backyard? That's not meaningful. The precipitation forecasts give a probability over a broad area. Their home could be in a micro-climate. In addition, how does their methodology handle when the forecasts were off in the timing of the precipitation, but not in the areal extent?

Furthermore, TV weather readers are as likely to be meteorologists as the news readers are likely to be journalists (whatever that means now). Most majored in communications, not meteorology. No professional qualifications are needed to read the news on TV other than good looks.

Some weather readers sought and received the American Meteorology Society (AMS) Seal of Approval or "Certified Broadcast Meteorologist" (CBM) designation. The AMS Seal of Approval was so contentious among AMS regular members, that the program is being phased out. The CBM designation was an attempt to tighten up the standards.

The main difference between the two programs is education and the exam. To apply for the CBM, applicants must hold a bachelor’s or higher degree in atmospheric science or meteorology (or the equivalent) from an accredited college/university. Current AMS Sealholders (those that earned the Seal prior to January 1, 2005 ) are not required to meet this criteria. These Sealholders may qualify for the CBM designation if they pass the written exam. All CBM applicants must pass the written exam to earn the CBM designation.

Go read the requirements; you could get the seal of approval by sending in videos of yourself reading the weather and looking good while doing it. Even with those loose requirements, most weather readers don't even hold that qualification. Current seal holders can get CBM designation by passing a multiple choice exam with a score of 75 or more. They better hurry because this deal expires next year.

Digression about TV weather forecasts:
A while back, Bulletin of the AMS (BAMS) published study of TV forecasts. The study determined that TV forecasters overplayed the likelihood of extreme events, particularly precipitation, for ratings. They would play teasers all night long, "Is there rain in your commute tomorrow?" and make people wait until the 11 o'clock news to say that there was a 10% chance of precipitation.

FYI
I am a regular member of the AMS that qualified under this category:

hold a baccalaureate or higher degree from an accredited institution of higher learning in some other science or a related field and be currently engaged in a professional activity in which his or her knowledge is applied to the advancement or application of the atmospheric or related sciences

If I recall correctly, another regular member of the AMS had to vouch for my scientific competence and that my job was substantially meteorology before they admitted me. They'll let anyone in. ;-)

Afterthought

Why is Eggleston so deeply involved with his daughter's science project?  Like I blogged about earlier, Mark and I were probably the only parents in Iris' first grade class that let her turn in a diorama that SHE created.  Don't be a helicopter parent.  Let kids do their own homework.