Fauxtomation

This new word hit the zeitgeist on June 19, 2017 thanks to @astradisastra.

The next day, Mel Healy wrote a smart essay about Fauxtomation and the Mechanical Turk.

For all their tech, many leading tech firms nowadays may be rather more “hi-Turk”, relying on cheap labour to do the day-to-day maintenance and moderation of their social media. Like the Mechanical Turk’s operator these people are largely hidden away inside big boxes, only this time the boxes are on the opposite side of the planet, in India or the Philippines. Vast armies of invisible workers in underdeveloped countries.

Each box is decidedly unglamorous compared with the shiny new HQs and campuses of Silicon Valley in California or Google Docks in Dublin. You won’t find any fancy games rooms and lavish staff restaurants, or “micro kitchens”, chillout zones, fitness centres, swimming pools, wellness areas, tech stops or phone booths.

June 23, 2017, Shira Ovide wrote about the army of workers needed to bring you Amazon's one-click convenience.

Fauxtomation is the word that crystalizes why I feel so angry about gushing articles like this about places like Eatsa, a restaurant that supposedly serves vegetarian food made by robots.

Customers tap their meal selections on an iPad or their smartphone and pay electronically. No cash is taken here. Then when the order is ready, hands slide the meal into a “cubby,” which lights up with the customer’s name. The plan is for it to be ready in less than minutes from the time the order is placed.

Silicon Valley reinvented the automat. But--most insidiously--this time, they are selling a guilt-free low-cost experience by pretending that a low-paid human did not make the food.

Eatsa is the brainchild of Scott Drummond, a techie focused on data-driven results. He says forgoing meat, along with staff, helps keep the cost of goods down.

Drummond is all about the data science and other buzz words/phrases such as “enhanced predictive and personal health engagement.” But can robots prepare these meals? If so, what a breakthrough in robotics!

How the kitchen will hold up remains to be seen. For now, at least, it relies on human components: about five employees involved in prepping, assembling, and expediting behind the store’s façade.

The dirty secret finally comes out, there are people hiding in the mechanical Turks. Even then, he obfuscates further by invoking the glamour of warfare and robotics.

But for now, Eatsa still needs a few good chefs, with some special skills. “They can’t be afraid of technology,” say Drummond. “Our first general manager used to be a military robotics specialist.”

You betcha that a robotics expert is not the guy making your $7 lunch in San Francisco.

Wonkblog explains the crisis in restaurant staffing, particularly in high cost areas such as SF-SV.

And don't even get me started on Blue Apron's unsafe working conditions necessary to bring us cheap, home-cooked meals.  Food, like clothing, is not going to be cheap and fast unless we sacrifice some people.  Are we willing to confront those choices head-on?

Addendum:
The Washington Post reports that shipping costs account take up 30% of the price of Blue Apron meals.  It's a big driver in why BA squeezes their kitchen staff to work at unsafe speeds or to work off the clock without pay.

Pushing back against Weapons of Math Destruction

I'm such a huge Cathy O'Neil fan, that I put Doing Data Science: Straight Talk from the Frontline on my very short list of recommended books for scientists and data scientists. I wrote:

The most hands-on of the meta books or the most meta of the hands-on books? Not many introductory books include a chapter on ethics but more should.

Weapons of Math Destruction is the book about data science ethics that I've been waiting for.

Listen to the interview with author Cathy O'Neil on All Things Considered.

I especially like this exchange:

MCEVERS: So it sounds like when you're saying, you know, we have these algorithms, but we don't know exactly what they are under the hood, there's this sense that they're inherently unbiased. But what you're saying is that there's all kinds of room for biases.

O'NEIL: Yeah, for example, like, if you imagine, you know, an engineering firm that decided to build a new hiring process for engineers and they say, OK, it's based on historical data that we have on what engineers we've hired in the past and how they've done and whether they've been successful, then you might imagine that the algorithm would exclude women, for example. And the algorithm might do the right thing by excluding women if it's only told just to do what we have done historically. The problem is that when people trust things blindly and when they just apply them blindly, they don't think about cause and effect.

They don't say, oh, I wonder why this algorithm is excluding women, which would go back to the question of, I wonder why women haven't been successful at our firm before? So in some sense, it's really not the algorithm's fault at all. It's, in a large way, the way we apply algorithms and the way we trust them that is the problem.

I hope you read or listen to the interview. Perhaps we can do a virtual book club and read it together?

In case you were not a reader of this blog in 2008, I wrote about my experiences using a proto credit-scoring algorithm while in high school student working part-time for Citicorp in the mid-1980s.  I did (with my boss' support) what I could to push back against arbitrary scoring algorithms when I felt they did not accurately capture an applicant's credit-worthiness.  It's also a time capsule for a time when we could assume that health insurance companies would eventually pay so that healthcare liabilities did not count against employed people.

What I didn't write in 2008 and should have, was that I applied to both Kelly and Kelly Technical Services. Kelly sent me to do the lower-skilled clerical work for slightly above minimum wage. Kelly Technical Services sent a male former classmate (who needed my help to debug one of his homework assignments) to work on implementing the software algorithm that eventually replaced the clerks like me. He got paid more. A lot more.

Bias was not created by algorithms.  We built the algorithms in our own image.

Feeling Patriotic

Another in the long list of reasons why I love this country...

Anyone can ftp into this NOAA server to download weather and climate data for free! Check out how easy it is to anonymously ftp into the data server.  Use "anonymous" as your username and an email address as your password*.

Grace> ftp nomads.ncdc.noaa.gov
Trying 2610:20:8040:2::166...
Connected to nomads.ncdc.noaa.gov.
220 2610:20:8040:2::166 FTP server ready
Name (nomads.ncdc.noaa.gov:Grace): anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
230-************************************************************
** WARNING ** YOU HAVE ACCESSED A US GOVERNMENT COMPUTER.**
** WARNING ** UNAUTHORIZED USE IS PUNISHABLE BY FINES OR **
** WARNING ** IMPRISONMENT UNDER PUBLIC LAW 99-474. **
** WARNING ** INDIVIDUALS USING THIS COMPUTER SYSTEM ARE **
** WARNING ** SUBJECT TO HAVING THEIR ACTIVITIES ON THIS **
** WARNING ** SYSTEM MONITORED AND RECORDED BY SYSTEM **
** WARNING ** PERSONNEL IN ACCORDANCE WITH ESTABLISHED **
** WARNING ** SECURITY PRACTICES. **
************************************************************
230 Anonymous access granted, restrictions apply
Remote system type is UNIX.
Using binary mode to transfer files.

And look at the available data:
ftp> ls
229 Entering Extended Passive Mode (|||62934|)
150 Opening ASCII mode data connection for file list
drwxrwxr-x 20 3682 nomads-prod 4096 Feb 16 2016 12
drwxrwxr-x 14 3682 3682 4096 Aug 16 11:43 32
drwxrwxr-x 23 3682 nomads-prod 4096 Sep 12 2012 33
drwxr-xr-x 3 root root 4096 Mar 30 2015 CFSR
drwxr-xr-x 2 root root 4096 Mar 30 2015 CFSRR
lrwxrwxrwx 1 root root 7 Aug 26 2013 GDAS -> 12/GDAS
lrwxrwxrwx 1 root root 7 Jun 6 2013 GENS -> 32/gens
drwxr-xr-x 2 root root 4096 Jun 6 2013 GFS
drwxrwxr-x 11 3682 3682 32768 May 8 2015 model
drwxr-xr-x 2 root root 4096 Jun 6 2013 NAM
lrwxrwxrwx 1 root root 7 Jun 6 2013 NARR -> 12/narr
lrwxrwxrwx 1 root root 10 Jun 6 2013 NARR_monthly -> 12/narrmon
lrwxrwxrwx 1 root root 7 Dec 18 2012 NDFD -> 33/ndfd
lrwxrwxrwx 1 root root 8 Aug 23 2013 NDGD -> 33/ndgd/
lrwxrwxrwx 1 root root 3 Sep 23 2013 RAP -> RUC
lrwxrwxrwx 1 root root 15 May 6 2015 reanalysis-2 -> 12/reanalysis-2
-rw-r--r-- 1 root root 26 Oct 13 2010 robots.txt
drwxr-xr-x 2 root root 4096 Dec 16 2015 RUC
drwxr-xr-x 2 root root 4096 Jun 6 2013 SST
-rw-r--r-- 1 root root 610 Mar 23 2011 welcome.msg
226 Transfer complete

Now what do you want to see?

ftp>

This is where all the weather apps and websites get their data. They just package it up and add advertising or charge you. You can do this yourself with a computer by getting the data from NOAA and the easy-to-use Panoply software from NASA for free.

* You don't have to use your email address, but it is standard courtesy in anonymous ftp to use your email address as a password so that they can log where users are coming from. For instance, if you are a teacher or student, use your school email address so they can log you as an education user. This is how we discover needs and allocate funding.

Data Thinking Before Data Crunching

Years ago, I was a data analyst and subject matter expert.  Now that Data Scientist is the job description du jour, I rebranded myself.

Rebranding really works.  My online resumes generated a lot more views, though I remained substantively the same person, with the same skills, education and experience.

Lemmings.

Anyway, my pet peeve is how fixated interviewers are about how much data you've crunched, instead of how well you did it.

I really, really, really like my job as a data specialist and educator.  In case you wonder why a forty-something married mother would move away from her family for a job that barely pays enough to run a second household and fly home, I thought I would show you a sample of my work.

I posted my slides and notes for a recent talk, Data Thinking Before Data Crunching, given to a mixed audience of students, working software developers and scientists-data providers.

I help people use data optimally with the (IMHO) most pressing big data use case facing the world--weather and climate.

It's a good thing that the computer-based work (and my management) allows me to work from my LA home part of the time.

Why RTW doesn't fit

A statistics blogger, John Cook, explains why ready-to-wear fits so few women.

In 1945, a Cleveland newspaper held a contest to find the woman whose measurements were closest to average. This average was based on a study of 15,000 women
[skip]
Out of 3,864 contestants, no one was average on all nine factors, and fewer than 40 were close to average on five factors. 

I want to point out that women were more homogeneous in 1945 Cleveland than they are today.  I would guess that the contestants were mainly young women of European descent.

Then Cook uses a normal distribution to simulate 3,864 women with 9 independent measurements to illustrate the point I made in Meeting Shams.

Body measurements are correlated; that's why RTW pants can be clustered as "curvy", "straight" or "favorite" fits.  But it's an interesting exercise and Cook provides his Python code so you can play around with your own simulations.

BTW, I'll be giving a talk, Data Thinking Before Data Crunching, at the CISL 2016 Software Engineering Assembly on April 5, 2016.  The following day, Mary Haley* and I will be co-teaching an all-day hands-on workshop for analyzing and visualizing spatial and atmospheric datasets with Python and NCL.

If you are in (or can get to) Boulder April 4th-8th, 2016, we'd love to host you at NCAR. This year's theme is Data Science.

See the program.
Apply for a student scholarship to attend.

* Mary is the lead software engineer for the visualization group and I am the education and outreach lead for the data support group.

Benchmarking government

I'm still processing the terrorist killings around the world right now.  I'll leave discussions about that to people who understand it better than I do.

Right now, I want to shed light on a little corner of the universe that I do know better than most.  Hopefully, the amount of understanding in the world will go up a little bit because of what I write.

I am a data specialist in a geophysical data archive so I follow news about geo-referenced data more than the average citizen.  Actually, I'm a bit obsessed with how data searches work or don't work and why.

This editorial appeared in my customized news feed and I was completely flummoxed by the ignorance displayed by the editorial board of a purportedly top-tier newspaper.

The LA Times Editorial Board was incensed by Governor Brown's request to the Division of Oil, Gas and Geothermal Resources--just days after the governor had appointed their new chief--to supply him with a report on the mineral history and the potential for mineral extraction of his family's ranch in rural California. The editorial said:

It's inappropriate for the governor to call the head of an agency for help with personal business, especially someone he had just installed in the job nine days before. It also was wrong for his aides to follow up with the agency to ensure that there would be a map and other specific information. State employees are paid to do state business, not take care of the governor's personal matters. Brown received his report within a couple of days after he asked for it — an uncommon alacrity in state government — and also received a satellite map drawn up especially for him.

When I read that, I was shocked, but not for the reason the editorial suggested.

I was impressed that Governor Brown, a 77 year-old philosophy major, understood the scientific method and how to apply it to data problems.

Whenever you tinker with a system, you run benchmark tests before and after.  If you install a new chief of a department, you measure his effectiveness by testing response time and job quality for a common task required by the department.  Moreover, you run this test for a case that you know well, so you can assess the accuracy of the results.

Asking for all the info on oil and gas extraction in the past, and potential for the future, for the family farm is a great idea.  His family has owned that land for more than 150 years.  If there had been oil and gas exploration on the land in the past, he would have known about it.

I asked my husband, a field scientist, what he thought of the story.  He said that you always test in an area you know really well, so you can gauge the quality of your measurements, before you go to an unknown area.  So that's two scientists who were impressed with the governor's grasp of the scientific method.

The governor got the correct answer in 24 hours, according to this later story with more details.

The wire service story said that "after a phone call from the governor and follow-up requests from his aides," the regulatory agency "produced a 51-page historical report and geological assessment, plus a personalized satellite-imaged geological and oil and gas-drilling map" of the area.

You know, just like any ordinary citizen would expect to receive.

But the characterization of the service appears to be a stretch. Except for a one-page personal memo, all the material collected for the governor amounted to merely a pile of old letters sent other property owners, historic data from yesteryear and some oil field maps.

"Everything is available on the [state] website," said Nancy Vogel, chief spokeswoman for the Natural Resources Agency, the umbrella entity for these regulators. "If you know how to find it.

"They did not do a formal assessment. That would have been many weeks of work."

The governor got back his answer within 24 hours. "The potential for significant oil or gas in this area is very low," the memo read. As for mining, that potential also "is exceptionally low."

Steve Bohlen, Brown's appointee as chief regulator, said the governor asked him about the geology of the land, past oil or gas production and potential for any future production. "I said that was easy to do," Bohlen told me. "It wasn't like 'drop everything.'"

Two petroleum experts who aren't necessarily Brown fans confirmed to me that all this stuff is available on the state's oil and gas website.

That "just like any ordinary citizen would expect to receive," is a low blow. Ordinary citizens in this data-driven era should be able to look up the mineral history of their land (or surrounding land) as that is the best predictor of future mineral development.

The Center for Public Integrity gave California a C- in their 2015 State Integrity report card.  The grade was largely brought down because of an F on Public Access to Information.

Making public information easily available to citizens should be a high priority and the governor should appoint public officials who are committed to improving data processes and data access for citizens.  Running a benchmark test at the start of a new department chief's tenure was the right thing to do.

Let's hope that this media 'gotcha' campaign doesn't deter him from running the 'after' benchmark test to see if they turn up more (or less) data faster (or slower) after Bohlen has been on the job for a while.

Background:

Geophysical data is extremely difficult to search for many reasons.  Records are messy, inconsistent, and often came from the pre-digital era.  So many things can get lost in the translation--or get plain lost.

We are so accustomed to nearly instantaneous searches on the internet, we forgot how much work goes into making this magic mundane.

For instance, do you know how much software and data engineering went into creating this data order form?

Do you know the international treaties that enable the sharing of this data? The small army of people who worked to clean up and standardize this global dataset? It took a whole lot of work to make this look easy. But it was anything but easy or simple.

Related:

I explain why this issue is important in Benchmarking fracking.

It was just one of those things...

It was just one of those weeks. Something that I thought should be straightforward, but tedious, turned out to be really hard. Several times, I couldn't believe what I was seeing. I thought I was going crazy because commands that used to work, generated screenfuls of error messages.

It got so bad at one point, I did the UNIX equivalent of "WhereTF am I?" and typed pwd (print working directory) and THAT generated errors. There was nothing left to do but to log out (of that interactive session on the supercomputer) and log back into a different node. I mentioned the odd behavior to a coworker, but neither of us connected that incident to the database migration that took place the day before.

I know, those of you more familiar with databases are probably laughing your head off right now.

Finally, after 4PM on a Friday afternoon, all but one interactive node and about half the (75,000) nodes lost contact with the disk system.  There was nothing I could do by hanging around the lab while the IT team worked to rescue the machine.

It was time to go out in the field for some riparian ecosystem research.

 I didn't use my car at all this weekend.

Boulder's bicycle network, and my condo's central location, let me do everything I needed to do by bike.  The trees are so glorious this time of year, I took the loooong way whenever I ran errands.

Oh, I bought the most expensive fabric I've ever bought this weekend.  I thought it was the Liberty Tana Lawn at $44/yard, but it was a heavier shirting at $66/yard.  I didn't check the price until after it was already cut and rung up.  But I love the print and the feel of the fabric; it is wide enough that 1.5 yards should yield a very luxurious shirt.  Think of all the money I saved on gas by not driving.  That can buy a lot of fabric.  ;-)

OOPS, I checked my receipt and Liberty's website.  It is a Tana Lawn after all, and I paid $66 for the 1.5 yard piece, not per yard.  

VW and the dark side of AI

When I heard VW got caught cheating on emissions tests, I was not surprised.

What was surprising is that I could smell french fries every time a coworker drove by in her 'clean' diesel VW bug running on biodiesel (a mixture of diesel and filtered used fryer oil from restaurants.)

How could the emissions from her car be clean if the aromatic chemicals that give off the characteristic french fry smell were not combusted beyond recognition?  How could such incomplete combustion pass the strict California emissions tests?

How did VW game emissions testing?

This Computer World article gave a log of possibilities but no definitive answer.

Arvind Thiruvengadam, a research assistant professor at West Virginia University, was involved in a project last year that evaluated tailpipe emissions of diesel cars made by European manufacturers for the American market.
...
Thiruvengadam said he hasn't researched the software that allowed Volkswagen to cheat on the tests. But he did say "there were lots of ways an electronic control unit could be programmed to identify testing and change its fuel mapping toward low emission in those rare scenarios."

For example, modern cars can sense when a hood is open for dynamometer testing, "so a smart hood switch could double as a defeat device."

Or, another sensor could detect when a vehicle's traction control unit was disabled, which is required during emissions testing, and place the emission system into a different mode.

"The possibilities are almost endless," he told Autoblog. "I'm pretty sure that if you're one of the largest car manufacturers, you could do a lot more."

Some of these methods could produce false positives--assume they are being tested when they are actually being driven on the road.

For instance, if the motor is running but the car is not being actively steered, can you assume the car is on rollers for a test?  What if the car is instead being driven on a straight desert road?

In that case, switching the car to 'clean' mode would drastically reduce the fuel efficiency and the car could run out of fuel in a remote location.  If many VW owners report the same problem when driving straight, often remote, roads, the gig would be up.

My aha! moment came when one article said the method involved steering, engine use, AND pressure sensor data.

According to the US EPA violation letter to VW, sensors that could be used in a 'defeat device' need to be disclosed along with the reason why the device is needed for a non-defeat purpose.

I could imagine pressure sensors being useful for a fuel injection system or a rough altimeter, but not while the car was in motion.  Other methods would work better due to Bernoulli's Principle.  So why would VW put pressure sensors in their cars?

A moving car is shaped a little bit like a wedge or an airplane wing.  The air above the car moves slightly faster than the air that flows below the car; Bernoulli's principle states that pressure exerted by a fluid decreases as the fluid velocity increases.

If the motor is running for many minutes without active steering AND the pressure sensor on top of the car does not drop, then the car is likely on rollers in an emissions lab.

Artificial intelligence, AI, in your car determines when to apply the ABS brakes, traction control and the ratio of air to fuel.  It can also be used to cheat on emissions tests.

How can we find a simple algorithm like this--that can be written in just a few lines--among the 100 million lines of code in some new cars?

Update:
The Upshot in the NYT answered (or tried to answer) Little Hunting Creek's question of excess mortality due to VW emissions.

Not a sewing video

In case you want to hear my actual voice (not the POV 'voice' you read at BMGM), you can view a tutorial I taped as part of the Yellowstone User Seminar series. Yellowstone* is NCAR's main supercomputer. (The older, retired ones are reserved for use by my section.)

Citizen scientists who don't have an account on Yellowstone may find the general info on how to find and use our open access weather and climate data useful.  In my years as a SAHM and citizen scientist, I downloaded and practiced data wrangling/mash-ups/analysis/science with RDA data.  When they had a job opening, I was in a good position to go pro.

* Since last June, YS has slipped from #29 to #50 on the list of world's fastest supercomputers.  You think supermodels have a short shelf life at the top?  Try being a supercomputer.  Read Slice of Sky.

Dumb 'Smart' Objects

Please read Zeynep Tufekci's Op-Ed, Why ‘Smart’ Objects May Be a Dumb Idea.  I've worked with enough software to NOT want my car to be connected to the Internet.

After reading Tufekci's Op-Ed, you may want to read my What Do Automobiles and Spacecraft Have in Common?, which I am reposting in its entirety here because The Atlantic is now serving tons of annoying ads with content I let them post for free as a favor. This blog is ad-free. If you want to link to this article, link to it on this blog.

I didn't explicitly state then, but I will now, hooking up objects to the internet, when you don't know what is in the software, is a VERY BAD IDEA.  Software is layered on top of each other.  Software written by different teams working in different companies or at different times, often without awareness or intent to be layered up other software, is prone to breaking.  Breaking is not the same as braking.

Do you want to learn that your car's software broke while you are driving 75 mph on I-5 and heading into 'the Grapevine'?

This article was originally written in March 2011.

What Do Automobiles and Spacecraft Have in Common?

Software, and lots of it!

The issue of Toyota's sudden acceleration problem came up at both the last PTA meeting I attended and at my mother-in-law's dinner table last weekend. Perhaps this digression about real-time software is of general interest.

Software has become embedded into so many things we use every day that it has become invisible to us. Because of its ubiquity, we are not using independent pieces of software, but rather systems made up of smaller interacting subsystems, each with their own software. Because of this complexity, it can be difficult to trace a root cause when a problem arises.

Like this? Follow The Atlantic on Facebook.
In grad school, I wrote and worked with computer models that try to describe physical processes. That led to numerical weather prediction (NWP), which led to satellite data processing -- how the signal off a satellite sensor gets turned into bits which flow down to a groundstation and get turned into information that helps NWP do a better job of predicting the weather.

That's all software. But there is a fundamental difference between the type of software I have written and real-time systems.

No one cares how long it takes for a grad student's model to run, except for the grad student that wants to finish up and graduate.

NWP predictions have to run faster than elapsed model time or else it would be a hindcast instead of a forecast.

Real-time software must run at the same speed as elapsed time. When they are inseparably installed in a machine to control or take readings from it, they are called real-time embedded software. So the answer to the title question should have been that both automobiles and spacecraft contain large amounts of embedded real-time software.

The similarities don't stop there. Both are complex systems of subsystems, each with their own computer and software. The manufacturer of the automobile or spacecraft likely did not build all of the subsystems. So they become an integrator of subsystems and software. The system becomes very complex, and difficult to test robustly. That is, it is difficult to design tests that cover all possible scenarios or paths down a decision tree.

In the satellite software world, the standard is to "test like we fly." (I imagine the automotive industry does the same.) All of the software in the system will be tested together for weeks or months at a time, with all manner of monkey wrenches thrown in the mix, to make sure everything behaves in the intended manner.

Toyota's engineers didn't find a sudden acceleration problem before the cars went to market. Even after the accidents, they tried to simulate the events the motorists described and they still couldn't find a software root cause. What were they missing?

When you can't find what you are looking for, it is logical to ask someone with fresh eyes to look for it. You can't hand over your software to a competitor to test. That's too much temptation. It's better to ask a neutral party, perhaps someone in government for help. In case there is an industry-wide myopia to the root cause, it helps to ask someone in a different industry who does the same functional task.

Hence, NASA Goddard Space Flight Center was called in. GSFC integrates many satellites and has both the laboratory capability and the human expertise to perform that kind of testing and analysis.

[Speaking as a private individual and not as a representative of my employer or any of the government agencies that fund us, I would like to make an appeal. Those scientists, engineers and technicians need to get paid so they don't drift into other industries that do pay. If they weren't there, who would be there for us when a problem arises? It pains me when our politicians spout off about "killing the beast" and "getting government off the backs of industry." I think we should send them on a fact-finding mission to Somalia so they can see what a country without a functioning government looks like. Perhaps they want to move there?]

Remember the part about asking fresh eyes for help when you are stuck? A group of similar individuals can fall prey to groupthink. How can we trust a piece of software or a new drug with our lives if we aren't sure about the design and testing process? And how can we do that if only a narrow cross-section of society engages in that process? That's the best argument for a diverse workforce I have ever seen.

In science and technology, we test and test again. The more the results stand up to multiple experiments in different laboratories, and different experimental designs, the more we can trust the results. STEM (Science/Technology/Engineering/Mathematics) needs people with all kinds of backgrounds to point out things that other people may have missed.

In closing, I would like to point out it is never too late to learn something new and learning come can come from unexpected places.

My daughter is on her middle school Lego robotics team. Like many public school teams, they are at a disadvantage relative to the boy/girl scout troops and private schools because they have one teacher to 32 students. A teammate's dad had been helping out one afternoon a week. When his project at his market work was heading into PDR*, he sent an e-mail asking me to take over.

I duly bought a Lego NXT 2.0 kit and some books, installed the software on my laptop and set out to learn how to program a Lego robot. It turns out that the Lego API (application programming interface) is a variation of LabVIEW, a real-time programming tool used in many labs. At the 2010 Workshop on Spacecraft Flight Software, I learned that LabVIEW is used by MIT students to prototype spacecraft control algorithms!

My daughter helped me program a robot to run a (American football) post play. That means I do have some real-time programming experience after all. But I had to learn it from my 10-year-old.

I would like to thank Keith Blount for his excellent introduction in DIY software. Let's turn every house into a software house.

*PDR stands for preliminary design review. All satellite programs go through thorough design reviews at various stages or milestones. Engineers and managers for both the contractor and the buyer work very long hours leading up to and during the design review process. PDR is the last chance to catch gotchas before proceeding to build the satellite.

I don't mean to shill for my blog, but I have covered this beat for years.

Mommy Art (and Science) explains why software jobs can be highly compatible with child-rearing.

Rockin' deals with the historical reason why the NWP field enjoys a relative abundance of women. I collected one oral history about what happened before, during the depression and WWII, but I need to collect more evidence. Better yet, a professional historian should cover this because I need to go to work now.

Data as a foreign language

Some say that data is a different language.  I agree that it's a specialized language, often requiring domain or subject matter expertise.

But it still amused me to see that Google Chrome detects this page as Norwegian.

Saying, "yes", to translation did not result in any noticeable changes.

Use this page to access an archive of NASA AIRS brightness temperature data in WMO BUFR format.  Yes, I need to update the software tab to include more recent readers in Java and Python.

BTW,
NCEP = National Center for Environmental Prediction
GDAS = Global Data Assimilation System (DA is the complex process of inserting data into numerical weather forecasts)

Are you going to AGU?

I should be working on my poster presentation.  But, I got hung up on some technical hurdles and further sidetracked by user questions.

Instead, I wrote up some user guidance documents that explains some things data users should understand so that they can use data appropriately and responsibly.  (This is the iceberg in big data.)

• The difference between the Final Analysis and the Global Forecast System
• Analysis, reanalysis, forecast—what’s the difference?
  

Go ahead, geek out!

I beg to differ

I've been thinking meta lately and coming up with more questions than answers. But, one of the things I am sure about is that there is something wrong with our socially constructed way of valuing art and artists and also science and scientists.

I'm halfway through Van Gogh on Demand, by Winnie Won Yin Wong, a book that explores the plight of many varieties of Chinese artists on different social and economic levels.  I highly recommend the book, based on what I have read so far.  If you can't find it, you can read the PhD thesis on which the book is based for free.

I'm doing major housekeeping on two major data sets.  This weekend, I'm babysitting supercomputer batch jobs that should (hopefully) run for several weeks.  While keeping an eye on things from home, I came across this Is Computer Coding an Art? via How Creative is Coding? This paragraph quoting, Vikram Chandra, stopped me cold:

But the virtues of what might be called “beautiful code” are different than those of beautiful art. “Beautiful code,” he writes, quoting Yukihirio “Maz” Matsumodo (the creator of the Ruby programming language), “is really meant to help the programmer be happy and productive.” It serves a purpose. Art, by its very nature, serves no purpose. Code is practical and logical. Art is about affect, associations, and emotional responses—part of what Chandra calls dhvani. The term, developed by Anandavardhana, a ninth-century Indian literary theorist, derives from a word meaning “to reverberate.” Dhvani is resonance or “that which is not spoken,” as Chandra says. Code is explicit. Art can be irrational and leave some of the most important things unsaid.

I'm especially repelled by "Art, by its very nature, serves no purpose."

Regular readers of this blog know me as a connoisseur of practical art and craft who enjoys amateur dabbling in same.  One friend calls my experimentation and documentation of remaking castoffs into new clothing a piece of performance art.  I take that as high praise.

Back to the point...

I am biased.  I think we should expand the definition of art to those fields and materials practiced primarily by women that produce beautiful as well as useful artifacts.

But, even if art objects serve no materially practical purpose, they .can. serve a purpose.  Does it illuminate some aspect of the world that was there, but not appreciated?  Do the viewers come away with more understanding of the world or a better grasp of what they don't know?  To repeat a cliche, art applies a mirror to society or a window into the human condition or insert your favorite phrase.

(Ok, I am not sure if making a balloon rabbit in polished metal is really art but I'll let other people go there.)

On the flip side, software aka code is not purely an abstraction.  It can control physical objects, such as how a satellite operates or, as I encountered this week, the behavior of tape robots.  One of these days, I want to attend Solid, a conference that explores this theme between software and tangible things.

Coding can create aesthetically-pleasing artifacts such as this 500 mb wind visualization made with help from NCEP and earth.nullschool.net.

This computational artifact of 500 millibar wind fields* helps explain weather (especially rainfall) patterns. Like (some) art, it is both pretty to look at, and provides insight.

Coders and artists both belong to the super set of makers.  That's all I know for certain.

Related:
Mommy Art (and Science)

* Sea level is roughly 1000 millibars.  500 mb is the half-height of the atmosphere, if you were to look at just one level, 500 mb is a good place to start.  The geopotential height of the 500 mb isobaric surface is an especially useful diagnostic tool to locate dry and wet areas; globally, proportional differences in the geopotential height are largest here.

Do you like the way I snuck in "computational artifact" several times?  That's a term I picked up after reading the new College Board and National Science Foundation Draft Curriculum Framework for the new AP Computer Science Principles class.

Janitor or sexy librarian?

I was hopping mad after I read For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights.

Equating my work with a janitorial service?  The nerve!

But, admiring the view from my window and investigating two reports of possible data corruption in one day took precedence over hyperventilating about one ill-informed article.

After rereading the article, I don't think it's as bad as the headline would suggest. Steve Lohr is only guilty of selecting unfortunate quotes and choosing to interview data gold rush miners while ignoring data veterans in the government.

How many times does he have to quote men saying that data science is "sexy" and data wranging/munging/cleaning is not?  Notice that only the men say that.  The women speak more holistically about data work.

If the majority of our time--whether it is the 50-80% quoted in the article or the 80-90% I hear in meetings with other data veterans--is spent on data preparation, then doesn't that make it our "real" work?

I'm going to risk stating the painfully obvious:

SCIENCE IS BUILT UPON A FOUNDATION OF DATA.  IF WE DO NOT ENSURE THE INTEGRITY OF THE DATA, THEN THE ENTIRE SCIENTIFIC ENTERPRISE COLLAPSES.

It's all about the data.  And data support work is a necessary and critical step in order to get correct answers.  Otherwise, it is GIGO (garbage in, garbage out).

It's late, and I need to write a tutorial to teach others how to use open-source data language, R, to read and manipulate GRIded Binary (GRIB) weather data from NOAA/NCEP in order to answer their real-world questions.

After that, I'll be writing tutorials to teach techniques for data fusion--combining different datasets--for new insights.

I'll do that in tandem with curation of an old dataset made for a defense purpose, but with value to many fields.  This requires writing new documentation to introduce the dataset to a new audience of researchers in disciplines as disparate as computer vision/pattern recognition and wind energy.  (Introducing non-expert users to new-to-them data has to be done carefully because terminology varies between fields.  That deserves a post of its own.)

OK, this won't all get done in one night.   More later.

How to take data out for a test drive, part 1

I wear many hats including data curator and data educator.   I'd like to share a couple of videos I made to help people access weather data from the National Center for Atmospheric Research's Research Data Archive (NCAR RDA).

As I mentioned in Flying Solo, I answer help desk questions about the most popular dataset at the archive, NCEP FNL (Final) Operational Global Analysis. NOAA NCEP creates a representation of the full atmospheric state every six hours using all the calibrated satellite, weather balloon, aircraft, ship, and surface data available at analysis time.  Anyone in the world (except in a few embargoed countries) can ftp to their real-time servers and obtain the data for free.  Most commercial weather services rely on this data and repackage it up as their own branded content.

If you have read my statistics and bullshit threads, you know that I am passionate about empowering people to perform their own data analysis.  I hope that spreading data knowledge helps the public become  savvier consumers of branded content and recognize analysis that doesn't pass the sniff test.

I'm creating a data course to teach people how to find, access and utilize free weather and climate data.  I'm posting a couple of proof of concept videos and I'd love to hear your feedback.

If you want to follow along and try this at home (and I hope you do!), then you need to sign up for a free account first. You can choose the appropriate type or organization or even select no affiliation.  (In that case, write "self" under "Organization Name".)

I demonstrated using a Mac with a browser window and a terminal running tcsh. The first method works on any OS, with any browser, not just Chrome.  Watch them full-screen so you can read the type.



After you take a file or two out for a test spin, and are ready to sift through a lot of of data, I recommend you use an automated batch script. You don't need to be able to write your own csh or perl script. Our software generates one for you based on your custom data request. You do need to know enough UNIX/Linux to work at the command line.  If you are using Windows, try the perl script instead.



Enjoy! And please give me your suggestions on how I can improve them.

I do need to explain that global grids of atmospheric data can become very large; they are packed in a gridded binary format (GRIB1 or GRIB2) agreed upon by all the nations belonging to the World Meteorological Organization.  You need specialized software to unpack and use them.  For these exercises, select the newer GRIB2

Next up, "GRIB1 or GRIB2? What's the difference and how do I choose?" and "What the heck do I do with this binary data?"

In case you can't wait for the next episode, download NASA's Panopoly viewer and you will be slicing and dicing data in your kitchen in no time.

Flying solo

On day four on the new job, my boss made me the data curator of record for a popular dataset.  Gulp.  Day FOUR!  I did NOT think I was ready.

Within an hour, I fielded the first phone inquiry.  Luckily, it was an easy question and I knew the answer off the top of my head.

In two weeks of answering questions and reading the download logs, I realized how many data users come from poor countries that cannot afford pricey data analysis and visualization software.

Our family trip to Tanzania in 2010 taught me the value of convertible currencies--or rather how hard  life can be when your nation's currency is not accepted as payment abroad.  If you need something made abroad, how do you pay for it?  You need to have something that people will pay you for in dollars, euros or yen so you can convert it into dollars (if needed) to purchase it.

This has major implications.  Say you need gas or diesel for fuel and asphalt for roads.  The term petrodollars refers to the OPEC agreement to set the price of crude oil in dollars.  If your currency isn't convertible to dollars, how do you get the crude oil to make that gas, diesel and asphalt?  What kind of transportation network would you have without dollars?  Without all-weather roads and vehicles, how do you grow your economy and get your goods to markets where they will fetch better prices?

I can't control OPEC, but I can help--in my own small way--by helping scientists and risk managers in poor countries access and use weather and climate data without spending scarce convertible cash.

Behold, my current obsession.

They still need color bar legends and I need to streamline the R code a bit.  But, I'm making good progress towards a dataflow that decodes GRIB data and makes it ready for further analysis--all using public domain or open-source software  Furthermore, it is my goal to figure out cross-platform and easily accessible ways to do this.

Can you pick out the summer hemisphere?  Or the day/night sides?  Yowza, Siberia and the Himalayas are cold.

Do you know why the tropopause temperature appears to be anti-correlated with the surface temperature?

Anyway, I was just getting the hang of being a reference librarian for this dataset and making plans to update the help pages (including adding tutorials!) when my boss added a half dozen more datasets to my workload.

They are not nearly so popular, so they shouldn't demand as much time.  But, seriously? People use BUFR in real life?

My boss has plans to put me on YouTube so I can teach people how to be bona fide weathergirls.  Stay tuned for a link when the tutorials go live.

Scratch and Pack

The MIT Club of Southern California is hosting Scratch Day with three family-friendly events in Los Alamitos, Irvine and Mission Viejo.

The Irvine location is SOLD OUT. Walk-in registrations at the Los Alamitos and Mission Viejo locations will be accepted on a space-available basis only.

While Bad Dad and Iris are bonding over Scratch tomorrow, I will be home packing up my sewing room and clothes.  I need help because I am still not fully mobile.

The kitchen, office and linens are mostly packed.  The bathroom stuff won't take much time.  I could really use a hand (and legs!) to help with my sewing and craft stuff tomorrow.

Do any LA-area sewists want to come help?  I suffer from SABLE, stash accumulation beyond life expectancy.  You can take home lots of free sewing supplies.  ;-)

Email and/or leave a comment if you can join me in my Redondo Beach home to sift through my sewing stuff.

That Ridiculously Resilient Ridge in (non)action!

Thanks to some sips tips from Stackoverflow and OSXDaily, my sister's pointer to gifmaker.me, and the NOAA/ESRL Daily Mean Composites visualizer,  I managed to create an animation of the quasi-stationary Ridiculously Resilient Ridge (RRR) between December 23, 2013 and January 21, 2014.

The RRR is a remarkably scary phenomena because it's been sitting there, deflecting rain from California, for THIRTEEN consecutive months.  In fact, the high-pressure ridge is often so large, it's deflecting moisture from most of the west coast of Canada and the Unites States!

Remember Newton's First Law of Physics?  An object at rest stays at rest unless acted upon by an external force.

The earth's atmosphere behaves like a shallow pan of fluid on top of our rock ball.  The atmosphere responds with waves when pinged (water analogy) or plucked (string analogy).  You can see the opposite phases of the waves in blue (low pressure) and red (high pressure) below.  Normally, the waves move around a bit, spreading the sunshine and rain over time and space.

However, the RRR has been quasi-stationary with practically zero momentum for more than a year.  It would take a lot of energy to budge something so big and so stationary.  The strength and persistence of the RRR makes that an unlikely event.

This is a severe event.  In a widespread drought like this, it's simply not an option to pull water from another water shed (e.g. the Colorado River Basin).  No one in the west has any water to spare.  This is not media hype.  This could be a catastrophic disaster.

It't time to prepare for the worst drought and wildfire season in California in my lifetime.

Aside:

I showed it in Lambert Conformal (conic projection) instead of Polar Stereographic this time.  Which do you prefer?

PSA about that annoying pop-up window

Are you getting an annoying pop-up window asking you to login to www.saraimitnick.com:80: when trying to view a page or leave a comment on this blog?

I fell prey to a Blogger glitch and I'm putting this info out in case you also experience it on your Blogger blog.  I'm generally careful not to hotlink to other sites.  I have made an exception for a very large file from the CIMSS satellite blog, but I asked permission first.

Imagine my surprise when I learned that I was inadvertently hotlinking to www.saraimitnick.com:80: and triggering this login window when trying to view any page on this blog displaying the Blogroll widget.

If you run the Blogger Blogroll widget on the sidebar of your blog, then it hotlinks to images on other blogs. That's not a problem for other Blogger blogs. But, for some non-Blogger blogs, such as the Colletterie blog, this can be construed as bandwidth theft.  The owners of those servers have a legitimate complaint and may use means (such as this pop-up) to restrict such uses.

Anyway, I took the Colletterie blog off my Blogger Blogroll widget.  Clear your server cache and this annoying thing will stop happening to you when you view my blog.  If you also have the Colletterie blog on your Blogger Blogroll, be aware that the same thing may happen to your blog.

The webmaster at the Colletterie blog is aware of the situation and trying to find a resolution that won't lose them inbound links.  If you experience this problem, I hope you find this easy short-term fix useful.

Admiral Grace Hopper gets her own Google Doodle

The San Diego Supercomputing Center wrote:

Her work embodied or foreshadowed enormous numbers of developments that are now the bones of digital computing: subroutines, formula translation, relative addressing, the linking loader, code optimization, and even symbolic manipulation of the kind embodied in Mathematica and Maple.

I've also written previously about her.