The problem with data...

...is that it is so darn hard to use!
This looks very easy, right? But it was a PITA to get from the data I downloaded from the Guardian website to the plots shown here.

I write notes to myself about data workups, both the statistical/science decisions and the software tricks. I thought I would show you how unnecessarily hard this is.

[I used to be embarrassed about how much time I spent on opening data files and getting different data files to play nice with each other--a process called data fusion. But, then I heard a bigwig at USGS state at a conference that her staff devote 85% of their time troubleshooting data reading issues and only 15% on the science stuff. I am so grateful for her openness. How else would I know that I hit par?]

So here are my data reading notes.
First, if you use your browser to go their link, All the US weather stations (CSV)
and then save it as plain text using Firefox (my browser of choice but YMMV), you get one file with all of the data written on one line!
I like to work up data on my Macbook Pro using X11 and the Mac flavor of Unix.
prompt> wc us.csv
1 21678 967827 us.csv
Yup, that's one line, 21678 words and 96827 characters.

To break it up into multiple lines that I (and my machine) can read,
I used vi and ex
prompt> vi us.csv
then type this easy to parse line (ctr-V means press ctr and v at the same time)
:g/ctr-V ctr-M/s//ctr-M/g
That turned the ^M characters that read as tabs on my computer
to Unix carriage returns.

Then I tried to use grep to find files tagged with
CA
and
CALIFORNIA
but discovered that the structure of the data for each station was slightly different.
Some were marked in the second field, after station name,
CALIFORNIA
CALIFORNI
CALIFORN
CA
and LOS ANGELES just omitted the state field altogether.

In all, I found 6 climate stations in California
(if you find more, please leave a comment!).
From North to South:
Eureka
Sacramento
San Francisco
Fresno
Los Angeles
San Diego

[If I were smart, I would have wrote a script to grep for every CA station in the Worldwide WMO station list by WMO identifier against the us.csv climate file. Consider this your homework assignment and send me the stations that I missed.]

I used unix again.
prompt> grep FRESNO us.csv > fre.csv
and so forth for each of the 6 stations to put them in a separate file.

Then I had to get rid of the extra column of CA variants in 5 of the 6 fields. I used MS Excel to open up each station's csv file, delete the column, and save again in csv.

There should be a way to do it using sed or awk, but I forgot how to do it. When I looked it up, the simple
prompt> awk < fre.csv '{print $1, $2, $3}'
showed that I had real problems.
prompt> awk < fre.csv '{print $1}'
gives FRESNO, as you would expect, but
prompt> awk < fre.csv '{print $2}'
gives
CALIFORNIA,723890,36.8,119.7,2009,8.8,11,13.4,16.8,24.1,24.3,29.4,27.7,26.5,17.6,,
because awk is expecting spaces as delimiters and there are no spaces after
CALIFORNIA FRESNO, CALIFORNIA,723890,36.8,119.7,1887,,,,,,,,,23.7,19.9,13.2,7.7

What to do?
prompt> awk < fre.csv '{FS = ","; print $1,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17}'
does the trick, but, by the time I remembered the awk syntax, I had already used MS Excel to do it for me.

I tried to make plots with Excel, but found it too limiting. So I resorted to my favorite programming language for parsing data, Interactive Data Language, IDL. If you want to be a meteorologist, you need to learn it. If you are at a university, chances are very good that your school has a site license for it.

If you have IDL version 7.1 and later, there is a handy-dandy built-in procedure that reads csv files called "read_csv.pro" !
IDL> eur = read_csv("eur.csv")
IDL> sac = read_csv("sac.csv")
IDL> sfo = read_csv("sfo.csv")
IDL> fre = read_csv("fre.csv")
IDL> lax = read_csv("lax.csv")
IDL> san = read_csv("san.csv")

Type help to find out what is in your data structure.
IDL> help, fre, /structure
** Structure <18d5208>, 18 tags, length=18704, data length=18696, refs=1:
FIELD01 STRING Array[123]
.
.
.
FIELD18 DOUBLE Array[123]

If you cleaned up your fields, then
field01 = station
field02 = wmo_id
field03 = latitude
field04 = longitude
field05 = year
field06 = Jan mean temperature in Celcius
field12 = Jul
field18 = Dec

I reversed the default plot and background colors
IDL> !p.background=255
IDL> !p.color=0
and used color table 39, rainbow + white
IDL> loadct,39

IDL> plot, lax.field05, lax.field06, xrange=[1850,2010], yrange=[0,30], min_value = 1., psym = 1, $
IDL> title = 'Los Angeles Monthly Mean Temps', xtitle = 'Year', ytitle = 'Degrees Celcius'
IDL> oplot, lax.field05, lax.field06, min_value = 1., psym = 1, color = 50
IDL> oplot, lax.field05, lax.field12, min_value = 1., psym = 2, color = 250
IDL> saveimage,'LAX_JanJul_Climate.jpg',/jpeg,quality=100

If you haven't already downloaded and installed Liam Gumley's saveimage.pro, what are you waiting for? You can find it here. Save it to somewhere early in your IDL path.

If you want to get fancy, you can plot all six CA stations on one page.

I want to discuss the features that I see in these graphs, but at a later date. Iris has a TKD party right now.

In loco parentis

Another parent wrote in chalk on the side of their cabin, "I don't know where my children are...I'm at the Lair". That's the Lair spirit! Compare that with this conversation with a mother who didn't have the Lair spirit.

[I agreed with other Lair parents who say that they bring their kids to the Lair so they can roam the woods and the creek without adults trailing after them. The understanding is that all the adults at camp are keeping an eye on all the kids, giving them a long leash, but also stopping dangerous/reckless behavior. E.g. one mom shouted at some kids that were not her own to get off the roof of the stage (albeit carefully); she reasoned that she wouldn't want her kids to be on anything higher than six feet, so she thought those kids' parents wouldn't want their kids up there either.]

This picture captures her in a cuddly mood, but she went mostly free-range last week. Our entire family loves that hammock.
She babysat her 5 yo cousin so that all the adults in our cabin could go on the hike to Sword lake. Notice that the flowers are just coming into flower at this elevation.
Her independence also provides us with independence. It's a bittersweet feeling, though.

Down by the creek for a little bonding time. We walked, talked, painted some watercolors, and observed the standing waves behind the rocks. (Our rafting guide in Colorado had discussed standing waves; I was surprised how much Iris remembered about wave and hydro-dynamics from the guide's talk.)
It's hard to convey how much joy that gave me, discussing and observing waves with my daughter. I worked as a teaching assistant for Quantum Mechanics for three semesters and work as a meteorologist now. Wave mechanics is near and dear to my heart and it warms my mommy cockles to see that they fascinate my daughter, too.

While we were having our internet-free camping trip, international pi-day, 22 July, passed. I still owe you a post about how to know any size circle. I even brought along this shawl so that I could photograph it down by the creek. But I was so in the moment of our vacation, I plumb forgot. You'll have to gaze upon it on our family room carpet instead. More on knitting circles later.

Seashore Dress

We went for a walk and had lunch at the beach on Sunday. Are they having a deep conversation?Or is she complaining .again. about the sand kicked up by the waves? Or how we have ruined her forever by raising her in a weather paradise so that she can never leave?

The dress sure looks great IMHO. Here it is, washed and drying on the dress form. It is outbound to her younger cousin.

I made a size 140 cm (height) or US size 10. But, I didn't realize the dress pattern is for knits until too late. It barely fits over her head and is a tad too short.

Because I was working with a thrifted ruffled skirt, I cut the dress pattern with the bottom along the skirt hem, being careful to align the sides. I forgot to pay attention to the total length and the hem rises up in the center back. It is more of a tunic than a dress.

The front has deep pleats so the front piece is wider than the back piece. Thus, I preserved one side seam so the F piece could extend slightly into the B of the skirt. I also had to make each pleat slightly shallower. I used up almost the entire skirt to make this dress/tunic.

• The full pattern review.
• The Burda 9546 Flickr set, including those hard-to-find Burda envelope front and back pictures and my slapdash interior finishing techniques for children's clothes.

I am under orders to make her a longer one with a wider head opening. I am not sure if I feel cooperative.

Scaling Up

Remember when I reknit my favorites in 2/3 scale for Iris here and here?
2/3 scale turned into 3/4 and then 4/5 scale. I was quite melancholy when I was cutting out yoga pants for the two of us and laid the two patterns on top of each other for comparison. She's growing so fast!
Compare with the all in one (F and B cut in one piece) pattern piece for the first pair of pants I made for her.
We bought her shoes this week for back to school. Remember last year when I realized I could fit in her shoes (though it was a tight fit)? She'll be wearing my shoes soon, if I let her. We are in negotiations.

My Marilyn

When we visited Wupatki National Monument (near Flagstaff, AZ) the blowhole was blowing. We couldn't resist taking this photo.
The sale of the Debbie Reynolds movie costume collection was in the news that week, so I thought about this.
The break-up of the collection fills me with sadness. Read this analysis.

As he picked up costume after costume, Randy Habercamp of the [Motion Picture] Academy shook his head as it sunk in how many iconic pieces were leaving this country for good. “Well,” he said, seeking to put it all in some perspective, “it happened to the treasures of Greece. It happened to Italy. I guess now it’s our turn.” (Randy was there for the experience—the Academy did not send an official representative to bid on costumes.)

Info about construction of the skirt is in Hello Goth!

Top-down road trip pullover

The sweater was knit top-down. We drove our hard top Prius from LA, CA to Boulder, CO.
This was intended as a hostess gift for our friend in Boulder, CO. It's a 16 hour drive so I hoped to have it mostly finished by the time we got there. Hah!

It did take ~15 hours to complete, but I didn't take into account pretty scenery and darkness. At least I was able to try it on the hostess and adjust the length to suit while the sweater was in progress. It will be mailed tomorrow.

It's a variation of the Lion Brand Yarn's free pattern for a top-down pullover. I substituted Dharma Trading's Andee (50/50 wool/alpaca) instead so I had to adjust the gauge. I made many other changes.

• Because it is knit from the top down, I used a stretchy tubular cast-on from a crochet foundation chain (in a waste yarn).
• I misread the instructions and knit a yoke for a size Small instead of the intended Medium.
• That meant increasing generously across the body at the end of the yoke when switching to stockinette stitch portion. Fortunately, it doesn't show much.
• The pattern has the front and neck at the same 4" height. The recipient wanted no more than 2" height. To avoid strangling her (we really like her and want her to stick around), I short-row shaped the neck to bring the center front 2" lower than the center back.
  
• I added waist shaping. You can see that the waist shaping starts 2" further beyond the yoke on the back than in the front.
• I shortened the sweater to fit the petite recipient. She likes things roomy. IMHO, a Small would have fit her just fine. But, this will give her room to layer a turtleneck underneath.
• Not that she will need it because this alpaca and wool blend is WARM.
• I added 3 garter ridges instead of the plain curling hem of the original pattern. I think it looks neater this way.
• The body was knit with size 7 needles, the ribbing and garter bands with size 5.
• The pattern has 3/4 sleeves. I lengthened them slightly to get full-length sleeves for a petite.
• The pattern has straight sleeves. I decreased from 65 down to 45 stitches.
  
• The sweater body hem wanted to flip up (until I blocked it into submission), so I decreased 3 stitches (from 45 to 42), evenly spaced, across the wrist hems to control the flip.
  

The yoke was completed between LA and St George, Utah. The sweater was divided for body and sleeves and the body was short-row shaped through Glenwood Canyon. We got stuck in a huge traffic jam when I-70 was periodically closed to move heavy equipment across the road.

Look at the water level of the Colorado river! It comes nearly up to the new highway and the railroad tracks. It flooded over the bike path and the old highway.
I was able to try the sweater on our hostess and plot the waist, hip and arm widths from her measurements. Then I finished the sweater body in Crested Butte and Mesa Verde, CO.

After lunch in Flagstaff, AZ, I started the first sleeve. It was just about done by dark.

Back at home, I finished the first sleeve and started the second.

During the drive to visit my in-laws in San Diego last weekend, I finished the second sleeve and wove in all the ends.

When I returned home, I washed and blocked the sweater. When I photographed it, I thought about where I was for every piece of the construction. It's full of memories. I hope she likes it.

This is a really nice, simple sweater. If I take another road-trip, I may make another.

Aside:
I asked the recipient to measure a sweater she likes. It sounded too roomy but I went with her measurement. (When buying RTW sweaters, you pick the size that fits your widest point. If you are pear-shaped like us, then RTW sweaters are often too roomy around the shoulders and chest if they fit our hips. Custom-knit sweaters can be knit to fit the wearer's measurements and don't need to be so roomy at the chest. That's a subtle point that non-knitters might not realize.)

But I forgot to ask her the length. So I called her home as soon as we got out of Glenwood Canyon's cell phone dark spot and her daughter said I had just missed her. I recalled that the two are nearly the same height so I asked the daughter to measure her back and front shoulder to waist length.

She said that it would take her a while to go get the tape measure.

When she got back, she asked, "Do you want it in centimeters or inches?"

Spoken like a true daughter of a NIST (National Institute of Standards and Technology) employee!

Ravelry project notes.

Is that all?

In catching up with the local news after vacation, I came across LAT's Buying clothing made locally. I buy sewing supplies from SAS Fabrics, an odd-jobber that buys fabrics and trims from area clothing manufacturers. I have noticed an uptick in the quality and selection of their offerings recently so I was not surprised to read that LA-area clothing production had started to grow slightly after years of decline.

I've even met two women making comfortable livings producing specialty clothing in small lots in my own neighborhood. (And that's not counting Renko.) What did surprise me was this quote.

L.A's Single brand can turn around 800 silk print dresses for Neiman Marcus and Lord & Taylor in as little as two weeks, now that 90% of its production is done at home. "We recently brought more of our production back from China," said Galina Sobolev, designer-founder of the $15-million brand. "The price difference was only about a dollar per [dress], and the quality control and timing were much better. We're also providing jobs and keeping domestic factories working."

It costs just $1 more to make a dress in LA than in China? I checked the retail prices at Single Dress and online retailers. That's $1 more for a dress that retails at $200-400.

In Deluxe, Dana Thomas wrote that her research showed that manufacturing costs account for about 10% of the price of luxury products; advertising and PR cost 11%. Manufacturing in higher wage countries (Europe and North America) costs about 20% more than in China.

The book came out in 2008 so Thomas' numbers are older than the statistic given in the LAT article. $200 is not quite a luxury price point. But, which is right? $1 or ~$5-10 price difference given by Thomas?

I asked Kathleen Fasanella, the creator of Fashion Incubator. She said that I should have read her more closely because she's been reporting about this for years. Kathleen is right, as usual, but she was so far ahead of my thinking, I didn't grock what she was saying until I read these links together in succession.

• Value Circularity: cotton, colanders & the specialty store market If you can read just one link, this is a must-read for why you can't find nice stuff anymore except at boutiques and specialty stores.
• Why you should start your own sewing factory She details why the cost-savings of Chinese production are evaporating. She also proselytizes that producing in small lots (of a few hundred instead of the 10,000-100,000 item lots common in Chinese production) is more cost-effective for the manufacturer and kinder to the planet.
  
• Consequences of the fashion school bubble My pet-peeve about too many chiefs, not enough Indians to do the actual work of making stuff. This is universally true in science, software and clothing.
• Slow vs Fast Fashion pt.1 She didn't recommend this, but I thought this was a much more nuanced position than the one I took in Slow Fashion.