overselling the weather

We haven’t had any rain for over seven weeks, and the garden is really suffering.  So we are constantly checking the forecast to see when this drought might end.  Here’s what my BBC Weather app says for next Wednesday:

Yay ! Rain at last!

So, we can look forward to the drought breaking on Wednesday, then?  Great!  When will it start to rain?  Let’s look at the hour-by-hour breakdown:

Umm

So, it’s going to rain for less than an hour?  How does that make the whole day summary be “light rain showers” and the background a dull grey?

And actually, there’s a mere 12% chance it will rain then.

Yet there’s a 17% chance of rain at 4pm, but no raindrop on that hour’s icon.

What do these symbols and percentages even mean?

sun and rain

You might expect the amount of solar power we generate to increase over April, May, June (we are in the northern hemisphere).  That happened last year.  This year was a bit different.

daily solar power generation spring 2104 (left) and spring 2015 (right)

The average power generated decreased month on month!  (Although, to be fair, the difference isn't very significant.)  Yes despite June having a lower average this year, it also had the sunniest day so far.

This average reduction wasn’t due to rain: May and June 2015 were the driest since (our) records began:

spring rainfall, 2006–2015, area proportional to rainfall

Ironically, as I sit writing this post, it is raining quite heavily…

April showers and sunshine

March winds and April showers bring forth May flowers.

Nope.

According to wikipedia:

One of the major causes of the often heavy downpours is the position of the jet stream. In early spring, the jet stream starts to move northwards, allowing large depressions to bring strong winds and rain in from the Atlantic. 

However, we live in the east of the country, rain-shadowed from Altantic winds.  Since records began (which is June 2005 in our case), April is consistently one of the driest months of our year.

rainfall for 2014 (bars), against statistics for all years June 2005 - April 2015

Even with that one massive outlier in 2012, our recorded mean rainfall for April is 11mm, about half that of the next driest month, March with a mean of 20mm.

However, this April was the sunniest month since records began (in January 2014).  That's the sunniest month, not the sunniest April.

daily solar power generation in kWh, by month

In April 2015 we generated an average of 37.66 kWh per day.  Compare this with the significantly lower 29.60 kWh per day in April 2014, and even with the 37.17 kWh per day in the month with the longest days: June 2014.  18 April 2015 also boasted the maximum power generation in a single day so far: 56.03 kWh.

So we need a new proverb for where we live.  Maybe

March winds and April sunshine bring forth May flowers.

Who cares about rhyme, anyway?

rainfall statistics in d3

My explorations of the d3 data visualisation library for JavaScript continue apace.  I have now added "box-and-whisker" plots to the 2012 rainfall, helping demonstrate the anomalously wet April and July, and dry August, last year.  This required calculating the various means, medians and quartiles, which wasn't particularly difficult, and has given me a feel for working with arrays.

Box and whisker plots show minimum, lower quartile, median, upper quartile, and maximum.
The mean value is overlayed as a blob.

This latest version replicates the previous chart displaying this information, which I did in Excel. (The bars are slightly different, because here I have included the 2012 data in the statistics, whereas I excluded it in the previous version.)

I think this d3 version looks much cleaner.  It certainly lets me fiddle with things at the programmatic level I prefer!

interaction with d3

My explorations of the d3 data visualisation library for JavaScript continue.  Last weekend I dipped my toe in the water, and drew some simple rainfall charts.  The code was clunky, and the data was hardwired, but the charts were pretty enough to convince me to continue.

And so this weekend I've explored some more.  I've modified the code to read in the data from a csv file, and then refactored like mad to make it more elegant.  The result of half a day's work was a web page indistinguishable (on the surface) from before: but now the charts, and the headings, are generated from the csv file data, so I can add more data each month, and the page will update without any code having to change.  So, a success.

Then I decided to add a little bit of interaction: changing the colour of a bar or spot on mouse-over, and showing the actual data value.  The d3 library is powerful enough that it was very easy to do.

in May 2007 we had 76mm of rain

in July 2009 we had 63mm of rain

So that's some tasks completed off my to-do list.  I've still got a long way to go.  But, as when I was learning Python, I'm finding StackOverflow an invaluable resource.

The actual web page is here, but it may change over time as I add more functionality.

my first d3

Way back when I was an undergraduate, I stumbled across a book called Curve Tracing, by Percival Frost.  The college library had a first edition, published in 1872.  Several years later, I came across a newer edition in a bookshop, and grabbed at it.

This beautiful little book has tons of curves defined by equations, and shown as graphs.

Plate IV from Curve Tracing

This was all originally done in the days before computers.  It's been a goal of mine to reimplement some of this work, in an interactive form, because many of these curves have parameters that affect their detailed shape.  So, I've been looking for a suitable tool.

Also, I'm interested in data visualisation (and have long been a disciple of Tufte).  I've tried to follow the "no chartjunk" ethos in my own work.  For example, to plot our rainfall statistics, I had to struggle with Excel charts to remove most of the garish "ink" provided by default. But it's still not perfect.  Additionally, I have some specific work I want to do with modified parallel coordinates, for which there is no existing library.  So, I've been looking for a suitable tool.

Last week a colleague of mine mentioned d3, a JavaScript library for Data-Driven Documents. I browsed the gallery for a while, and fell in love.  I spent yesterday playing around with evaluating it, on simple charts, to see what it could do.  This meant reading Scott Murray's d3 tutorial, and implementing a few simple charts to show the rainfall data. (Oh, and learning my first JavaScript.)

First off, I tried a simple bar chart, starting from Murray's tutorial example (when learning a new language, I usually find it easier to modify existing code than start from an empty file). After fiddling around to get the ordinal x-axis working, I got:

d3 chart of 2012 rainfall in mm

I think that looks suitably chartjunk-free and minimal.  It also has the nice feature of combining the actual numerical values into the bars, giving what Tufte calls both a macro-reading (the bars: gosh, July was wet!) and a micro-reading (the numbers: July had 113mm of rain) in one chart.

What's nice about d3 is the way the axes scale automatically.  Exactly the same code produced these charts (the only difference is the July data value: 13, 113, 233):

automatic axis scaling with changing data values

But, of course, we have several years of rainfall data.  A little more hacking coding gave me a grouped bar chart:

rainfall, June 2005 -- May 2013

Now it's possible to see how wet April 2012 was, compared to normal Aprils, even if it wasn't as wet as that July!  My implementation of this is a little kludgy, with too much hard-wiring, since my goal was to evaluate the capability, not (yet) learn the entire language; my next task is to code it more elegantly.

I've always found grouped bar charts rather cluttered, and so I wondered if there was a better way to show the data.  Rather than use some sort of surface plot, I decide to try a projection where the size of a spot is related to the amount of rainfall.  With a surprisingly small change to the code, the grouped bar chart metamorphosed into a "blob" chart:

(left): blob area proportional to monthly rainfall; (right) blob radius proportional to monthly rainfall

This enables comparisons in both dimensions (years, or months), depending on whether you view rows or columns.

So, based on a day's work, I'm very impressed with d3.  However, there are a lot more d3 facilities I need to get up to speed with before I can start my reFrost project in earnest:

• csv data import -- currently the data is hard-wired into the scripts (ugh)
• data manipulation -- to calculate medians and quartiles for box and whisker plots
• lines -- to draw graphs rather than charts
• maths -- to calculate the functions: and, presumably I'll need a lot more JavaScript
• interaction -- so parameter values can be chosen by the user
• transitions -- so the graphs will smoothly change as parameters are varied
• more -- stuff I don't know about yet, but will need

So, a way to go, but I think I might have identified the tools I need.

a normal April again

Last April was anomalously wet.  This year, things were back to "normal", rainfall-wise, if not temperature-wise.  Despite being much drier, 2013 is still the second wettest April since (our) records began, with a measured 14mm:

April rainfall, in mm

wet 2012

April was wet. Very wet.  For April.

But what about the whole of 2012?  It certainly felt wet.  How wet was it?

Well, since records began (our records, that is, which began in mid-2005), 2012 was indeed the wettest year ever:

annual rainfall, in mm

It certainly bucked the trend of the previous two "drought" years.  No hosepipe bans for a few months more, I expect.  So, April was wet; 2012 was wet.  Was April a particularly wet month in 2012?

mean, median, and 2012 monthly rainfall, in mm

So, April was only the third wettest month, beaten by December, and washed away by July. Strangely, when the north of England was being flooded out in September, down our way (near Cambridge) it was one of the driest months of the year!

Those mean/median figures don't tell anything like the whole story, though.  Here's more information:

min, lower quartile, median, upper quartile, max, and 2012 monthly rainfall, in mm

The box plots summarise the 2005-2011 data, and the blue bars are the 2012 data.  A blue bar within the lower/upper quartile box is nothing special: half the data falls there.  So May, September (the flooding month!), and November were fairly typical.

A blue box between the min/max lines is a little bit special, as only a quarter of the data falls in the first or last quartile (hence the name!).  So January and March were a bit on the dry side, while June, October and December were a bit wet.  (Well, December was quite a bit wet, being nearly at the maximum.)

I love that difference between the May and June data: very similar medians, minima and maxima, but wildly different lower and upper quartiles.  May is essentially bimodal -- wet or dry (this year was one of the dry ones) -- whilst June is middling damp with a couple of outliers (this year was an outlier, too).

A blue box outside the min/max lines is a driest/wettest seen so far.  Four months in 2012 managed this (but since we have only 8 years of prior data, I'm not going to over-interpret the significance of this).  2012 had the driest February (despite it being a longer leap-year month!) and  August, and the wettest April and July (by far!) since our records began.

So, a bit difficult to make any general statements about the rainfall, then.  This is why in the UK were talk about the weather all the time.  It changes, all the time.  All we can conclude from above is the lovely statement I saw in a newspaper many years ago: "it's usual to have unusual weather this time of year" -- or any time, really!