sequestering carbon, several books at a time CLXIII

The latest batch, comprising his'n'hers reciprocal birthday presents.

sequestering carbon, several books at a time CLVII

The latest batch, including a few belated Christmas presents that were ordered well in advance, but delivered less so.

daylight savings costing coding

We have four sources of data we combine to get our various solar power usage charts.  We have data from the production of the solar panels on the house (PV1&2) and separate data from the new solar panels on the garage roof (PV3).  We have data from a Wattson meter on how much power we consume in the house, and data from the Zappi on how much we use to charge the car (either overnight off the grid, or during the day from excess solar).  Combining these is tricky, because they all have different granularities (from every 30s to every hour).

The combination is even trickier when the clocks change.  None of the data sources tell us whether the timestamps are in GMT, BST, or something else, but we have guessed, probably correctly.  When the clocks went back last week, the Wattson and PV3 systems simply repeated the times for the relevant hour, with the different data.  But the PV1&2 and Zappi systems overwrote the first hour of data.  Aaargh.

Thankfully for my sanity at least, it's my other half who combines the data.  I use the combined data to plot the charts.  So he was tearing his hair out, rather than me doing so.  He has now completed the task.

But still.  Honestly, it's not as if daylight savings is a new thing!  It reminds me of the very first item on the list of Falsehoods programmers believe about time.  At least it gives me an excuse to re-read the list, anyway.)

sequestering carbon, several books at a time CLIV

The latest batch:

Some of these are due to recommendations at the Belfast Eastercon; some from the Seattle Worldcon (which we attended virtually).

sequestering carbon, several books at a time CXXXIX

The latest batch:

sequestering carbon, several books at a time CXXVI

 It's been a while since the last update, so there are rather more in the pile than usual:

sequestering carbon, several books at a time CXX

 The latest batch:

• The Sandman boxed set is still in its clingwrap, so a bit fuzzy
• The two Natural Computing series Springer volumes are complimentary copies: I have just joined the series editor team.  If you want to write or edit a book for this series, in the field of Unconventional Computing or Artificial Life, please contact me!

sequestering carbon, several books at a time CXIX

 The latest batch:

sequestering carbon, several books at a time CXVIII

 The latest batch:

sequestering carbon, several books at a time CXVI

 The latest batch:

King Charles I surviving

 I don't know whether to be horrified, amazed, disturbed, or admiring.  Why not all four?

What in god’s name have i done pic.twitter.com/VylSEdUx9X

— James 💜 Loxley (@oldnorthroad) March 12, 2021

(h/t BoingBoing)

 

sequestering carbon, several books at a time CX

 The latest batch:

book review: Generative Design

Hartmut Bohnacker, Benedict Gross, Julia Laub, Claudius Lazzeroni.
Generative Design: visualize, program, and create with Processing.
Princeton Architectural Press. 2012

---

Having read Pearson’s introduction to Generative Art with Processing I was in the mood to move on to the next level. Hence this book, also based on the interactive Processing language, but with many more, and more sophisticated, projects. These cover bothart and design.

The book is in three main parts. First, Project Selection, is over 100 pages of glossy pictures, whetting the appetite for what is to come. Second, we get Basic Principles, starting with an introduction to Processing, and chapters on working with colour, shape, text and images; these projects are quite sophisticated in their own right, but each focusses on a single aspect. Finally, we get Complex Methods: more ambitious projects combining the concepts introduced earlier.

All the code is available online (in Java mode), which provides an incredibly rich resource to start working from. I didn’t directly use any of this code; I did, however, get inspiration from the Sunburst Trees project to write some of my own (Python) code to draw basins of attraction of elementary cellular automata:

A lovely book all round: great content, and beautifully typeset.

---

For all my book reviews, see my main website.

book review: Generative Art

Matt Pearson.
Generative Art.
Manning. 2011

---

This is an introduction to producing generative art using the Processing language. I had a brief fiddle around with Processing a while ago, and produced a little app for playing around with the superformula; I read this book to see how Processing is used for art. Processing was invented to be an “easy” language for artists to learn. In its original form, it is based on a stripped down version of Java. I discovered with a bit of Googling that there is also a Python Mode available, which I find preferable.

The book has an introduction to generative art, and introduction to Processing (Java Mode), and three example sections on its use for art: emergent swarming behaviour, cellular automata, and fractals. There are lots of good examples to copy and modify, and also lots of pictures of somewhat more sophisticated examples of generative art.

There is a lot of emphasis on adding noise and randomness to break away from perfection: [p51] There is a certain joylessness in perfect accuracy. Now, fractals are one area that can provide exquisite detail, but are they too accurate? I decided to take his advice, and add some randomness to the well-known Mandelbrot set: instead of a regular grid, I sampled the space at random, and plotted a random-sized dot of the appropriate colour:

It certainly has a different feel from the classic Mandelbrot set picture, but I’m not going to claim it as art. However, the Python Mode Processing code is certainly brief:

def setup():
    size(1200, 800)
    noStroke()
    background(250)
    
def draw():
    cre,cim = random(-2.4,1.3),random(-1.6,1.6)
    x,y = 0,0
    n = 0
    while x*x + y*y < 4 and n < 8 :
        n += 1
        x,y = x*x - y*y + cre, 2*x*y + cim
        
    fill((n+2)*41 %256, (256-n*101) % 256, n*71 %256)
    r = random(2,15)
    circle(cre*height/4+width/2,cim*height/4+height/2,r)

Note for publishers: don’t typeset your books in a minuscule typeface, grey text on white, with paper so thin that the text shows through, if you want anyone over the age of 25 to read it comfortably. I frankly skimmed in places. Nevertheless, this book should provide a good introduction to Processing for artists, providing basic skill that can then be incrementally upgraded as time goes by.

sequestering carbon, several books at a time CIII

The latest batch.  They didn’t all arrive at once; they’ve been slowly piling up, but I’ve only just got around to databasing them.

sequestering carbon, one Christmas at a time VII

His ’n’ hers Christmas presents:

sequestering carbon, several books at a time XCIX

The latest batch

sequestering carbon, several books at a time XCVII

The latest batch, photographed in late evening solstice sunlight:

a student of the t-test

Statistics tests can seem like magic, but many of them work on the same underlying principle.  The starting point is usually some null hypothesis (which you hope to refute):

• you have some (possibly conceptual) population \(P\) of individuals, from which you can take a sample
• there is some statistic \(S\) of interest for your problem (the mean, the standard deviation, ...)
• you have a null hypothesis that your population \(P\) has the same value of the statistic as some reference population (for example, null hypothesis: the mean size of my treated population is the same as the mean size of the untreated population)

A statistical test of this hypothesis can work as follows:

• the statistic \(S\) will have as a corresponding sampling distribution \(X\)
• this is the distribution of the original statistic \(S\) if you measure it over many samples (for example, if you measure the mean of a sample, some samples will have a low mean – you were unlucky enough to pick all small members, sometimes a high mean – you just happened to pick all large members, but mostly a mean close to the true population mean)
• if you assume the population has some specific distribution (eg a normal distribution), you can often calculate this distribution \(X\)
• when you look at your actual experimental sample, you see where it lies within in this sampling distribution
• the sampling distribution \(X\) has some critical value, \(x_{crit}\), dependent on your desired significance \(\alpha\); only a small proportion of the distribution lies beyond \(x_{crit}\)
• your experimental sample has a value of the statistic \(S\), let’s call it \(x_{obs}\); if \(x_{obs} > x_{crit}\), it lies in that very small proportion of the distribution; this is deemed to be sufficiently unlikely to have happened by chance, and it is more likely that the null hypothesis doesn’t hold; so you reject the null hypothesis at the \(1-\alpha\) confidence level 

This is all a bit abstract, so let’s look in detail how it works for the Student’s \(t\)-test.  (This test was invented by the statistician William Sealy Gossett, who published under the pseudonym Student, hence the name.)  One of the nice things about having access to a programming language is that we can look at actual samples and distributions, to get a clearer intuition of what is going on.  I’ve used Python to draw all the charts below.

Let’s assume we have the following conceptual setup. We have a population of items, with a population mean \(\mu\) and population variance \(\sigma^2\). From this we draw a sample of n items. We calculate a statistic of that sample (for example, the sample mean \(\bar{x}\) or the sample variance \(s^2\)). We then draw another sample (either assuming we are drawing ‘with replacement’, or assuming that the population is large enough that this doesn’t matter), and calculate its relevant statistic. We do the \(r\) times, so we have \(r\) values of the statistic, giving us a distribution of that statistic, which we show in a histogram.  Here is the process for a population (of circles) with a uniform distribution of sizes (radius of the circles) between 0 and 300, and hence a mean size of 150.

Even with 3000 samples, that histogram looks a bit ragged.  Mathematically, the distribution we want is the one we get in the limit as the number of samples tends to infinity.  But here we are programming, so we want a number of samples that is ‘big enough’.

The chart below uses a population with a standard normal distribution (mean zero, standard deviation 1), and shows the sampling distribution that results from calculating the mean of the samples.  We can see that by the time we have 100,000 samples, the histogram is pretty smooth.

So in what follows, we use 100,000 samples in order to get a good view of the underlying distribution.  An the population will always be a normal distribution.

In the above example, there were 10 items per sample.  How does the size of the sample (that is, the number of items in each sample, not the number of different samples) affect the distribution?

The chart below takes 100,000 samples of different sizes (3, 5, 10, 20), and shows the sampling distributions of (top) the means and (middle) the standard deviations of those samples.  The bottom chart is a scatter plot of the (mean, sd) for each sample.

So we can see a clear effect on the size of samples drawn from the normal distribution

• for larger samples, the sample means are more closely distributed around the population mean (of 0) – larger samples give a better estimate of the underlying population mean
• for larger samples, the sample standard deviations are more closely and more symmetrically distributed around the population std dev (of 1) – larger samples give a better estimate of the underlying population s.d.

The distribution of means varies with the underlying distribution (its mean and standard deviation) as well as the size of samples taken. We can reduce this effect by calculating a different statistic, the \(t\)-statistic, rather than the sample mean \(\bar{x}\).
$$ t = \frac{\bar{x}-\mu}{s / \sqrt{n}}$$ The chart below shows how the distribution of the \(t\)-statistic varies with sample size.

In the limit that the number of samples tends to infinity, and where the underlying population has a normal distribution with a mean of \(\mu\), then this is the ‘\(t\)-distribution with \(n-1\) degrees of freedom’. Overlaying the plots above with the \(t\)-distribution shows a good fit.

The \(t\)-distribution does depend on the sample size, but not as extremely as the distribution of means:

Note that the underlying distribution being sampled is normal with a mean of \(\mu\), but the sd is not specified. To check whether this is important, the chart below shows the sampling distribution with a sample size of 10, from normal distributions with a variety of sds:

But what if we have the population mean wrong? That is, what if we assume that our samples are drawn from a population with a mean of \(\mu\) (the \(\mu\) used in the \(t\)-statistic), but it is actually drawn from a population with a different mean?  The chart below shows the experimental sampling distribution, compared to the theoretical \(t\)-distribution:

So, if the underlying distribution is normal with the assumed mean, we get a good match, and if it isn’t, we don’t. This is the basis of the \(t\)-test.

• First, define \(t_{crit}\) to be the value for \(t\) such that the area under the sampling distribution curve outside \(t_{crit}\) is \(\alpha\) (\(\alpha\) is typically 0.05, or 0.01).
• Calculate \(t_{obs}\) of your sample.  The probability of it falling outside \(t_{crit}\) if the null hypothesis holds is \(\alpha\), a small value.  The test says if this happens, it is more likely that the null hypothesis does not hold than you were unlucky, so reject the null hypothesis (with confidence \(1-\alpha\), 95% or 99% in the cases above).
• There are four cases, illustrated in the chart below for three different sample sizes:
• Your population has a normal distribution with mean \(\mu\)
• The \(t\)-statistic of your sample falls inside \(t_{crit}\) (with high probability \(1-\alpha\)).  You correctly fail to reject the null hypothesis: a true negative.
• The \(t\)-statistic of your sample falls outside \(t_{crit}\) (with low probability \(\alpha\)).  You incorrectly reject the null hypothesis: a false positive.
• Your population has a normal distribution, but with a mean different from \(\mu\)
• The \(t\)-statistic of your sample falls inside \(t_{crit}\). You incorrectly fail to reject the null hypothesis: a false negative.
• The \(t\)-statistic of your sample falls outside \(t_{crit}\).  You correctly reject the null hypothesis: a true positive.

Note that there is a large proportion of  false negatives (red areas in the chart above).  You may have only a small change of incorrectly rejecting the null hypothesis when it holds (high confidence), but may still have a large chance of incorrectly failing to reject it when it is false (low statistical power).  You can reduce this second error by increasing your sample size, as shown in the chart below (notice how the red area reduces as the sample size increases).

So that is the detail for the \(t\)-test assuming a normal distribution, but the same underlying philosophy holds for other tests and other distributions: a given observation is unlikely if the null hypothesis holds, assuming some properties of the sampling distribution (here that it is normal), so reject the null hypothesis with high confidence.

But what about the \(t\)-test if the sample is drawn from a non-normal distribution?  Well, it doesn’t work, because the calculation of \(t_{crit}\) is derived from the \(t\)-distribution, which assumes an underlying a normal population distribution.

sequestering carbon, several books at a time LXXXV

The latest batch: