a memory less ephemeral

Saturday, 8 March 2014

sequestering carbon, several books at a time XX

Labels: books

A few more join the hordes:

Friday, 7 March 2014

spring

Labels: garden, weather

It’s hard to believe that this time last year was freezing cold, and snow. This year is a little different.

catkins in sun and shade

luminous crocuses

It’s not just the vegetation. The frogs clearly think it’s spring, too.

Sunday, 2 March 2014

Back in August I blogged about Trello, a great browser-based list manager. I’ve been using it constantly ever since. I use Evernote to manage the past, and Trello to manage the future.

In that Trello blog post, I mentioned I written a bit of Python to backup my boards. Yesterday PK asked in a comment for that code. I thought briefly of putting it up on something like GitHub, but the code is so short, I decided I’d give it here instead.

I also thought about refactoring the code a bit first. I wrote this a while ago, and I know a bit more Python now. But it works. So I’ll leave any refactoring until I have to make some changes. (Well, okay, maybe I made a few changes to remove the worst uglinesses…)

First, there are some libraries to import.

import json
import unicodedata
import urllib2
import datetime

I strip the unicode out of the Trello data, for simplicity. (One day I will spend the time to really understand all the intricacies of unicode in HTML, but for now, out it goes.) As usual, Stack Overflow was of invaluable help in finding a way to do this.

def stripUnicode(ustr):
    # translate unicode (accents) to non-accented chars
    ustr = unicodedata.normalize('NFKD', unicode(ustr))
    ustr = u"".join([c for c in ustr if not unicodedata.combining(c)])
    
    # translate whole string to ascii
    return ustr.encode('ascii', 'ignore')

The overall structure of the code is simple: loop over the boards; for each board, print an HTML header, print the lists (which involves printing their cards), then print an HTML footer.

The HTML header involves a title, importing the CSS style file, and a bit of stuff to control the width (a fudge, frankly, but this is for me, not general purpose code!).

def print_header(fhtm, board, lists):
    print >> fhtm, '<html>'
    print >> fhtm, '<head>'
    print >> fhtm, '  <title>' + board['name'] + '</title>'
    w = len([lst for lst in lists if not lst['closed']])*318
    print >> fhtm, '  <style type="text/css">'
    print >> fhtm, '    @import url(trello.css);'
    print >> fhtm, '    body {min-width: ' + str(w) + 'px;}'
    print >> fhtm, '  </style>'
    print >> fhtm, '</head>'
    print >> fhtm, '<body>'

Printing the lists involves looping over the lists, and for each open list printing a heading, and printing the cards in the list.

def print_lists(fhtm, lists, cards, checklists):
    for trello_list in [lst for lst in lists if not lst['closed']]:
        print >> fhtm, '<div class="list">'
        print >> fhtm, '<H2>' + trello_list['name'] + '</H2>'
        print_cards(fhtm, cards, checklists, trello_list['id']) 
        print >> fhtm, '</div>'

Printing the cards involves looping over the list of cards, and for each card, printing its due data, description, and checklists (including strike-through for completed items). There is potentially more information in a card, but that’s all I currently use. If you use more, you will have to update this bit, but the structure of what’s needed is fairly obvious.

def print_cards(fhtm, cards, checklists, id_list):
    for trello_card in [c for c in cards if c['idList'] == id_list]:
        print >> fhtm, '<div class="card" >'
        print >> fhtm, '<H3>' + stripUnicode(trello_card['name']) + '</H3>'
        print_card_due(fhtm, trello_card)    
        print_card_desc(fhtm, trello_card)    
        for id in trello_card['idChecklists']:
            print_checklist(fhtm, checklists, id)    
        print >> fhtm, '</div>'
    
def print_card_desc(fhtm, card):
    if card['desc'] != "":
        print >> fhtm, '<P>' + stripUnicode(card['desc'])
        
def print_card_due(fhtm, card):
    if card['due']:
        print >> fhtm, '<UL><LI type="square"><I>due</I> ' + card['due'][:10]
                     + '</UL>'
        
def print_checklist(fhtm, checklists, id):
    checklist = [c for c in checklists if c['id'] == id][0]
    print >> fhtm, '<H4>' + checklist['name'] + '</H4>'
    print >> fhtm, '<UL>'
    items = checklist['checkItems']
    items.sort(key=lambda x: x['pos'])   # pos = screen posn, for list order
    for item in items:
        print_checklist_item(fhtm, item)
    print >> fhtm, '</UL>'
    
def print_checklist_item(fhtm, item):
    if item['state'] == "complete":
        print >> fhtm, '<LI><I><DEL>' + stripUnicode(item['name'])
                     + '</DEL></I>'
    else:
        print >> fhtm, '<LI type="circle">' + stripUnicode(item['name'])

The HTML footer is trivial.

def print_footer(fhtm):
    print >> fhtm, '</body>'
    print >> fhtm, '</html>'

There are some secret keys and tokens needed, which you get when you register at Trello, to get access to your own boards.

key = 'a 32 hexdigit string'
token = 'a 64 hexdigit string'

I have hardwired in the various boards I want to backup. The 24 hexdigit string can be found right at the beginning of the board’s JSON file, so you have to download that manually, and copy the string from it. But you only have to do that once.

board_dict = {
    'board1': 'a 24 hexdigit string',
    'board2': 'a 24 hexdigit string',
    ...,
    'boardn': 'a 24 hexdigit string'
}

Then, finally, there’s the loop over the boards in this dictionary: reading in the board data, which requires reading the board, its lists, its cards, and its checklists separately, then printing out the relevant HTML, including the date of the backup:

for filename, board_id in board_dict.iteritems():
    print filename
    
    trello = 'https://api.trello.com/1/boards/' + board_id
    key_token = key + '&token=' + token

    get_url = urllib2.urlopen(trello + '?key=' + key_token)
    board = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/lists?key=' + key_token)
    lists = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/cards?key=' + key_token)
    cards = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/checklists?key=' + key_token)
    checklists = json.loads(get_url.read())

    fhtm = open(filename + '.htm', 'w')
    print_header(fhtm, board, lists)
    print >> fhtm, '<H1>' + board['name'] + ' (' + str(datetime.date.today())
                 + ')</H1>'
    print_lists(fhtm, lists, cards, checklists)
    print_footer(fhtm)
    fhtm.close()

Then to get this to format looking a bit like a Trello board, we just need a bit of simple CSS for the list and card classes used above:

.list {
  margin: 0 3px 1em 3px;
  padding: 0 4px 4px 4px;
  background-color: #eee;
  border: 1px #aaa solid;
  border-radius: 4px;
  float: left; 
  width: 300px;
}

.card {
  margin-bottom: 4px;
  padding: 0 0.6em 0 0.6em;
  border: 1px #aaa solid;
  border-radius: 4px;
}

I’ve also modified the various headings styles to make things look nice, but I’ll leave that as an exercise for the student!

Python makes writing this kind of stuff so straightforward. It took me far longer to work out the JSON format and API of the Trello stuff than to write the code.

If I was writing this today, I might think about using an library like Beautiful Soup, or, given how straightforward it was, I might just do it this way again.

Saturday, 1 March 2014

what a difference a month makes

Labels: astronomy, solar power, statistics, weather

The end of another month, and so we have solar stats for February now. In the chart below, time runs along the x-axis, number of kW being produced up the y-axis. The black line is the energy generation on that day. The orange background shows the minimum, quartile, median, and maximum values, averaged over the month.

These February stats compare interestingly with the stats for January:

These carts are plotted with the same horizontal and vertical scales. The February averages are very noticeably wider (longer days) and higher (sunnier days) than January. And we aren’t even yet at the point where the day length is changing fastest – at the equinox, in March.

A different view shows all the days on a single plot. Time during the day runs along the x-axis; days run up the y-axis; the colour represents the amount of kW being generated at that time, on that day.

This clearly shows how the days getting longer, too. I look forward to seeing what March brings us.

All the up-to-date charts can be found on my other web site.

Sunday, 23 February 2014

sequestering carbon, several books at a time XIX

Labels: books

Hardly enough to be worth mentioning, but here is the latest lot:

Tuesday, 18 February 2014

a surfeit of packaging

Labels: electricity

My phone started behaving strangely over the weekend: switching off for no reason, then claiming the battery was dead, but when I plugged it in to the charger, claiming the battery was half, then fully, charged. After some searching on the web, I discovered this probably meant the battery was pining for the fjords. Maybe not that surprising: it is 2 and a half years old, and has been maxed out on certain games at times.

More hunting on the web, and I found a place selling genuine Samsung batteries (as opposed to much cheaper batteries that the reviews seem to think are rip-offs) at eye-watering prices (I shouldn’t be surprised at the price, I suppose; I recently bought a new battery for my watch, and previously a new strap: the combined price was more than that of the original watch). I ordered one; it arrived promptly today; the phone is now fine. I have my life back.

Phone batteries are small and compact, so I was expecting a small parcel. I received the battery in a plastic wrapper, in a jiffy bag, in a plastic envelope:

a lot to recycle…

Saturday, 15 February 2014

the chaos game

Labels: algorithm, fractals, graphics

By iteratively applying specific sets of transformations to the unit square, we can generate fractals. For example, the famous Sierpinski triangle can be generated from three simple transformations:

Three transformations (shown in black, red, green), shown by their action on the unit square (grey), produce the Sierpinkski triangle

Rather than iteratively transforming shapes (the base Iterated Function System approach), these plots are produced using the “chaos game” algorithm: start with a random point, and plot its position as it is successively transformed by a random choice of transformation:

x := rnd
repeat
   choose transform w_i with probability p_i
   x := w_i(x)
   plot x

The resulting plot converges to the relevant fractal.

Different sets of transformations give different fractals.

Here, the black and green transformations are as before, but the red transformation includes a rotation

One nice feature of this approach is that smoothly changing the transformations smoothly changes the fractal.

Of course, there can be more than three transformations:

A pentagonal fractal produced by five transformations

The transformations don’t all have to have the same scale factors:

A “snowflake” produced from four transformations, one with a smaller "shrinkage" than the others

This is where the probabilities in the algorithm come into play. When all the transformations have the same scale factor, we can choose transformations with a uniform random distribution. When different scale factors are used together, a good heuristic is to choose transformations with probabilities weighted by the area of the relevant transformed square (given by the determinant of the transformation matrix).

The transformations don’t actually have to be squares: we can have different scalings in different directions, and shears, giving more complicated looking fractals:

A “coral tree”

And then we can smoothly transform between quite dissimilar fractals, by linear interpolation between their transformation sets:

From pentagon to coral, and back.

Probably the most famous fractal of this form is the “Barnsley fern”.

What is marvelous for playing about with these systems is that the plots can be produced with very little code: the code to define the transforms, plot the points, and construct the animated gifs, outweighs the core of the chaos game algorithm itself.