Monday 31 March 2014

no more semicolons

Having been fiddling around with HTML, CSS and JavaScript recently, I understand a little of what this marvelous rant is about.
Such an initial semicolon is called a “defensive semicolon.”
That is the saddest programming concept that I’ve ever
heard, and I am fluent in C++
But saddest of all is the statement in Mickens’ intro: “This is my last column!”


For all my social networking posts, see my Google+ page

Sunday 30 March 2014

The labourer is worthy of his reward

Unpaid internships privilege those who don’t need the money.



For all my social networking posts, see my Google+ page

Wednesday 26 March 2014

t distribution

… and do read the hovertext




For all my social networking posts, see my Google+ page

Saturday 22 March 2014

book review: Think Complexity

Allen B. Downey.
Think Complexity.
O'Reilly. 2012

The problem with being an autodidact is the unknown unknowns: if you are teaching yourself something, how can you fill the gaps in your knowledge that you don’t even know are there? I am teaching myself Python. Not from scratch, because I can already program in other languages. But that’s part of the problem: because I know how to program, I am learning Python from the on-line documentation (which could be better) and Stack Overflow (which in invaluable). This means I can find the constructs I look for; but what about the ones I don’t know exist?

So I’ve been thinking about getting a book, to help fill the gaps. I came across Think Complexity, a slim book (130pp) that claims to be targeted at an intermediate level, with the bonus of using examples from Complexity Science, a subject I also study.

It starts off well, with a mention of Python generators (which I had come across as a concept) and their “yield” statement (which was new to me). Yet the discussion is very brief: less than a page. I wanted to know more, as it sounded interesting, and made me wonder if the approach would allow coroutines (am I showing my age?). So I googled, and found David Beazley’s excellent tutorials, one on Python generators, using them in a functional manner to implement processing pipelines, and one, indeed, on coroutines. There is a lot more than even hinted at in Think Python.

Next comes a chapter on algorithmic complexity and “big O” notation that introduces Python list comprehensions. Now, it’s virtually impossible to have visited Stack Overflow more than a few times without having come across list comprehensions: they are marvellous beasts. However, their introduction here crystallised my apprehension with the book: they are explained with just a few examples only. Examples can be great for showing what is possible, and the examples here are good in that they start trivial and get more complicated. But you also need a description of the underlying syntax, so that you know that you have inferred the structure correctly from the examples, and to cover usages not illustrated by the examples.

The chapter on Cellular Automata uses NumPy arrays, but doesn’t talk about them much. NumPy is excellent for doing anything with arrays, and if you have come to Python via Matlab, like I have, you will feel right at home with them. One interesting snippet made here is an efficient way to implement Conway’s Game of Life using convolution from SciPy (although Bill Gosper’s HashLife, underlying Golly, is faster, and more interesting algorithmically). 

Then there are brief chapters on fractals, self-organised criticality, and agent-based models. But not a lot more Python. The book finishes up with several case studies prepared by students following up some of the concepts in the book; these are probably the most interesting parts. However, they are interesting mainly from a complexity viewpoint, not really from a Python viewpoint.

In summary, although this is advertised as “intermediate level” Python, it doesn’t go very far beyond what you can pick up readily from Stack Overflow. However, the idea of teaching a programming course using fun examples from Complexity Science is a good one: so many texts use relatively boring examples with little motivation. It is clear here from the chapters contributed by the students that they really engaged with the material.

For all my book reviews, see my main website.


Friday 21 March 2014

Not a sound from the pavement

Midnight is the only time to return False in Python.  Is that desired behaviour?


For all my social networking posts, see my Google+ page

Wednesday 19 March 2014

the past is a foreign country, with foreign food

Don’t you long for some of that wholesome food people used to eat, rather than today’s prepackaged awfulness?

Well, here's some of that food from the 50s (I’m sure it’s not just the colour balance, or “soft” focus making it look totally unappetising), accompanied by delicious commentary.

The recipe says “toss lightly,” but I suppose that depends on how much you eat and how bad the cramps get.


(Via Butterflies and Wheels)

For all my social networking posts, see my Google+ page

Sunday 16 March 2014

sunset

It almost looks like the clouds are streaming into the sun!

18:37 GMT, looking west

Friday 14 March 2014

pie no more

Support the “one less pie” initiative!



For all my social networking posts, see my Google+ page

Thursday 13 March 2014

selling out students

Ucas sells access to student data for phone and drinks firms’ marketing


Utterly outrageous!
Hint: when your excuse is “it isn’t illegal”, you've already lost the argument.

(via BoingBoing)

For all my social networking posts, see my Google+ page

Wednesday 12 March 2014

favourite page of a freezer manual



“If there’s a cat in your freezer drinking wine, eating Ryvita, with a fly ironing, it’s fine”
(via New Scientist Feedback)

For all my social networking posts, see my Google+ page

Tuesday 11 March 2014

pronouncing Noether

Here is a very nice explanation of Noether’s theorem.


I had heard of Emmy Noether before, because we covered Noether’s theorem in my Physics degree.  But I hadn’t realised about all the other mathematical contributions she had made. (Additionally, I’ve recently learned how to pronounce “Noether”.  In my head it will probably always be “Noy-ther”, as the lecturer pronounced it all those years ago, but now I say “Nur-tuh”)


For all my social networking posts, see my Google+ page

Monday 10 March 2014

Brits in space

Sue Black points out: Tim Peake.  First Brit in space.  Not.


Props to Tim Peake for pointing out their mistake, but no points at all to the Grauniad for propagating the error in their headline, given that he had corrected them! 


UPDATE: The Guardian has since modified its headline to First British man in space.  They are still wrong, though.  (Thanks to the commentors on Sue's blog for these three names.)


For all my social networking posts, see my Google+ page

Sunday 9 March 2014

book review: Tempo

Venkatesh Rao.
Tempo: timing, tactics and strategy in narrative-driven decision-making.
Lightning Source. 2011

Life is a verb, not a noun.
Charlotte Perkins Gilman
Despite Gilman’s aphorism, people seem much more able to think in terms of things rather than actions, stuff rather than change, objects rather than processes. This may be because, as Mark Bickhard points out, we have a substance metaphysics, but no acceptable process metaphysics. Rao’s slim book (150 pages in total) goes some of the way to providing the latter for our social environment.

Rao starts from the idea of process, of a sequence of events, and goes from there, to bring in the concept of time. His time is not merely an ordering of events, but is a measure of the rate, of the tempo, of those events. Understanding the tempo of the various processes, and how these interact, and interweave, is the basis of his discussion of decision making, from individual cooking to multi-person businesses.

Understanding the dynamics of the decision-making process exposes various phenomena that can be part of the process. Rao identifies fait accompli, brinkmanship, procrastination, second-guessing, passive aggression, time-outs, defaulting, and building mindshare, among others.

We constantly need to make decisions, and those decisions need to be situated in the current context. Context switching includes gaining situational awareness (grokking where things are and what they are doing, now and in the near future) in the new context. Situational awareness crucially includes awareness of the situation’s tempo.
p115. The major claim in this book is that tempo can be used to modulate enactments that move along too quickly to manage one micro decision at a time.
Once you have an awareness of the tempo, you can modulate it to improve your performance, either in competition, or cooperation, with others in the environment.
p118. The primary effect of operating at a faster tempo is that you can maneuver inside the decision cycle of your opponent, disrupting his or her enactment …
    This style is obviously effective in adversarial settings, but it is equally effective in cooperative settings …. The alert waiters at upscale restaurants, fulfilling your needs before you even recognize them, are a good example.
The situations we encounter are not arbitrary. They may have a greater or lesser degree of legibility to us, and we modulate our environment to control situations and their legibility. Rao explains how we do this. We comprehend the world through mental models, and we organise the world to externalise these models. This organisation, this modulation of our environment helps us improve our efficiency, our ability to perform.
p127. To externalize a mental model is to arrange our physical environment to serve our intentions more efficiently. In cognitive psychology, this idea is called distributed cognition.
The full process of organisation is a combination of objective and deliberative design and of participatory dance dependent on situational awareness and tempo; how much of each component we use depends on circumstance and preference.

Different styles of decision making follow different patterns; Rao identifies reactive, deliberative, opportunistic, and procedural. For example, a procedural, or bureaucratic, style has the environment organised by another, so that you can follow a codified process without necessarily understanding it, yet still achieve the desired outcome.
Civilisation advances by extending the number of important operations which we can perform without thinking about them.
Alfred North Whitehead
Following such an opaque procedure can be very efficient, because you do not need to think so deeply, but if the context deviates from that assumed by the organiser, you cannot then readily modify what you do not understand.

On the other hand, our personal, efficient organisation of our environment has meaning to us. This meaning may be legible to others, or it may not. If we wish to organise in concert with others, we need to be legible to them, but this might impact our own efficiency. Bosses who demand tidy desks are prioritising their own need for legibility, for control, over their staff’s efficient meaningful organisation.

Central planners can have the same desire for control, but this can kill the very thing they are trying to create.
p133. … central planners begin by poorly internalizing a rich, emergent collectively externalized mental model (such as the structure of an old city). They proceed inevitably to failure when they attempt to replace it with the externalization of their own impoverished mental models, constructed largely out of primitive platonic ideals, a case of design destroying dance.
Nouns and verbs tend to focus on single specific things. Rao lifts these concepts into the environment, as fields and flows.
p138. Fields are arrangements of the physical environment. Flows are the behaviors that result when humans interact with a field. Flows are the observable parts of others’ enactments.
Rao is not concerned with a general model of all physical reality here, but with an approach to situated decision making, of people in social environments.
p139. For a given individual, the environmental field … combines with the other group members in the field at the same time, to create a social field. When there is little momentum or meaning in the primary environmental field, the dynamics of the social field start to dominate. An environmental field that is deliberately designed to catalyze complex social behaviors generally contains one or more social objects…. Chairs and tables are among the simplest social objects.
Unlike the models and processes imposed by central planners and rigid bureaucrats, Rao advocates a balance between the aspects of design and dance.
p141. Artistic approaches to field-flow complexes exploit the natural chemistry among mental models in richer ways. The key to such approaches is a tasteful blending of preparatory design and participatory dance. Successful field-flow complexes can neither be designed nor danced into existence. They must be orchestrated. A conductor or choreographer, rather than a pure designer or dancer, is necessary.
The metaphor of conductor or choreographer, rather than the extremes of rigid pre-programmed design, or of completely fluid and potentially chaotic dance, is interesting. Denis Nobel uses a similar metaphor to explain biology. It seems to be a metaphor well-suited to a variety of complex systems.

Complex systems comprise a collection of agents interacting in and with a complex environment. For a single agent, its environment comprises not only the organised spaces, but the other agents, too. In Rao’s model, the agents are people making their context-dependent situated decisions.
p143. Groups cannot be realistically studied unless you take into account the effects of the field-flow complexes that they inhabit. This is a natural consequence of narrative rationality: you can only understand a group with reference to the specific narrative within which it is situated.
That is, the same group of people in a different environment, or even in the same physical environment in a different context, will behave differently. This should not be at all surprising, but the environment and context are often neglected in complex systems analysis in order to simplify the problem; just think of in vitro versus in vivo experiments, or many experiments performed in psychology labs.

Rao demonstrates how a consideration of tempo, situation and context, and of patterns of decision making, can illuminate many more patterns than the standard procedural bureaucratic one. Moreover, these other patterns and metaphors have distinctly different flavours, especially when applied to changing an organisation.
p144. Where change in the machine metaphor is a process of stepwise re-engineering, in the other, more organic metaphors, change is a process of generative growth, ontogeny and self-organization.
This book nicely complements a lot of books trying to apply complexity theory to organisations. Here Rao focuses on complex human systems, involved in descion making, aspects often missing from the more theoretical models of complex systems exploited by the other approaches.

This deserves rereading. Some of the earlier discussions, which can have a slightly hippy-Zen flavour (despite many examples being from military command and control) take on more meaning in light of the later application. Recommended.




For all my book reviews, see my main website.


Friday 7 March 2014

spring

It’s hard to believe that this time last year was freezing cold, and snow.  This year is a little different.

catkins in sun and shade
luminous crocuses
It’s not just the vegetation.  The frogs clearly think it’s spring, too.


Thursday 6 March 2014

word of the day

elsewhere, elsewhen, and now…

elseweb: this is a useful word, found (invented?)  in the comments on Scalzi’s blog


For all my social networking posts, see my Google+ page

space is really big

Space is big when looked at linearly.  It’s still big even when looked at logarithmically.  This page is a brilliant “powers of ten” display that goes all the way from the Planck length to the Universe.  (There's not a lot around the Planck length.)

It’s amazing what things are about the same size…

the slider goes from 10^-35 to 10^27


For all my social networking posts, see my Google+ page

space is big

Space is big.  Really big.  This tediously accurate map of the solar system gives a “feel” just for our solar system, minuscule compared to the galaxy, itself minuscule compared to the universe.




For all my social networking posts, see my Google+ page

insane detail

Another amazing zoom, here of a detailed fractal landscape.  It not only has insane detail, but it is dynamically changing!  Some people clearly have way too much computing power on their hands…

   

 (via BoingBoing)

For all my social networking posts, see my Google+ page

Tuesday 4 March 2014

10 Misconceptions about Creativity

Is creativity all it’s cracked up to be?

Sabine Hossenfelder lists 10 Misconceptions about Creativity.


For all my social networking posts, see my Google+ page

Sunday 2 March 2014

my Trello backup code

Back in August I blogged about Trello, a great browser-based list manager.  I’ve been using it constantly ever since. I use Evernote to manage the past, and Trello to manage the future.

In that Trello blog post, I mentioned I written a bit of Python to backup my boards. Yesterday PK asked in a comment for that code.  I thought briefly of putting it up on something like GitHub, but the code is so short, I decided I’d give it here instead.

I also thought about refactoring the code a bit first.  I wrote this a while ago, and I know a bit more Python now.  But it works.  So I’ll leave any refactoring until I have to make some changes.  (Well, okay, maybe I made a few changes to remove the worst uglinesses…)

First, there are some libraries to import.
import json
import unicodedata
import urllib2
import datetime
I strip the unicode out of the Trello data, for simplicity.  (One day I will spend the time to really understand all the intricacies of unicode in HTML, but for now, out it goes.)  As usual, Stack Overflow was of invaluable help in finding a way to do this.
def stripUnicode(ustr):
    # translate unicode (accents) to non-accented chars
    ustr = unicodedata.normalize('NFKD', unicode(ustr))
    ustr = u"".join([c for c in ustr if not unicodedata.combining(c)])
    
    # translate whole string to ascii
    return ustr.encode('ascii', 'ignore')
The overall structure of the code is simple: loop over the boards; for each board, print an HTML header, print the lists (which involves printing their cards), then print an HTML footer.

The HTML header involves a title, importing the CSS style file, and a bit of stuff to control the width (a fudge, frankly, but this is for me, not general purpose code!).
def print_header(fhtm, board, lists):
    print >> fhtm, '<html>'
    print >> fhtm, '<head>'
    print >> fhtm, '  <title>' + board['name'] + '</title>'
    w = len([lst for lst in lists if not lst['closed']])*318
    print >> fhtm, '  <style type="text/css">'
    print >> fhtm, '    @import url(trello.css);'
    print >> fhtm, '    body {min-width: ' + str(w) + 'px;}'
    print >> fhtm, '  </style>'
    print >> fhtm, '</head>'
    print >> fhtm, '<body>'
Printing the lists involves looping over the lists, and for each open list printing a heading, and printing the cards in the list.
def print_lists(fhtm, lists, cards, checklists):
    for trello_list in [lst for lst in lists if not lst['closed']]:
        print >> fhtm, '<div class="list">'
        print >> fhtm, '<H2>' + trello_list['name'] + '</H2>'
        print_cards(fhtm, cards, checklists, trello_list['id']) 
        print >> fhtm, '</div>'
Printing the cards involves looping over the list of cards, and for each card, printing its due data, description, and checklists (including strike-through for completed items).  There is potentially more information in a card, but that’s all I currently use.  If you use more, you will have to update this bit, but the structure of what’s needed is fairly obvious.
def print_cards(fhtm, cards, checklists, id_list):
    for trello_card in [c for c in cards if c['idList'] == id_list]:
        print >> fhtm, '<div class="card" >'
        print >> fhtm, '<H3>' + stripUnicode(trello_card['name']) + '</H3>'
        print_card_due(fhtm, trello_card)    
        print_card_desc(fhtm, trello_card)    
        for id in trello_card['idChecklists']:
            print_checklist(fhtm, checklists, id)    
        print >> fhtm, '</div>'
    
def print_card_desc(fhtm, card):
    if card['desc'] != "":
        print >> fhtm, '<P>' + stripUnicode(card['desc'])
        
def print_card_due(fhtm, card):
    if card['due']:
        print >> fhtm, '<UL><LI type="square"><I>due</I> ' + card['due'][:10]
                     + '</UL>'
        
def print_checklist(fhtm, checklists, id):
    checklist = [c for c in checklists if c['id'] == id][0]
    print >> fhtm, '<H4>' + checklist['name'] + '</H4>'
    print >> fhtm, '<UL>'
    items = checklist['checkItems']
    items.sort(key=lambda x: x['pos'])   # pos = screen posn, for list order
    for item in items:
        print_checklist_item(fhtm, item)
    print >> fhtm, '</UL>'
    
def print_checklist_item(fhtm, item):
    if item['state'] == "complete":
        print >> fhtm, '<LI><I><DEL>' + stripUnicode(item['name'])
                     + '</DEL></I>'
    else:
        print >> fhtm, '<LI type="circle">' + stripUnicode(item['name'])
The HTML footer is trivial.
def print_footer(fhtm):
    print >> fhtm, '</body>'
    print >> fhtm, '</html>'
There are some secret keys and tokens needed, which you get when you register at Trello, to get access to your own boards.
key = 'a 32 hexdigit string'
token = 'a 64 hexdigit string'
I have hardwired in the various boards I want to backup. The 24 hexdigit string can be found right at the beginning of the board’s JSON file, so you have to download that manually, and copy the string from it. But you only have to do that once.
board_dict = {
    'board1': 'a 24 hexdigit string',
    'board2': 'a 24 hexdigit string',
    ...,
    'boardn': 'a 24 hexdigit string'
}
Then, finally, there’s the loop over the boards in this dictionary: reading in the board data, which requires reading the board, its lists, its cards, and its checklists separately, then printing out the relevant HTML, including the date of the backup:
for filename, board_id in board_dict.iteritems():
    print filename
    
    trello = 'https://api.trello.com/1/boards/' + board_id
    key_token = key + '&token=' + token

    get_url = urllib2.urlopen(trello + '?key=' + key_token)
    board = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/lists?key=' + key_token)
    lists = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/cards?key=' + key_token)
    cards = json.loads(get_url.read())
    get_url = urllib2.urlopen(trello + '/checklists?key=' + key_token)
    checklists = json.loads(get_url.read())

    fhtm = open(filename + '.htm', 'w')
    print_header(fhtm, board, lists)
    print >> fhtm, '<H1>' + board['name'] + ' (' + str(datetime.date.today())
                 + ')</H1>'
    print_lists(fhtm, lists, cards, checklists)
    print_footer(fhtm)
    fhtm.close()
Then to get this to format looking a bit like a Trello board, we just need a bit of simple CSS for the list and card classes used above:
.list {
  margin: 0 3px 1em 3px;
  padding: 0 4px 4px 4px;
  background-color: #eee;
  border: 1px #aaa solid;
  border-radius: 4px;
  float: left; 
  width: 300px;
}

.card {
  margin-bottom: 4px;
  padding: 0 0.6em 0 0.6em;
  border: 1px #aaa solid;
  border-radius: 4px;
}
I’ve also modified the various headings styles to make things look nice, but I’ll leave that as an exercise for the student!

Python makes writing this kind of stuff so straightforward.  It took me far longer to work out the JSON format and API of the Trello stuff than to write the code.

If I was writing this today, I might think about using a library like Beautiful Soup, or, given how straightforward it was, I might just do it this way again.

Saturday 1 March 2014

what a difference a month makes

The end of another month, and so we have solar stats for February now.  In the chart below, time runs along the x-axis, number of kW being produced up the y-axis.  The black line is the energy generation on that day.  The orange background shows the minimum, quartile, median, and maximum values, averaged over the month.


These February stats compare interestingly with the stats for January:


These charts are plotted with the same horizontal and vertical scales.  The February averages are very noticeably wider (longer days) and higher (sunnier days) than January.  And we aren’t even yet at the point where the day length is changing fastest – at the equinox, in March.

A different view shows all the days on a single plot.  Time during the day runs along the x-axis; days run up the y-axis; the colour represents the amount of kW being generated at that time, on that day.


This clearly shows how the days getting longer, too.  I look forward to seeing what March brings us.

All the up-to-date charts can be found on my other web site.