I then decided to use Lisp-like S-expressions; at least a simplified version thereof. The data I read from the file is basically a list which can contain other lists, strings (which technically are also just lists), and atoms. A while ago I wrote a simple Erlang module to process something similar, but that had made certain assumptions that didn’t hold anymore, and I felt something more maintainable was required. And what better way to do that than by using a formal grammar to describe the file format and a tool to generate a parser from that?

A simple and straight forward grammar formalism is PEG, Parsing Expression Grammar, and there is already an Erlang parser available for it, Neotoma by Sean Cribbs. Installation was easy, and so was writing a grammar:

list <- open elem* close;
elem <- list / atom / sstring / dstring;
atom <- [a-z0-9_]+ space*;
dstring <- '"' [^"]* '"' space*;
sstring <- "'" [^']* "'" space*;
open <- '(' space* ;
close <- ')' space* ;
space <- ' ' / '\t' / eol;
eol <- '\r\n' / '\n' / '\r';

A list is something (or nothing) enclosed in quotes (with optional spaces). An element is a choice of things, atoms are lower case letters and digits (at least one), and with strings I allow both double and single quotes. This grammar is saved in a file “terms.peg”:
Eshell V5.7.3 (abort with ^G)
1> neotoma:file("terms.peg").
ok
2> c(terms).
{ok,terms}

and you’re ready to go. I created four short one-line test files, with the following content:

1. (atom)
2. ( “string” )
3. (foo bar)
4. (())

This is the output:
3> terms:file("test1").
[["(",[]],[["atom",[]]],[")",["\n"]]]
4> terms:file("test2").
[["(",[" "]],[["\"","string","\"",[" "]]],[")",["\n"]]]
5> terms:file("test3").
[["(",[]],[["foo",[" "]],["bar",[]]],[")",["\n"]]]
6> terms:file("test4").
[["(",[]],[[["(",[]],[],[")",[]]]],[")",["\n"]]]
Not all that helpful, as there is a lot of noise in there, such as the spaces in “test2″, and all the line-breaks. So I need to go back to the AST and extract just those bits from the parse tree that I actually want. In Neotoma you can do this by adding bits of Erlang code to the grammar definition, like so:
list <- open elem* close
`[Open, Elem, Close] = Node, Elem`
;
atom <- [a-z0-9_]+ space*
`[Atom, Space] = Node, list_to_atom(Atom)`
;
dstring <- '"' [^"]* '"' space*
`[Quote, Str, Quote, Space] = Node, Str`
;
sstring <- "'" [^']* "'" space*
`[Quote, Str, Quote, Space] = Node, Str`
;

(All other lines are unchanged as in the grammar listed above)

What I do here is to split the Node into its component parts, and then discard the bits I don’t want. In the ‘list’ rule I am only interested in the elements, but not in the enclosing brackets, so I just return ‘Elem’. For the ‘atom’ I ignore the spaces and convert the matched character sequence into an atom. Now the output looks like this:
7> neotoma:file("terms.peg").
ok
8> c(terms).
{ok,terms}
9> terms:file("test1").
[atom]
10> terms:file("test2").
["string"]
11> terms:file("test3").
[foo,bar]
12> terms:file("test4").
[[]]
Much better, and just what I wanted. The ‘terms.elr’ file that neotoma generated is 7kb in size, just over 220 lines, and just under 8kb compiled.

The only issue is speed and memory consumption: on my 8GB MacBook Pro a file of less than 40k runs out of memory and crashes after 30+ seconds. If I take a part off at the end to make it 35k, the parser succeeds, but needs 35 seconds (hand-timed). So I think I will have to revisit my hand-made parser again after all… :(

Yesterday we had Fish & Chips for dinner. The girls were comparing the sizes of their chips, and then we came on to the topic of the longest chip in the world. Obviously, with ‘proper’ chips this is limited by the size of the source potato. However, industrially produced chips are made of mashed potato formed into chip-shapes, so there is not really any fixed limit on the length. So, thinking of Hofstadter, I asked them how many potatoes we would need to make a chip that spans around the whole world.

Rough assumptions: one potato contains enough matter to produce 10cm worth of chip (the thickness is not specified). So, how many potatoes do we need for one metre of chip? This is also useful to practice basic primary-school-level maths… – 10. How many for a kilometre? 10,000. How many kilometres do we need to span the world? Roughly 40,000 km. So how many potatoes do we need? 400 million.

The next question is whether there are enough potatoes in the world to do this. Assuming a potato weighs 100g, how much do our 400 million potatoes weigh? 40 million kilogrammes, or 40,000 (metric) tons. What is the world’s potato production? According to Wikipedia, this is 315 million metric tons, so plenty enough. Now, if we were to turn the annual potato crop into one long chip, how many times would it go round the Earth? 7,875 times.

So, with a bit of basic maths (and Wikipedia for the data) you can make maths exciting for kids, practice how to multiply and divide, teach problem-solving, and have fun at the same time. And they also get a feeling for numbers: 400 million – that’s how many potatoes you need for a chip to span the Earth.

The term “global warming” is obviously problematic, for several reasons, two of which I will discuss here: firstly, the climate is a complex system, and secondly, the climate is not the weather. Both reasons have links to linguistics, which is my justification to talk about them on this blog.

Climate is a complex system

Chaos theory was discovered by meteorologists, running climate models on computers. Small changes in the starting conditions result in largely different outcomes. That is why nobody can really predict reliably what is going to happen, as there is no clearly visible link from A to B, the conditions today to the conditions at whatever time in the future. Reducing this to a simple statement such as “temperatures will increase globally” is dangerous, as climate is not that simple itself, and you then get people who demolish your argument on the grounds of inaccuracy.

Language can be seen as a complex system as well; it is influenced by so many factors that it is not possible to make any predictions about how language will change. Any statements such as those on the bad influence of text messaging on the English language are clearly not appropriate; broadly general statements of this kind miss the point about the varieties and different language communities that make up the “English” language.

The climate is not the weather

This point is somewhat related: ‘weather’ is what we’ve got now, but ‘climate’ is a broader, more general tendency. So while we might indeed have a cold winter, if we have a correspondingly hotter summer, the average annual temperature might indeed rise, even if it doesn’t feel like that as you shiver your way to work in the morning. And this year is a single event, which in context might be an outlier if it becomes really warm next winter. Weather is somewhat unpredictable and chaotic, otherwise the Met Office would be out of work.

Global climate would also mean that it could become colder in Western Europe, while other regions of the Earth heat up, and the people of Tuvalu will have have a different view about melting ice caps than American farmers in Arizona.

Michael Halliday compares langue and parole (or competence and performance) with climate and weather: while we can observe one (weather/parole/performance), the other can only be perceived indirectly (climate/langue/competence) through studying the former. But essentially they are different views on the same phenomenon, one short-term and one long-term.

A solution?

Coming to the point of this post, I would suggest abandoning the term “Global Warming” in favour of “Climate Change”. Change can go in different directions, and so it is harder for climate-change-deniers to win easy points whenever the weather is colder, and it also emphasises the climate as opposed to the weather. This might seem like a simplistic point similar to the political correctness debate, but lexical choices when representing reality in language are really important.

And thus we have moved from complex systems via Halliday’s view of langue and parole to Critical Discourse Analysis.

Most? I would say all… but apparently whatever system YouGov use for sentiment analysis cannot cope with idioms. And shooting yourself in the foot is not exactly a tricky one to identify I should think.

But this raises a more complex issue: there are many ways to express opinions, attitudes, judgements, etc in language. This is a much larger problem than counting the number of ‘positive’ and ‘negative’ words in a text. To begin with, words in isolation rarely have a meaning; opinions are usually subjective; and then there’s irony and sarcasm.

Yes, Clegg did really well when he supported the Tories on tuition fees…

Continuing on this theme, here’s another issue: the assumption that the scope of a sentiment is the whole text. Here’s an opinion (positive) from the same site about student protests:

I completely support the right to protest; however, violence is unreasonable.

This is somewhat positive, supporting the students, but in the second clause there is an additional judgment condemning the violent incidents that happened at the demonstration. This seems to suggest that the proper carrier of attitude should be the clause, rather than the sentence, let alone the text. Not everything is just black and white.

Merging the values is tricky, as it involves relative frequencies, so I needed to make some subtle changes. First, the term_freq/1 method now deals with absolute frequencies, and returns a tuple containing the frequency values and the document size in tokens, so that the relative frequencies can easily be computed if required:

term_freq(Text) ->
    term_freq(Text, 0, dict:new()).

term_freq([], Sum, Dict) ->
    {Dict, Sum};

term_freq([Token|Rest], Sum, Dict) ->
    term_freq(Rest, Sum+1,
           dict:update_counter(Token,1,Dict)).

No change really, only the terminating clause of term_freq/3 has dropped its dict:map to compute the relative values, and instead returns the tuple with the frequency dictionary and the document size.

This also requires a minor change in the inv_doc_freq/3 function, where we need to deal with the tuple and extract the dictionary from it in the second and final clause:

old

inv_doc_freq([Doc|Rest], DocNum, Dict) ->

new

inv_doc_freq([{Doc, _Sum}|Rest], DocNum, Dict) ->

The biggest change, however, is in the combined tf_idf/1 function, as the algorithm has somewhat changed. Originally the function was a full screen in the editor, but I have extracted two functions to make them easier to follow; the gain in clarity will surely outweigh the minute performance penalty…

tf_idf(Docs) ->
    Idfs = inv_doc_freq(Docs),
    DocLen = total_doc_size(Docs),
    DocTotalFreqs = total_token_freqs(Docs),
    dict:map(
        fun(Key, Value) ->
            dict:fetch(Key, Idfs) * Value / DocLen
            end,
        DocTotalFreqs).

I need to calculate the overall size (in tokens) of the full document collection, and then add up the token frequency over all documents. These have been factored out into separate functions. Then all is left is a map over all tokens to calculate the tf-idf value from the relative frequency in the document collection multiplied by the idf value as calculated earlier.

Computing the total document size is trivial: we loop over the list of term frequency dictionaries and this time extract the lengths, ignoring the actual dictionaries:

total_doc_size(Docs) ->
    lists:foldl(
        fun({_Doc, DocSum}, Total) -> Total + DocSum end,
        0,
        Docs).

And finally, that leaves computing the total frequencies of all tokens.

total_token_freqs(Docs) ->
    lists:foldl(
        fun({Doc, _Sum}, Current) ->
            dict:fold(
                fun(Key, Value, AccIn) ->
                    dict:update_counter(Key,Value,AccIn)
                    end,
                Current,
                Doc)
            end,
        dict:new(),
        Docs).

Here we process the document list (as there are likely to be fewer documents than tokens) and fold each dictionary, adding the tokens with their respective frequencies to our accumulator dictionary.

Apologies for this correction; but sometimes you only really realise that a particular interpretation of an algorithm is not the right one when you actually need to use it. The curse of developing libraries without proper specification of the requirements…

What I actually wanted is tf-idf, the product of term frequency (tf) and inverted document frequency (idf); essentially you see how often a word/term/token occurs in a text, and multiply that with a measure of how ‘bursty’ it is. The idea being that common words (the, of, and etc) occur in pretty much every document and are thus useless for categorisation of the content. In a way it is a more sophisticated approach to using a stop word list. Sophisticated because you don’t have to create such a list, and it is also not binary include/exclude, but assigns each token a continuous weight depending on its distribution.

Term Frequency

This is simply the relative frequency of occurrence, the number of times a token occurs in the text divided by the text length. As input I assume that the text has already been tokenised and is represented as a list of tokens. The output should be a dictionary (ie a set of key/value tuples) with each token as a key and its tf as the value:

term_freq(Text) ->
    term_freq(Text, 0, dict:new()).

term_freq([], Sum, Dict) ->
    dict:map(
        fun(_Key, Value) -> Value / Sum end,
        Dict);

term_freq([Token|Rest], Sum, Dict) ->
    term_freq(Rest, Sum+1,
        dict:update_counter(Token,1,Dict)).

In case another token is available, I simply update its frequency by one, add one to the text size, and re-run the function on the rest of the text. If no more tokens are left, then I map the dictionary (which at this point contains absolute frequencies) to another dictionary by way of dividing each value by the text size; this new dictionary is then returned.

Inverted Document Frequency

For the idf I count how many documents each token occurs in, and divide the total number of documents by that number; so the rarer the token, the larger the resulting value. The token the should just give a result of 1.0; however, to make it a bit more complicated we then take the logarithm (base-10) of it, so that the final value will be greater than or equal to zero.

This time the input is a list of dictionaries, one for each document. The dictionary representing each document is the output of our term_freq/1 function, ie the keys are the tokens, and the values the term frequencies. We don’t really care about the frequencies here, as they all will be greater than zero – a word that does not occur in a text will not be a key in the respective dictionary. As output we will have a single dictionary of all tokens that occur in our document collection, with the values being the idf of each token.

inv_doc_freq(Docs) ->
    inv_doc_freq(Docs, 0, dict:new()).

inv_doc_freq([], DocNum, Dict) ->
    dict:map(
        fun(_Key, Value) -> math:log10(DocNum/Value) end,
        Dict);

inv_doc_freq([Doc|Rest], DocNum, Dict) ->
    inv_doc_freq(Rest, DocNum+1,
        dict:fold(
            fun(Key, _Value, AccIn) ->
               dict:update_counter(Key,1,AccIn) end,
            Dict,
            Doc)
    ).

Again we iterate over all elements of our input list (ie the documents), and this time we iterate over all tokens of the document using a dict:fold/3 function, by adding 1 to the count for each token of the current document that we have already encountered, or entering it with a frequency of 1 if we haven’t yet. We also increment the document count by 1. This time the dict:map/2 function performs the calculation for the idf value as soon as we have reached the end of our document list.

tf-idf

At this stage we have a dictionary for each document containing the term frequencies, and a dictionary for the whole document collection containing the inverted document frequencies for all the tokens. Combining the two we then get the value for the tf-idf, which is different for each document (so the output is a list of dictionaries, one per document).

To make things easier, the call to compute the idf is integrated into the tf_idf/1 function, so the input is the same as for the inv_doc_freq/1 function, a list of term frequency dictionaries:

tf_idf(Docs) ->
    Idfs = inv_doc_freq(Docs),
    lists:map(
        fun(TFs) -> dict:map(
            fun(Key,Value) -> Value *
                dict:fetch(Key, Idfs) end,
            TFs) end,
        Docs).

Here we map the list of term frequency dictionaries (Docs) to a list of dictionaries containing the tf-idf values. For this mapping we map each (document) term frequency dictionary to the respective (document) tf-idf dictionary by multiplying each token’s term frequency by its idf value as computed by inv_doc_freq/1.

Summary

Calculating a set of values from texts is very concise with Erlang. In languages like C or Java one would have to code various (nested) loops, but this can easily be accomplished by using the map and fold functions that operate on lists and dictionaries in Erlang. It does need a bit of mental acrobatics, but if you are familiar with Prolog, then the basic structure of an Erlang program is not too difficult to follow. It’s those nested mappings that sometimes can be a little confusing.

The beauty of Erlang, of course, is that each map can be done in parallel; if you have a large list of documents and a processor with several cores then it is not hard to make use of its full power by simply using a parallel map function. To do this in other languages where nested loops are used in place of the map function is not trivial.

So Erlang is not only very concise, but it can also be future-proof by allowing easy concurrency.

First, it did take me a bit to get used to Objective-C. Not because it is very different from Java (I used to program in C after all before Java came along), but because all the classes in the Cocoa framework need to be learned. There are subtle differences between those and their Java cousins, and after a bit more experience I believe that the Cocoa classes are actually more powerful and easier to use than their Java counterparts.

Some teething troubles, lack of automatic memory management on the iPhone, and a surfeit of squa brackets meant further delays. Finally I had a program written, but it needed more work on the graphics side, artwork and so on. The stuff that really makes a difference, but is very time-consuming and hard if you’re not used to using graphics software. So the easier way out was to write a different program, which is lighter on the artwork.

This then was a todo-list program, which is also suitable for planning small projects. I wanted a program like that, but didn’t want to fork out the money for Things, which also looked a bit like overkill. On the life hack blog I read an article by Dustin Wax on his moleskine setup, and that seemed like something usable, which I then went about implementing as an iPhone app. With a bit of help from a friend with the icon design, and thanks to freely available sound files and icons, ePlanner was born.

In ePlanner I tried out Core Data, which is really a lot easier than messing about with SQLite directly. It uses both tabs and navigation views, and a lot of tables. I found it rather tedious in that all the classes were almost identical, but only almost, not 100%, and it’s hard to see how that could be changed. The behaviour of those classes is ever so slightly different.

The submission procedure was very easy, thanks to a description I found on the web. My app did get rejected, due to a crash on a 3GS; but I don’t have a 3GS, so I could only test it on a 3G and an iPod touch. Thanks to Instruments I could track down the error, which was of course a memory management issue, but one without consequences on the machines I could test it on. After that was changed, the app went through, and has indeed been bought by people all over the world.

It is really a nice feeling to think that someone in Argentina is using my app, as is someone in Hong Kong, some people in the US, Sweden, etc. I used some free Google advertising at the beginning, but that is really expensive, though when I stopped it, sales began to trail off. But that could also have been an effect of it slipping out of the ‘newly released’ slots.

It is indeed not too hard coming up with a program that does sell. The overall process is not too hard, though there were some frustrating moments battling with the various code signing and certificate issues that Apple requires.

I since have bought an iPad, and am thinking of porting ePlanner to this; however, I’ll give it a while so that I get used to how the iPad works. Knowing your way round the platform makes it a lot easier to develop good software, and I am not yet sure how the UI design for the small iPhone screen can best be translated to the iPad’s larger display. But it will come, and I will describe the process on this blog…!

In the meantime, I will re-visit some of my previous program ideas, as it is really not hard to turn them into something that will end up in the App Store, and it is really satisfying to do so.

I suspect that in our house the iPad will be a multi-user gadget.

I have an iPhone, my wife has got an iPod touch, and the kids currently use a clapped-out ancient Sony Vaio laptop whose disk is about to fail. Everybody has their own device to do things on. However, this is likely going to change when we acquire an iPad. I envisage this as being a general device that just lies on the sitting room table, to be picked up by whoever wants to use it for reading their email, checking something on Wikipedia, playing a quick game, adding an entry to a calendar, looking up a phone number, and so on.

When I check mail on my iPhone, it is set up to look at my email accounts. Similarly, the calendar is sync’d with my general calendar, and the address book is too. I don’t (and cannot) easily switch between identities (though it is possible to do so with mail and calendar). Some games that I play on my iPhone store their state in case of interruption, and I can resume them later. The same applies to other applications, which usually have one set of data they work with.

This is OK for a phone. Typically you have a phone, and you’re the only one using it, otherwise it’d lose some of its usefulness if you don’t know who you will reach when calling a particular number. But if the iPad is shared between people, how can I avoid reading my wife’s email, or swamping her address book with all my student email addresses (which Google puts into it automatically)? If I play a game, and then have to stop and go back to it later, what if one of my daughters wants to have a go in-between? She might not only end the game I’m playing, but also mess up my high-score records. My todo-list application only has one set of todos, so what if I look at it and suddenly find “Feed my puffle on Club Penguin” on top of my priorities?

On a Unix system you have different user accounts, and you log in and out; this avoids the problem on Mac OS X. But logging in and out is tricky if a system is shared. If I forget to log out when I interrupt my task, and the device is locked, nobody else can use it until I have come back. And how can this be made easy, without constantly having to remember a user name and password?

I somehow suspect that this will be an issue for which there is no satisfactory solution. But if the iPad is to become a general household item like a television or a radio, then there needs to be some non-intrusive way that allows easy sharing. I’m looking forward to finding out what Apple came up with here…!

There were a few things I liked about go. It’s smallish, C-like, has garbage collection, built-in support for concurrency, and unicode strings. Hash-tables (‘maps’) as a first-class data type. A nicely-looking set of libraries for all sorts of purposes. Not quite fast, but with lots of scope for performance improvements. No header files. First class support for unit tests.

This was looking attractive as opposed to Erlang, which is older and more mature/stable, but still not very high-performance, has slightly awkward string handling, and exactly three data types (list, tuple, atom). And a Prolog-style syntax with a number of inconveniences about the use of commas, semicolons, and full stops. Editing a clause is never straightforward.

I have since abandoned go again. It also has inconsistencies (the use of ‘new’ for some data types and ‘make’ for others), and worst of all, there was so much talk about wanting to add generics to the language that I fear they will become a feature of it. I don’t like generics: they seem to me to be more trouble than it’s worth. They make code really hard to read, and inflexible. They might make some kinds of bugs impossible, but in my view that is a feeble gain for wrecking a language. As Knuth (I think) said, part of writing programs is aesthetics. I cannot like Java code full of abstract type annotations. Objective-C is so much cleaner in comparison. And so was go, until now.

Another reason is the concurrency support. Go uses pipes for that, which seems awkward. I much prefer Erlang’s mailboxes, which neatly work together with pattern matching to respond to certain messages and ignore others. You do not need to worry about the order in which messages arrive as much, and the whole communication process is a lot easier with only the basic data types.

So I’m going back to Erlang. I will dig out the string library that I started, and get back into thinking recursively. At least I know where I am with it, and it is not suddenly going to change!

I also adore the capability that I have to flag articles from folks I follow on Twitter and save them to Instapaper.

This sentence has (at least) two readings, which are probably only obvious to a linguist (and who else would care?); I highlight the differences by adding commas:

1. I adore the capability, that I have to flag articles…
2. I adore the capability that I have, to flag articles…

In the first case you adore the capability. And the capability is that you have to do something (flag articles). Sounds rather odd, doesn’t it? The second case is more clear-cut and easy to understand: you can flag articles, and that’s the capability you have and adore.

So in terms of pattern grammar, you’re either looking at N that or N to-inf with capability. If you consult the Cobuild Dictionary, you’ll find that capability only occurs with the second pattern, the to-infinitive, so that you can rule out the first reading.

Another possibility would be to look at it in terms of modality: here we could argue that capability prospects a modality of ability, but have to expresses obligation; the two don’t go together. Hence the first reading sounds odd, as a capability does not usually force you to do anything, but rather enables you. It could, however, be used to signal sarcasm or irony, as in (the obviously made up) I really like that my new computer gives me the capability to have to save my work every five minutes. This is clearly an odd sentence, suggesting that modality works along similar lines as discourse prosody as described by Louw (1993) [for the full reference follow the previous link].

Here we have discussed two ways of disambiguating a sentence, one based on grammatical properties (or typical environments), and one on a non-syntactic phenomenon (modality). Pattern grammar allows us to identify what the typical usage would be, whereas modality explains to us why the first reading is at odds with the corresponding words. Now all we need is a ‘pattern grammar’ for modality!