Earlier this week I gave a talk titled Deferred Gratification (slides) at EuroPython in Birmingham. The title was supposed to hint at how much I love Twisted‘s Deferred class, and that it took me some time to really appreciate it.

While thinking about Deferreds in the days before the talk, I think I put my finger on one of the reasons I find Deferreds so attractive and elegant. The thought can be neatly summarized: Deferreds let you replace synchronous data structures with elegant asynchronous ones.

There are various ways that people get exposed to thinking about programming in an asynchronous style. The great majority do so via building user interfaces, usually with widget libraries wherein one specifies widgets and layouts, sets up event handlers, and then surrenders control to a main loop that thereafter routes events to handlers. Exposure to asynchronous programming via building UIs is becoming much more common as Javascript programmers build client-side web apps that operate asynchronously with web servers (and the “A” in AJAX of course stands for asynchronous). Others become aware of asynchronous programming via writing network code (perhaps using Twisted). Relatively few become aware of asynchronous programming via doing async filesystem I/O.

Because Twisted’s Deferreds don’t have anything to do with UIs or networking or filesystems, you can use them to implement other asynchronous things, like an asynchronous data structure. To show you what I mean, here’s a slightly simplified version of Twisted’s DeferredQueue class, taken from twisted/internet/defer.py:

I find this extremely elegant, and I’m going to explain why.

But first, think about code for a regular synchronous queue. What happens if you call get on a regular queue that’s empty? Almost certainly one of two things: you’ll get some kind of QueueEmpty error, or else your code will block until some other code puts something into the queue. I.e., you either get a synchronous error or you get a synchronous non-empty response.

If you look at the get method in the code above, you’ll see that if the queue is empty (i.e., the pending list is empty), a new Deferred is made, added to self.waiting, and is immediately returned to the caller. So code calling get on an empty queue doesn’t get an error and doesn’t block, it always gets a result back essentially immediately. How can you get a result from an empty queue? Easy: the result is a Deferred. And because we’re in the asynchronous world, you just attach callbacks (like event handlers in the UI world) to the Deferred, and go on your merry way.

If you can follow that thinking, the rest of the code in the class above should be easy to grok. In put, if there are any outstanding Deferreds (i.e., earlier callers who hit an empty queue and got a Deferred back), the incoming object is given to the first of these by passing it to the callback function of the Deferred (and popping it out of the waiting list). If there are no outstanding Deferreds expecting a result, the incoming object is simply appended to self.pending. On the get side, if the queue (i.e., self.pending) is non-empty, the code creates a Deferred that has already been fired (using the succeed helper function) and returns it.

By now, this kind of code seems routine and straightforward to me. But it certainly wasn’t always that way. So if the above seems cryptic or abstract, I encourage you to think it over, maybe write some code, ask questions below, etc. To my mind, these kinds of constructs – simple, elegant, robust, practical, obviously(?) bug-free, single-threaded, etc. – are extremely instructive. You too can solve potentially gnarly (at least in the threaded world) networking coding challenges in very simple ways just by building code out of simple Twisted Deferreds (there’s nothing special about the DeferredQueue class, it’s just built using primitive Deferreds).

For extra points, think about the concept of asynchronous data structures. I find that concept very attractive, and putting my finger on it helped me to see part of the reason why I think Deferreds are so great. (In fact, as you can see from the use of succeed, Deferreds are not specifically asynchronous or synchronous: they’re so simple they don’t even care, and as a consumer of Deferreds, your code doesn’t have to either). That’s all remarkably elegant.

I sometimes hope O’Reilly will publish a second volume of Beautiful Code and ask me to write a chapter. I know exactly what I’d write about :-)

Back in December 2006 I wrote about finishing Proust and made a rough argument about how often anyone on earth finishes the whole thing. The argument was a bit subtle. I was never 100% convinced it was sound, but no-one I showed it to found a hole in it. I still think about the question from time to time. The other day I mentioned the original post to Tim O’Reilly. Later that day, I realized there’s a much simpler way to get an estimate, with far fewer assumptions.

The new approach is simply to divide the number of hours that have passed since In Search of Lost Time was published by the number of people who’ve ever finished it. That average is a crude measure, but it may be nevertheless quite accurate and it’s irresistibly interesting to me to see how it compares to my original 2006 estimate of 2.19 hours.

So, assume 2B people were alive in 1927 when the final volume was published, and 6.4B alive at the end of 2006 (source).

Assume that no-one alive in 1927 was still alive in 2006 (obviously not the case, but not unreasonable and not a significant error). I.e., there were 4.4B births in those 79 years. Note: This is ignoring a significant number of people who were born after 1927 and who died before 2006. But it is including everyone born from 1990 onwards, essentially zero of whom would have read Proust by 2006.

In my original post I estimated that one person in 10K actually finishes the whole book. So that’s 4.4B/10K = 440K people who read the book during the 79 years.

79 years is 28,835 days, or 692,040 hours. Doing the division, 692,040 / 440,000 = 1.57 hours.

I.e., by the above rough reasoning, someone, somewhere on earth, finishes Proust every 1.57 hours, on average.

I find the closeness of the two estimates quite remarkable. There’s only one shared assumption (1 in 10,000 finishes). Both estimates are quite crude, yet there’s only about a 30% difference in the answers. I was expecting them to be much more divergent.

In case you missed it, FluidDB has (finally) launched. I wont be blogging here about FluidDB or Fluidinfo, though will continue to post personal things and of course random bits of code that seem interesting (and small) enough to warrant mention. I have yet another Twisted snippet coming up, though I’m not sure when I’ll get there.

We’re all exhausted and thrilled to have FluidDB out the door. I wont try to describe the feelings, except to say that it’s all incredibly exciting, and that I haven’t been getting much sleep recently. The reaction in the programmer community has been astounding: there are 9 client-side libraries already written (with more on the way), there are tools, there’s a FluidDB Explorer, and little apps are now starting to pop up. We couldn’t be happier. You can see a list of those things here.

To find out more about FluidDB, here are your best choices:

• The FluidDB blog. Recent posts describing FluidDB are:
  • Truly social data.
  • Cloud storage: the real opportunity.
  • Kaleidoscope: 10 takes on FluidDB.
• The FluidDB high-level overview goes into the information model of FluidDB, its permissions system, and the query language.
• The FluidDB-discuss mailing list is a great place to ask questions, and has already seen some detailed discussion on spam and the FluidDB about tag.
• If you’re a programmer looking to learn about FluidDB, the #1 thing you should do is join the #fluidDB IRC channel on irc.freenode.net. There are many people there learning about FluidDB, exchanging tips, and writing new libraries, tools, and apps. You can also read the API documentation.

Thanks for reading along! The real journey is probably only just beginning…

Here’s the video from my talk on May 22, 2009 at the Postgres Conference (PGCon) in Ottawa. This blog theme cuts off the slides on the right; a better place to watch it is here.

Here are the slides from my talk on May 22, 2009 at the Postgres Conference (PGCon) in Ottawa. The video will be available soon.

Here’s just a quick note to mention that I’m talking at the annual Postgres Conference aka PGCon. The talk is titled The design, architecture, and tradeoffs of FluidDB, and is at 3pm on May 22nd. So if you happen to be in Ottawa this week…

I could have added the subtitle “How someone who knows nothing about databases wound up in a project to build a database.”

Here are some more thoughts on the (now official) influenza pandemic.

I would like again to emphasize that I’m not an authority and I’m not trying to pass myself off as one.

I’ve already been accused of deliberate fear-mongering. That’s the opposite of my purpose. On the contrary, it’s important to stay calm and there are good reasons for doing so. If you don’t want to know a bit of history and to have some sense of things that have happened in previous pandemics, then you don’t have to read what follows. There’s no harm in staying calm via not knowing. On the other hand, there is harm in being gripped by fear due to ignorance.

If you do read, try to keep in mind that the main point here is that you shouldn’t be overwhelmed by fear or begin to panic. There’s no reason to. Plus it will only make things worse.

On the subject of being an authority and fear-mongering, after I wrote up the first set of thoughts I was invited to be a guest commentator on a radio show. I declined. It feels very irresponsible to say anything about influenza given that I am not an expert and don’t even work in the field anymore, but OTOH it feels irresponsible to remain totally silent given that I know at least some historical things fairly well.

To illustrate the conflict: On April 26 it seemed crystal clear to me that the virus was going worldwide. You only had to have seen the Google map that I twittered about the day before to see that it was going to be all over the place in days. But I didn’t want to point that out, and when I was asked I told the asker to be his own judge. I linked to a map showing Mexico and the US earlier and said “hopefully we wont have to zoom out” – trying to get people to consider that we would probably soon have to zoom out.

So I think I’ve been quite restrained. This post is also restrained. As I said above, there are good reasons not to sensationalize things or to create the impression that people should panic.

So here we go again, a few more thoughts as they come to mind. These are things that I find interesting, with a few scattered opinions (all of which are just guesses). There’s no real structure to this post.

The WHO have announced today that we’re officially in a pandemic. That doesn’t really mean much, but it’s good to have a candid and early declaration – part of the problem historically has been slowness to even admit there’s a problem. The WHO didn’t even exist in 1918.

In case you don’t know, there’s pretty good evidence that humans have been fighting influenza for thousands of years.

The most interesting thing to me in reading about the 1918 pandemic is the social impact of the disease.

One thing to make clear is that the current pandemic is not the 1918 pandemic. I tend to agree with those who say that a pandemic of that nature could not take place today – but note that people, perhaps especially scientists – would have said that at all times prior to and after 1918. We often under-estimate the forces of nature and over-estimate our own knowledge and level of control.

BTW, something like 75% of people who died during World War 1 did so because of the flu pandemic, which didn’t really take off until November of 1918. Amazing.

As I mentioned in my earlier post, under normal circumstances (even in a pandemic), flu doesn’t kill you. It leaves you susceptible to opportunistic follow-on disease. The good news is that we are vastly more informed now than we were in 1918 about the nature of infectious diseases. For example, we know a lot about pneumonia, which we did not in 1918. See the moving story of the amazing Oswald Avery, who dedicated his life to the disease and along the way fingered DNA as the vehicle of genetic inheritance – and never won a Nobel prize.

So the care of people who have been struck down by flu is going to be much more informed this time around. And it will probably be better in practice too. I put it that way because odd societal things happen in a pandemic. I hesitate to go into detail, because some people will assume that things that happened way back when will necessarily happen again this time around.

One of those things is that medical systems get overrun by the sick. Plus, doctors and nurses understandably decide that their jobs have become too dangerous and they stop showing up for work. So there can be a sharp drop-off in the availability of medical help.

So much so that there were reports of doctors and nurses being held hostage in houses in 1918. I.e., if you could get a doctor to visit to attend to your family, the situation was so dire you might consider pulling a gun on him/her and suggesting they make themselves comfortable for the duration.

The problem is not so much that many people are dying, it’s that a much larger number are simultaneously extremely ill and that panic grips them and everyone else. Roughly 30% of all people caught the 1918 flu. I have another post I may write up on that. Normal (epidemic) flu catches 10-15% of people in any give year.

Many of our systems are engineered to provide just-in-time resources, to cut the fat in order to maximize profitability, etc. That means that we’re closer to collapse that would seem apparent. How many days of fresh food are there on hand in a major city?

None of this is meant to be alarmist. But the reality is that alarming things have happened in the past.

Most interesting and revealing to me is that our cherished notions of politeness, of our generosity, our goodwill towards our neighbors, etc., can all go out the window pretty quickly. I’ve long held that all those things are the merest veneer on our underlying biological / evolutionary reality. We’re very fond of the ideas that we’re somehow no longer primates, that we’re not really the product of billions of years of evolutionary history, that somehow the last centuries of vaunted rationality have put paid to all those primitive lower impulses. I think that’s completely wrong. Behavior during a full-scale pandemic is one of the things that makes that very clear.

In a pandemic, if things get ugly, you can expect to see all manner of anti-social behavior. If you read John Barry’s book The Great Influenza or Crosby’s America’s Forgotten Pandemic you’ll get some graphic illustrations.

If I had a supply of tamiflu (which I don’t), I wouldn’t tell anyone. That’s deliberately anti-social. Ask yourself: What would you do if you had kids who were still healthy and your neighbor called you in desperation to tell you that his/her kids seemed to have come down with influenza? Get out your family’s tamiflu supply and hand it over? Lie? What if they knew you had it and you refused to give it or share? What if your neighbor’s kid died and yours never even got the flu? What kind of relationship would be left after the pandemic had passed?

This may all seem a bit extreme and deliberately provocative of me, and yet those sorts of dilemmas (sans tamiflu, naturally) were commonplace in 1918. As you might expect, they don’t always get resolved in ways that accord well with our preferred beliefs about our own natures in easier times. Crosby speculates that the reason the pandemic of 1918 is “forgotten” is due largely to the fact that it coincided with the war, and that people were generally exhausted and dispirited and wanting to move on. I’d speculate further that people en masse frequently behaved in ways that they weren’t proud of, and wanted to forget about it and act as if it hadn’t happened ASAP. That’s just a guess, of course.

In any case, if there’s a full-blown pandemic, societal structures that we take for granted are going to be hugely transformed. Medical services, emergency services, food supply, child care and education, job absenteeism, large numbers of the people who would normally be in charge of things coming down sick and being unable to do their normal jobs, etc. All sorts of things are impacted and lots of them are interconnected. The system breaks down in many unanticipated ways as all sorts of things that “could never happen” are all happening at once.

You might think I’m fear-mongering here, but I’m not. In fact I’m refraining from going into detail. Go read John Barry’s book, or any of the others, and see for yourself.

The important thing to remember in all this is that we are no longer in 1918. BTW, there were also influenza pandemics in 1957 (killing just a couple of million people) and 1968 (killing a mere million).

Apart from the fact that we’ve advanced hugely in medical terms, we are also much better connected. I can sit in the safety of my apartment in Barcelona and broadcast calming information like this blog post to thousands of people. We are better informed. We know that panic and fear greatly compound the impact of a pandemic. They feed on one another and prolong the systemic societal collapse. Because we can communicate so easily via the internet – provided our ISPs stay online – we can help keep each other calm. That’s an important advantage.

So my guess is that this wave isn’t going to be so bad, certainly not in terms of mortality. One thing to keep in mind though is that the virus isn’t going away. It will likely enjoy the Southern hemisphere winter, and we’ll see it again next Northern hemisphere winter. And yes, those are guesses. Because influenza is a single-stranded RNA virus it mutates rapidly (you don’t get the copy protection of a double strand). So this is the beginning, not the end, even if the pandemic fizzles out in the short term. It will be back – probably in less virulent form – but by then we’ll also have a good leg up on potential vaccines, and we’ll also know it’s coming.

OK, I’ll stop there for now. I have tons of other things I could write now that I’m warmed up. You can follow me on Twitter if you like, though I doubt I’ll be saying much about influenza.

If you truly believe I’m fear mongering, please send me an email or leave your email address in the comments. I’ll send you mail with some truly shocking and frightening stuff, or maybe fax you a few pages from some books. Believe me, it gets a lot nastier than anything I’ve described. Things are not that bad, certainly not yet. We’re not in 1918 anymore

So, stay calm, and do the simple things to keep yourself relatively safe. If everyone follows instructions like those, the virus wont have a chance to spread the way it could otherwise. That may sound like pat concluding advice, but there’s actually a lot to it – the epidemiology of infectious disease – in part the mathematics of infection – can be hugely altered depending on the behavior the typical individual. Following basic hygiene and getting your kids to too will make a big difference. There’s no denying that this is going to get worse before it gets better, but we can each do our part to minimize its opportunities.

Here are some thoughts on the current swine influenza outbreak. These are just off the top of my head – I will undoubtedly think of more to say and add it in the comments or another posting. I apologize for the lack of links. I may come back and put some in.

I am both unqualified and qualified to make a few comments. I’m unqualified because I no longer work on influenza virus, because I’m not a virologist, because I have no inside information at all about the current outbreak. OTOH, I have some claim to know what I’m talking about. I worked on influenza virus as part of the Antigenic Cartography team at the University of Cambridge for a few years. We helped the WHO choose the H3N2 strain for the human vaccine. I’ve met the heads of the 4 international flu centers and even been in the WHO Situation Room in Geneva – a self-contained underground fortress. I spent a lot of time hanging out and talking to influenza virologists, many from the Erasmus Medical Centre in Rotterdam. I was even an author on a Science paper on the global spread of epidemic influenza. Plus I’ve read all the books on the 1918 pandemic, which gives some (largely retrospective) insight into what happened back then, and perhaps some insight into what could be about to happen.

I also feel it’s good for someone like me to comment because I’m outside the flu world and the people inside it will be unlikely to say much. Flu is a highly political issue, to put it mildly. People working in the flu research community will be reluctant to speak up. So I should make it very clear that the comments below are just my opinions, and don’t represent anyone else’s thoughts.

I’ll try to just make a few points that I think are fairly sober – neither alarmist, nor dismissive – and to keep speculation out of it.

Apart from the details of the actual virus, the social side of a potential pandemic is extraordinarily interesting. Very few people will have really concrete information, and those that do will still only be making their best guesses.

In a pandemic, or something that looks like it might be one, wild rumors sweep through the population. That will happen on an unprecedented scale this time round.

The virus has, as far as we know, not spent much time in humans yet. Once it does, it will begin to adapt itself in unpredictable ways. It may become more virulent, or less virulent. It may develop resistance to the antivirals that are currently effective. Antiviral resistance has been a topic of great concern for at least a couple of years. The current virus is already known to be resistant to both amantadine and rimantadine, though oseltamivir is still effective.

If you ask virologists what the probability is that there will be another pandemic, they’ll tell you it’s 1.0. It’s just a matter of time until it happens. it’s like a non-zero probability state in a Markov process. When it does happen, what you do in the first phase is critically important. In the case of the avian influenza they would try to immediately cull all potentially infected birds, to stop the virus spreading and mutating and becoming more likely to enter the human population. When it did get into the human population, there would be swift action to isolate it, again to reduce the spread and the time the virus has to adapt. In the case of the avian influenzas in humans, there has been very little airborne transmission, and we’re lucky for that. But the current virus seems to already have that property, which is of great concern.

It would be a miracle if the current epidemic vaccine provided any protection against this virus. The human vaccine does contain a strain against H1N1, but that’s a strain picked based on sampled human viruses from many months ago. The epidemic vaccine is aimed at thwarting what’s known as antigenic drift – the relatively slow accumulation of point mutations in the virus. Pandemic strains arise through antigenic shift in which large chunks of viral genetic material, sometimes whole genes, are mixed between influenza viruses from different species. In a pandemic strain some of the genetic material and the proteins it expresses will very likely never have been seen by a human immune system.

The current WHO standard influenza test kit is not very useful in identifying this strain. They have issued instructions warning against false negatives.

Some aspects of the current outbreak are, to my mind, cause for great concern.

The acting-director of the CDC has already said: “There are things that we see that suggest that containment is not very likely.” That is a remarkably candid statement. I think it’s very clear that the cat is out of the bag. The question is how bad is it going to be. That’s impossible to tell right now, because we do not know what the virus will look like in the future, after it has had time to mutate and adapt inside humans.

In normal circumstances it takes about 6 months to make the world’s supply of epidemic vaccine. It’s a long and difficult process requiring tons of virus to be grown in chicken eggs. A canidate vaccine strain has to be identified, it has to be one that grows well in the chicken egg (including not killing the chick). Even under the high pressure of a potential pandemic, making a new vaccine is going to take months. By then the virus may have moved on (via mutation) and the vaccine’s efficacy may be less. Note that the 1918 virus killed tens of millions of people over a period much shorter than this.

Diverting the world’s influenza resources to covering a pandemic threat necessarily diverts them from work on epidemic vaccines. Epidemic flu kills roughly 0.5M people a year as it is. Not being able to pay due attention to the epidemic strains is also a bad thing.

The new virus has been popping up in various places in the US in the last days. I expect it will go global in the next couple of days, maximum. What’s to stop it? The virus has been isolated in several diverse areas and in many cases is genetically identical. The 1918 virus also popped up, in many cases inexplicably, across the US. The book America’s Forgotten Pandemic is worth a read.

There were 3 waves of the 1918/19 pandemic. The first was in summer of 1918 – very unusual, as influenza normally falls to extremely low rates during summer. Note that the current outbreak is also highly unseasonal.

The 1918 pandemic killed with a very unusual age pattern. Instead of peaks in just the very young and the very old, there was a W shape, with a huge number of young and healthy people who would not normally die from influenza. There are various conjectures as to the cause of this. The current virus is also killing young and healthy adults.

The social breakdown in a pandemic is extraordinary. If you read The Great Pandemic by John Barry, you’ll get some sense of it. America’s Forgotten Pandemic also helps give some idea of what it must have been like.

No-one knows just how many people died as a result of the 1918 pandemic. Estimates generally range between 40M and 100M, and have trended upwards over the years. Influenza is not the easiest to diagnose (hence the category ILI – influenza-like illnesses). It also strips the throat of protective epithelial cells, leaving you susceptible to opportunistic follow-on infections, such as pneumonia, which often do the killing.

No-one knows how bad another pandemic might be in terms of mortality. Low estimates are in the single digit millions. Someone from the WHO suggested a significantly higher number about 4 years ago in the context of avian influenza and that number was quickly retracted. Jeff Taubenberger, who was responsible for resurrecting and sequencing the 1918 virus (an extraordinary story, related in a couple of books) has published work saying 100M might be possible. No-one knows, and it depends on many factors, including the characteristics of the virus, how early it is detected, how easily it spreads, how virulent it is (obviously), the social measures taken to combat it, antiviral resistance, and many other factors.

I don’t think anyone knows how the balance between vastly increased medical knowledge and vastly increased national and international travel will play out. If this virus is not popping up all over the world within a week’s time, I’ll be surprised. Airports are already screening people arriving from Mexico, but I imagine it’s too late and it’s certainly not being done globally.

History dictates that you should probably not believe anything any politician says about pandemic influenza. There has been a strong tendency to downplay risks. All sorts of factors are at work in communicating with the public. You can be sure that everything officially said by the WHO or CDC has been very carefully vetted and considered. There’s no particular reason to believe anything else you hear, either :-)

Facemasks have an interesting history, and have made it into law several times. In 1918 we didn’t even know what a virus was, let alone how tiny they are, so the gauze on the masks was likely totally ineffective.

In conclusion, I’d say that the thing is largely out of our hands for the time being. We’re going to have to wait and see what happens, and make our best guesses along the way.

The influenza people at the CDC and the other international labs are an amazing team of experts. They’ve been at this game for a very long time and they work extremely hard and generally get a bad rap. It’s no wonder flu is such a political issue, the responsibility is high and the tendency towards opaqueness is understandable. Despite all the expertise though, at bottom you have an extremely complex virus – much of whose behavior is unknown, especially in the case of antigenic shift, especially when it is so young, and especially when you don’t know what nearby mutational opportunities may exist for it in antigenic space – spreading in a vastly more complex environment (our bodies), and with us moving and interacting in odd ways in a complex and extremely interconnected world. It’s a wonder we know as much as we do, but in many ways we don’t know much at all.

Once in a while I run across a piece of writing that has little or nothing to do with being an entrepreneur, but which reads as though it did. I posted an example in 2007: Orwell writing about and quoting T.S.Eliot: “Each venture is a new beginning, a raid on the inarticulate with shabby equipment always deteriorating“.

Below is one I encountered a few days ago. Can you place it? You can find the answer on Google in a flash.

The glamour of youth enveloped his parti-coloured rags, his destitution, his loneliness, the essential desolation of his futile wanderings. For months — for years — his life hadn’t been worth a day’s purchase; and there he was gallantly, thoughtlessly alive, to all appearances indestructible solely by the virtue of his few years and of his unreflecting audacity. I was seduced into something like admiration — like envy. Glamour urged him on, glamour kept him unscathed. He surely wanted nothing from the wilderness but space to breathe in and to push on through. His need was to exist, and to move onwards at the greatest possible risk, and with a maximum of privation. If the absolutely pure, uncalculating, unpractical spirit of adventure had ever ruled a human being, it ruled this be-patched youth. I almost envied him the possession of this modest and clear flame. It seemed to have consumed all thought of self so completely, that even while he was talking to you, you forgot that it was he— the man before your eyes— who had gone through these things.

Andrew Hensel was an extraordinary human being.

We were graduate students together at The University of Waterloo in Canada in 1986-88. I met him on my first day there and we spent many hours together on a daily basis over the next 2.5 years. I don’t want to try to say too much about him now. It occurred to me a few days ago that I might post a few stories here. We did lots of crazy things. At one point I had wanted to write something titled “100 things to a Hensel” and I made a bunch of notes, but it went no further.

I wrote about him in my Ph.D. acknowledgments in 1995:

Andrew Hensel, with whom I shared so much of my two and a half years at Waterloo, was the most original and creative person I have ever known well. Together, we dismantled the world and rebuilt it on our own crazy terms. We lived life at a million miles an hour and there was nothing like it. Five years ago, Andrew killed himself. There have been few days since then that I have not thought of him and the time we spent together.

I still think about him frequently. Today I was remembering one of his many, many oddball projects (most of which went unfinished), which he called “10,000 things”. It was to be a list of 10,000 things that he thought of. By the time he started sending them to me we had both dropped out of Waterloo. He was back in Australia and I was in Munich.

He only sent me 300 of the to-be 10,000. Of course I still have them. They’re all very short. At the risk of being thought macabre I’ve decided to bring Andrew back a very little and post them to Twitter, chosen at random, one a day. You can follow adhensel to get just a glimpse of his mind. The first tweet, people being planted into earth, is already up.

There are at least half a dozen twitterers who knew Andrew, including one who knew him probably better than anybody. Once in a while I get email from someone who finds my online mentions of him. Invariably they also found him extraordinary.

What would Andrew have made of Twitter? I have no doubt at all that he’d have immediately dismissed it as “weak”. That was one of his favorite adjectives. Almost everything was weak. It’s a small miracle to me to partly bring him back to life 18 years after he died, by posting just some of his 10,000 things to Twitter.

And… my apologies to anyone who knew Andrew and who finds this upsetting.

On Monday Tim O’Reilly posted a Twitter tweet suggesting to Robert Scoble that he contact me while in Barcelona.

First off, Tim is very generous in doing this. He’s ultra connected and he spends a significant amount of his time in Twitter pointing things out, connecting people, and re-tweeting stuff he finds interesting. Re-tweeting is really important because when you tweet you only reach the people who are already following you. But when someone re-tweets you, you reach new people who likely have no idea of your existence. And when Tim does the re-tweeting there can be a big impact. 24 hours after his message to Robert I had 50 new followers. Tim explicitly tries to help people doing things he finds interesting, but who have just a small number of Twitter followers. He filters and amplifies information, broadcasting it out to his 16,000+ followers. Robert was in a hotel about 10 minutes’ walk from my place and I had no idea. A mutual friend in California noticed and took a minute to connect us. That’s really something, and it perfectly illustrates some of the value of Twitter.

I met Robert yesterday afternoon and we spent 6 hours together. It was great. You can see at once why he’s been so successful: he’s smart, he’s thoughtful, he’s sympathetic, and he’s a careful listener. I had no idea what to expect, and seeing as what we’re building can take some time to sink in, I wondered what sort of an audience he’d be.

After we’d climbed around up in the Sagrada Familia (official site, wikipedia), Robert came back to my place to see a demo of the things I’d been describing. We sat down and he pulled out his cell phone and asked if he could film me. I didn’t really think about it and said of course. It didn’t dawn on me that we were doing an informal interview, and I was totally unprepared – which is probably a good thing.

In the end we filmed 4 segments: parts one, two, three, and four. There’s also been some discussion here on Robert’s FriendFeed page.

So if you’ve been wondering what we’re building in here, go watch the videos.

I had no idea all this was about to come down. The Fluidinfo web site (a generous word) was a single page with no contact information, no nuthin’. We simply haven’t needed a web site of any description yet. I went and added a box so you can sign up to receive news of the alpha launch.

And then there was this, posted on Twitter, and which I have absolutely no shame in reproducing (this is a blog, after all):

Wow, what @terrycojones showed me last night (a new kind of database that he’s been workng on for 11 years) blew me away. Uploading vids now

Now I have to put my head back down with Esteve to get the alpha out the door ASAP.

In March 2004 I was in a silly mood (yes, a euphemism) and dreamed up the idea of airports installing giant mood rings as security devices. Prospective passengers would be made to walk through the mood ring. If it showed a happy and peaceful color, you could get on the flight. You could carry a gun or even a bomb, no problem. But if the mood ring indicated anger, you’d be denied entry or forced to fly in a special Angry Class. In Angry Class the service is shit. The airline knows the passengers are angry before they get on the plane, so why bother?

I started to write the news:

Today the US Department of Homeland Security announced that the US is insisting that countries install the new Passenger Mood Assessment Security Screening (PMASS) in all airports with direct or connecting flights to the US. The system, developed by Kellogg Brown & Root (a wholly-owned subsidiary of Halliburton, an oil-services company), is based on techniques borrowed from functional magnetic resonance imaging and, most controversially, the mood ring industry.

Ashcroft, Bush and Rumsfeld flanked Homeland Security Director Tom Ridge today as he announced the new measures. "People have no right to turn up at the airport with a bad attitude."

Halliburton wins a no-bid contract, shortly after moving to buy up all known mood ring manufacturers, mystifying investors (stock falls sharply, then rises sharply on news of contract). To fly people to the US, other countries are forced install giant mood rings and to hand over the mood data of all embarking passengers to US authorities.

You get the picture. I sent an outline of this to the folks at The Onion, but never heard back.

It’s a joke, of course. Couldn’t possibly happen. Right? Think again.

I signed up for a dedicated server from Serverpronto in late 2006. At $30/month for a real Linux box the price seemed great. I recommended them to a friend who also signed up.

On July 29, 2007 we noticed that both our machines had been hacked into. We’re both very security conscious and we didn’t know how this had happened. Both machines had been broken into in the same way, from the same IP address, leaving behind the same evidence. It was pretty clearly a root kit.

I immediately contacted Serverpronto and asked for help. They offered to do a full system restore for $69 and were otherwise completely passive. I suggested that they might want to take a bit more interest as perhaps their other clients had also been hacked. After a couple of mails I got an email saying that they would make their best effort. But it was too late and I replied asking them to cancel service.

From terry Mon Jul 30 02:32:59 +0200 2007
Message-ID: <18093.12731.650708.403211@terry-jones-computer.local>
Date: Mon, 30 Jul 2007 02:32:59 +0200
From: Terry Jones <terry@XXX.es>
To: "SP1443" <support@serverpronto.com>
Subject: Re: [WYL-89853]: Machines broken into?
In-Reply-To: Your message at 18:34:08 on Sunday, 29 July 2007
References: <jlypcw.k87@serverpronto.infolink.com>

Thanks, but no. I would simply like to cancel service. What more
do I need to do?

Terry

====== Please reply above this line ======
Machines broken into?

Mr. Jones. I will transfer to the abuse team, and the Customer
Service your issue.

We will find the best solution.

They replied:

From support@serverpronto.com  Tue Jul 31 00:00:10 2007
Message-ID: <jlzkyj.p6l@serverpronto.infolink.com>
From: "SP1422" <upport@serverpronto.com>
Subject: [WYL-89853]: Machines broken into?
Date: Mon, 30 Jul 07 05:56:43 -0400
To: terry@jon.es

Machines broken into?

69.60.109.173 has been taken offline.
ServerPronto Support

Ticket Details
=========
Ticket ID: WYL-89853
Tracking URL: https://serverpronto.infolink.com/esupport/index.php?_a=tickets&_m=viewmain&emailre=terry@jon.es&ticketkeyre=dee55eec&_i=WYL-89853
Department: Technical Support
Created On: 29 Jul 2007 07:57 AM
Last Update: 29 Jul 2007 05:34 PM
Status: Closed

That’s it, right? End of story.

Unfortunately not. First of all, my friend could not get Serverpronto to stop billing after he also canceled his contract. They just kept billing. After many phone calls and emails, they eventually had to get rid of Serverpronto via getting the credit card company to refuse all charges.

I thought I had gotten off their books. But I got a phone call from Deutsche Bank a couple of months ago telling me that a company called Serverpronto was trying repeatedly to charge my now-expired credit card. I was completely surprised and told them that I had nothing to do with Serverpronto and that I had canceled service with them. I had been receiving mail from Serverpronto telling me that my credit card expiry date was coming up. I mailed them to tell them there was a mistake and that I was no longer a customer, and they promised to look into it.

And now, today, I get a statement from my bank showing that they have approved two charges for a total of $356.30 from Serverpronto!!!! That means those evil assholes have somehow called Deutsche Bank and managed to charge me for a whole year of costs on an account that was closed, for a server that was not up, and on a credit card that had expired.

I am assured by my friend that I will never see a cent of that money again.

Deutsche Bank are about to hear from me.

DON’T USE SERVERPRONTO.

Go search Google for things like Serverpronto billing cancel ripoff. You’ll get tons of hits. I should have done more homework before I signed up. They actually had good support. But the people running the billing department deserve a slow painful death. This is obviously a deliberate policy designed to extract the maximum out of their trying-to-depart customers.

To whom do I complain?

Here, finally, are some thoughts on the creation of value. I don’t plan to do as good a job as the subject merits, but if I don’t take a rough stab at it, it’ll never happen.

I’ll first explain what I mean by “the creation of highly non-uniform value”. I’m talking about ideas that create a lot of (monetary) value for a very small number of people. If you made a graph and on the X axis put all the people in the world, in sorted order of how much they make from an idea, and on the Y axis you put value they each receive, we’re talking about distributions that look like the image above, but much more skewed.

In other words, a setting in which a very small number of people try to get extremely rich. I.e., startup founders, a few key employees, their investors, and their investors’ investors. BTW, I don’t want to talk about the moral side of this, if there is one. There’s nothing to stop the obscenely rich from giving their money away or doing other charitable things with it.

So let’s just accept that many startup founders, and (in theory) all venture investors, are interested in turning ideas into wealth distributions that look like the above.

I was partly beaten to the punch on this post by Paul Graham in his essay Why There Aren’t More Googles? Paul focused on VC caution, and with justification. But there’s another important part of the answer.

One of the most fascinating things I’ve heard in the last couple of years is an anecdote about the early Google. I wrote about it in an earlier article, The blind leading the blind:

…the Google guys were apparently running around search engine companies trying to sell their idea (vision? early startup?) for $1M. They couldn’t find a buyer. What an extraordinary lack of.. what? On the one hand you want to laugh at those idiot companies (and VCs) who couldn’t see the huge value. OK, maybe. But the more extraordinary thing is that Larry Page and Sergei Brin couldn’t see it either! That’s pretty amazing when you think about it. Even the entrepreneurs couldn’t see the enormous value. They somehow decided that $1M would be an acceptable deal. Talk about a lack of vision and belief.

So you can’t really blame the poor VCs or others who fail to invest. If the founding tech people can’t see the value and don’t believe, who else is going to?

I went on to talk about what seemed like it might be a necessary connection between risk and value.

Following on…

After more thought, I’m now fairly convinced that I was on the right track in that post.

It seems to me that the degree to which a highly non-uniform wealth distribution can be created from an idea depends heavily on how non-obvious the value of the idea is.

If an idea is obviously valuable, I don’t think it can create highly non-uniform wealth. That’s not to say that it can’t create vast wealth, just that the distribution of that wealth will be more widely spread. Why is that the case? I think it’s true simply because the value will be apparent to many people, there will be multiple implementations, and the value created will be spread more widely. If the value of an idea is clear, others will be building it even as you do. You might all be very successful, but the distribution of created value will be more uniform.

Obviously it probably helps if an idea is hard to implement too, or if you have some other barrier to entry (e.g., patents) or create a barrier to adoption (e.g., users getting positive reinforcement from using the same implementation).

I don’t mean to say that an idea must be uniquely brilliant, or even new, to generate this kind of wealth distribution. But it needs to be the kind of proposition that many people look at and think “that’ll never work.” Even better if potential competitors continue to say that 6 months after launch and there’s only gradual adoption. Who can say when something is going to take off wildly? No-one. There are highly successful non-new ideas, like the iPod or YouTube. Their timing and implementation were somehow right. They created massive wealth (highly non-uniformly distributed in the case of YouTube), and yet many people wrote them off early on. It certainly wasn’t plain sailing for the YouTube founders – early adoption was extremely slow. Might Twitter, a pet favorite (go on, follow me), create massive value? Might Mahalo? Many people would have found that idea ludicrous 1-2 years ago – but that’s precisely the point. Google is certainly a good example – search was supposedly “done” in 1998 or so. We had Alta Vista, and it seemed great. Who would’ve put money into two guys building a search engine? Very few people.

If it had been obvious the Google guys were doing something immensely valuable, things would have been very different. But they traveled around to various companies (I don’t have this first hand, so I’m imagining), showing a demo of the product that would eventually create $100-150B in value. It wasn’t clear to anyone that there was anything like that value there. Apparently no-one thought it would be worth significantly more that $1M.

I’ve come to the rough conclusion that that sort of near-universal rejection might be necessary to create that sort of highly non-uniform wealth distribution.

There are important related lessons to be learned along these lines from books like The Structure of Scientific Revolutions and The Innovator’s Dilemma.

Now back to Paul’s question: Why aren’t there more Googles?

Part of the answer has to be that value is non-obvious. Given the above, I’d be willing to argue (over beer, anyway) that that’s almost by definition.

So if value is non-obvious, even to the founders, how on earth do things like this get created?

The answer is passion. If you don’t have entrepreneurs who are building things just from sheer driving passion, then hard projects that require serious energy, sacrifice, and risk-taking, simply wont be built.

As a corollary, big companies are unlikely to build these things – because management is constantly trying to assess value. That’s one reason to rue the demise of industrial research, and a reason to hope that cultures that encourage people to work on whatever they want (e.g., Google, Microsoft research) might be able to one day stumble across this kind of value.

This gets me to a recent posting by Tim Bray, which encourages people to work on things they care about.

It’s not enough just to have entrepreneurs who are trying to create value. As I’m trying to say, practically no-one can consistently and accurately predict where undiscovered value lies (some would argue that Marc Andreessen is an exception). If it were generally possible to do so, the world would be a very different place – the whole startup scene and venture/angel funding system would be different, supposing they even existed. Even if it looks like a VC or entrepreneur can infallibly put their finger on undiscovered value, they probably can’t. One-time successful VCs and entrepreneurs go on to attract disproportionately many great companies, employees, funding, etc., the next time round. You can’t properly separate their raw ability to see undiscovered value from the strong bias towards excellence in the opportunities they are later afforded. Successful entrepreneurs are often refreshingly and encouragingly frank about the role of luck in their success. They’re done. VCs are much less sanguine – they’re supposed to have natural talent, they’re trying to manufacture the impression that they know what they’re doing. They have to do that in order to get their limited partners to invest in their funds. For all their vaunted insight, roughly only 25% of VCs provide returns that are better than the market. The percentage generating huge returns will of course be much smaller, as in turn will be those doing so consistently. I reckon the whole thing’s a giant crap shoot. We may as well all admit it.

I have lots of other comments I could make about VCs, but I’ll restrict myself to just one as it connects back to Paul’s article.

VCs who claim to be interested in investing in the next Google cannot possibly have the next Google in their portfolio unless they have a company whose fundamental idea looks like it’s unlikely to pan out. That doesn’t mean VCs should invest in bad ideas. It means that unless VCs make bets on ideas that look really good – but which are e.g., clearly going to be hard to build, will need huge adoption to work, appear to be very risky long-shots, etc. – then they can’t be sitting on the next Google. It also doesn’t mean VCs must place big bets on stuff that’s highly risky. A few hundred thousand can go a long way in a frugal startup.

I think this is a fundamental tradeoff. You’ll very frequently hear VCs talk about how they’re looking for companies that are going to create massive value (non-uniformly distributed, naturally), with massive markets, etc. I think that’s pie in the sky posturing unless they’ve already invested in, or are willing to invest in, things that look very risky. That should be understood. And so a question to VCs from entrepreneurs and limited partners alike: if you claim to be aiming to make massive returns, where are your necessary correspondingly massively risky investments? Chances are you wont find any.

There is a movement in the startup investment world towards smaller funds that make smaller investments earlier. I believe this movement is unrelated to my claim about non-obviousness and highly non-uniform returns. The trend is fuelled by the realization that lots of web companies are getting going without the need for traditional levels of financing. If you don’t get in early with them, you’re not going to get in at all. A big fund can’t make (many) small investments, because their partners can’t monitor more than a handful of companies. So funds that want to play in this area are necessarily smaller. I think that makes a lot of sense. A perhaps unanticipated side effect of this is that things that look like they may be of less value end up getting small amounts of funding. But on the whole I don’t think there’s a conscious effort in that direction – investors are strongly driven to select the least risky investment opportunities from the huge number of deals they see. After all, their jobs are on the line. You can’t expect them to take big risks. But by the same token you should probably ignore any talk of “looking for the next Google”. They talk that way, but they don’t invest that way.

Finally, if you’re working on something that’s being widely rejected or whose value is being widely questioned, don’t lose heart (instead go read my earlier posting) and don’t waste your time talking to VCs. Unless they’re exceptional and serious about creating massive non-uniformly distributed value, and they understand what that involves, they certainly wont bite.

Instead, follow your passion. Build your dream and get it out there. Let the value take care of itself, supposing it’s even there. If you can’t predict value, you may as well do something you really enjoy.

Now I’m working hard to follow my own advice.

I had to learn all this the hard way. I spent much of 2008 on the road trying to get people to invest in Fluidinfo, without success. If you’re interested to know a little more, earlier tonight I wrote a Brief history of an idea to give context for this posting.

That’s it for now. Blogging is a luxury I can’t afford right now, not that I would presume to try to predict which way value lies.

When I was younger, I didn’t know what to make of it when people rejected my ideas. Instead of fighting it, trying again, or improving my delivery, I’d just conclude that the rejector was an idiot, and that it was their loss if they didn’t get it.

For example, I put considerable time and effort into writing academic papers, several of which were rejected, to my surprise. I’d never considered that the papers might not be accepted. When this happened, I wouldn’t re-submit them or try to re-publish them. By then I would usually have moved on to doing something else anyway.

When I applied for jobs, it never entered my mind that I might not be wanted. How could anyone not want me? After a couple of years working on my current ideas, I applied for a computer science faculty position at over 40 US universities. I refused to emphasize my well-received and published Ph.D. work, of which I was and am still proud, because I was no longer working in that area.

I was convinced the new ideas would be recognized as being strong.

But guess what? I was summarily rejected by all 40+ universities. I only got one interview, at RPI. No other school even wanted to meet me. I kept all the rejection letters. I still have them. (Amusingly, I was swapping emails with Ben Bederson earlier this year and it transpired that he’d had the same experience, also with 40 universities, and he too kept all his rejection letters!)

You never learn more than when you’re being humbled.

I’ve now returned to those same ideas and have been working on them for the last 3 years. In January 2007 I went and met with a couple of the most appropriately visionary VCs to tell them what I was building. I was naïve enough to think they might back me at that early point. Wrong. They suggested I come back with a demo to concretely illustrate what the system would allow people to do. That was easier said than done – the system is not simple. I spent 2007 building the core engine, a 90% fully-functional demo of the major application, several smaller demo apps (including a Firefox toolbar extension built by Esteve Fernandez), and added about 20 sample data sets to further illustrate possibilities.

That’ll show ‘em, right? I went out in November 2007 armed with all this, and began talking to a variety of potential investors. I was sure VCs would be falling over themselves to invest, especially given that we were working on some mix of innovative search, cloud computation, APIs, and various Web 2.0 concepts, and that tons of VCs claimed to be looking for the Next Big Thing in search, and for Passionate Entrepreneurs tackling Hard Problems who wanted to build Billion Dollar Companies, etc., etc.

You guessed it. Over the next year literally dozens of potential investors all said no. The demo wasn’t enough. Would people use it? Could we build the real thing? Would it scale? Where was the team? What are you doing in Barcelona? “Looks fascinating, do please let us know when you’ve released it and are seeing adoption,” they almost universally told me. The standout exception to this was Esther Dyson, who agreed to invest immediately after seeing the demo, and whose courage I hope I can one day richly reward.

What to make of all this rejection?

One thing that became clear is that if you’re smarter than average, you’ll almost by definition be constantly thinking of things too early. Maybe many years too early. Your ideas will seem strange, doubtful, and perhaps plain wrong to many people.

This makes you realize how important timing is.

Being right with an idea too early and trying to build a startup around it is similar to correctly knowing a company is going to fail, and immediately rushing out to short its stock. Even though you’re right, you can be completely wiped out if the stock’s value rises in the short term. You were brilliant, insightful, and 100% correct – but you were too early.

Getting timing right can clearly be partly based on calculation and reason. But given that many startups are driven by founder passion, I think luck in timing plays an extremely important role in startup success. And the smarter and more far-sighted you are, the greater the chance that your timing will be wrong.

So the that’s the first thing to understand: if you’re smarter than average, your ideas will, on average, be ahead of their time. Some level of rejection comes with the territory.

But I’d go much further than that, and claim that if you are not seeing a very high level of rejection in trying to get a new idea off the ground, you’re probably not working on anything that’s going to change the world or be extremely valuable.

That might sound like an outrageous extrapolation (or even wishful thinking, given my history). Later tonight I plan to explain this claim in a post on the connections between passion, value, non-obviousness, and rejection. That’s the subject I really want to write about.

For now though, I simply want to say that I’ve come to understand that having one’s ideas regularly rejected is a good sign. It tells you you’re either on a fool’s errand, or that you’re doing something that might actually be valuable and important.

If you’re not going to let rejection get you down, you might content yourself by learning to ignore it. But you can do better. You can come to regard it as positive and affirming. Without becoming pessimistic or in any way accepting defeat, you can come to expect to be rejected and even to embrace it.

If you can do that, rejection loses its potential for damage. As Paul Graham pointed out, the impact of disappointment can destroy a startup. That’s an important observation, and a part of why startups can be so volatile and such a wild ride.

I don’t mean to suggest that you don’t also do practical things with rejection too – like learn from it. That’s very important and will help you shape your product, thoughts, presentation, expectations, etc. Again, see Paul’s posting.

But I think the mental side of rejection is more important than the practical. The mental side has more destructive potential. You have to figure out how to deal with it. If you look at it the right way you can turn it into something that’s almost by definition positive, as I’ve tried to illustrate.

In a sense I even relish it, and use it for fuel. There are little tricks I sometimes use to keep myself motivated. I even keep a list of them (and no, you can’t see it). One is imagining that some day all the people who rejected me along the way will wring their hands in despair at having missed such an opportunity :-)

I’ve not been universally rejected, of course. There are lots of people who know what we’re doing and are highly supportive (more on them at a later point). If I’d been universally rejected, or rejected by many well-known people whose opinions I value, I probably would have stopped by now.

I’ve had to learn to see a high level of rejection as not just normal but a necessary (but not sufficient!) component of a correct belief that you’re doing something valuable.

Stay tuned for the real point of this flurry of blogging activity.

In 1997 I had a fairly simple idea.

I spent a year thinking about it obsessively: where it might lead, what it would enable, how to implement it, etc. As I considered and refined it during this time, I began to appreciate its generality and power. I soon realized you could use it computationally to do anything and everything.

By day I was running Teclata S.L. a too-early web company in Barcelona. By night, very often until 4 or 5am, I was up thinking and writing, trying to get a better grip on what I’d dreamed up. I wrote hundreds of pages of notes. That’s a year of thinking, with no coding at all.

Teclata was killing me. I was burning to work on my ideas instead. I was exhausted, and several times literally in tears with the frustration. So in 1998 we sold the company and I started as a postdoc in Jim Hollan‘s Distributed Cognition and Human Computer Interaction lab in the Department of Cognitive Science at UCSD. I spent a year there writing code to build a prototype. I worked typically 15 hours a day for a whole year, producing roughly 30K lines of C code. During that year I think I went out only a few nights, saw exactly one movie, and made no new friends apart from Al Davis, a brilliant homeless guy who hung around campus. The prototype implementation worked, and I built some trivial applications on top of it.

I’ve now spent most of 2006-2008 back working on the same ideas, founding Fluidinfo in the process.

From March 2006 I began re-coding my UCSD work in Python. During 2007 I built a demo version, writing the core engine myself. One other person, Daniel Burr, worked on building a web front-end for a particularly important application. That was another 30K lines of code. To fund that effort, we sold a small apartment we owned in Barcelona and I took a small investment from my parents. During 2008, with funding from Esther Dyson, I’ve worked on a full distributed implementation, with Esteve Fernandez. We’re getting there, and are planning to release something in early 2009. I can’t wait to be able to talk about it all more openly.

That’s 5 years of my life working to reduce this idea to practice.

The opportunity cost in the last few years has been huge. The career risk feels huge. The salary cost has definitely been huge. I’m 45, and my most expensive personal possession is either my CD player or a €200 bicycle I bought about 5 years ago (I have a few computers – but they’re all company/job bought). Plus, I have 3 kids.

I have many other blog posts I’d love to write about this journey (e.g., pond scum), but blogging is a low priority right now. I’ve just jotted down these notes in order to point to them from an upcoming post on the importance of passion, innovation, and the creation of value.

I’ve had plenty of time to think about the subject.

I drove from Oakland up to the O’Reilly Foo camp last Friday. The O’Reilly offices are just outside Sebastopol, CA. I stopped at an ATM and my GPS unit got totally confused. So I took a few turns at random and wound up on Florence Avenue. I drove a couple of hundred meters and started seeing big colorful structures out the front of many houses. They were so good I stopped, got out my camera, and took a whole bunch of pictures.

I talked to a man washing his car in his driveway. He told me that “Patrick” had created all the figures, and installed them on the front lawns. I got the impression that it was all free. Soon after I found the house that was unmistakably Patrick’s and seeing a man loading things into a pickup truck I went up and asked if he was Patrick. It was him and we had a friendly talk (mainly me telling him he was amazing). He gave me a calendar of his work.

Click on the thumbnails below to see bigger versions. There’s even a FC Barcelona structure. As I found out later, lots of people (of course) have seen these sculptures. When I got to Foo, there was one (image above) outside the O’Reilly office. Google for Patrick Amiot or Florence Avenue, Sebastopol and you’ll find much more. And Patrick has his own web site.

OK, this will be a quick one. I’m trying to post occasional videos taken in my neighborhood.

Here you have a typical Catalan scene: a band playing and people dancing Sardanas. You can see this any weekend in front of the cathedral. But this was in the Born and I happened across it on the way home. It’s 60 steps from my front door (yes, I counted).

I don’t really like this music. Like living in Santa Fe and eating Southwest cuisine, I thought it was great at first but that quickly changed. I don’t enjoy the too-reedy quality of the sound and that it’s almost always identical. It’s also really long. But you may go ahead and enjoy it. Be my guest. I really like it when the first person does the initial piping and beats the tiny drum attached to his forearm near the elbow. The dancing starts a couple of minutes into the video. It’s cute.

The church in the background is Santa Maria del Mar, whose stained glass windows and gargoyles are about 4 meters from my balcony.

[This is a bit rambling / repetitive, sorry. I don't have time to make it shorter, etc.]

Last year at FOWA I had a discussion with Paul Graham about programming and programmers in which we disagreed over the importance of knowing the fundamentals.

By this I mean the importance of knowing things down to the nuts and bolts level, to really understand what’s going on at the lower levels when you’re writing code. I used to think that sort of thing mattered a lot, but now I think it rarely does.

I well remember learning to program in AWK and being acutely aware of how resource intensive “associative arrays” (as we quaintly called them in those days) were, and knowing full well what was going on behind the scenes. I wrote a full Pascal compiler (no lex, no yacc) in the mid-80′s with Keith Rowe. If you haven’t done that, you really can’t appreciate the amount of computation that goes on when you compile a program to an executable. It’s astonishing. I did lots of assembly language programming, starting from age 15 or so, and spent years squeezing code into embedded environments, where a client might call to ask if you couldn’t come up with a way to reduce your executable code by 2 bytes so it would fit in their device.

But you know what? None of those skills really matter any more. Or they matter only very rarely.

The reason is that best practices have been worked out and incorporated into low-level libraries, and for the most part you don’t need to have any awareness at all of how those levels work. In fact it can be detrimental to you to spend years learning all those details if you could instead be learning how to build great things using the low-level libraries as black-box tools.

That’s the way the world moves in general. Successive generations get the accumulated wisdom of earlier generations packaged up for them. We used log tables, slide rules, and our heads, while our kids use calculators with hundreds of built-in functions. We learned to read analog 12-hour clocks, our kids learn to read digital clocks (so much easier!) and may not be able to read an analog clock until later. And it doesn’t matter. We buy a CD player (remember them?) or an iPod, and when it breaks you don’t even consider getting it “fixed” (remember that?). You just go out and buy another one. That’s because it’s cheaper and much faster and easier to just get a new one that has been put together by a machine than it is to have an actual human try to open the thing and figure out how to repair it. You can’t even (easily) open an iPod. And so the people who know how to do these things dwindle in number until there are none left. Like watch makers or the specialist knife sharpeners we have in Barcelona who ride around on motorcycles with their distinctive whistles, calling to people to bring down their blunt knives. And it doesn’t matter, at least from a technical point of view. Their brilliance and knowledge and hard-won experience has been encapsulated and put into machines and higher-level tools, or simply baked into society in smaller, more accurate and easier to digest forms. In computers it goes down into libraries and compilers and hardware. There’s simply no need for anyone to know how, learn how, or to bother, to do those sorts of things any more.

Note that I’m not saying it’s not nice to have your watch repaired by someone with a jeweler’s eyepiece or your knife or scissors sharpened in the street. I’m just noting the general progression by which knowledge inevitably becomes encapsulated.

In my discussion with Paul Graham, he argued that it was still important for tech founders to be great programmers at a low level. I argued that that’s not right. Sure, people like that are good to have around, but I don’t think you need to be that way and as I said I think it can even be detrimental because all that knowledge comes at a price (other knowledge, other experience).

I work with a young guy called Esteve (Hi Esteve!). He’s great at many levels, including the lower ones. He’s also a product of a new generation of programmers. They’re people who grew up only knowing object-oriented programming, only really writing in very high-level languages (not you Esteve! I mean that just in general), who think in those terms, and who instead of spending many years working with nuts and bolts spent the years working with newer high-level tools.

I think people like Esteve have a triple advantage over us dinosaurs. 1) They tend to use more powerful tools; 2) Because they use better tools, they are more comfortable and think more naturally in the terms of the higher-level abstractions their tools present them; and 3) they also have more experience putting those tools and methods to good use.

The experience gap widens at double speed, just as when a single voter changes side; the gap between the two parties increases by two votes. Even when the dinosaur modernizes itself and learns a few new tricks, you’re still way behind because the 25 year-old you’re working with (again, excluding Esteve) has never had to work at the nuts and bolts level. They think with the new paradigms and can put more general and more powerful tools directly into action. They don’t have to think about protocols or timeouts or dynamically resizing buffers or partial reads or memory management or data structures or error propogation. They simply think “Computer, fetch me the contents of that web page!” And most of the time it all just works. When it doesn’t, you can call in a gray-haired repair person or, more likely, just throw the busted tool away and buy another (or just get it free, in the case of Open Source software).

That’s real progress, and to insist that we should make the young suffer through all the stuff we had to learn in order to build all the libraries and compilers etc., that are now available to us all is just wrong. It’s wrong because it goes against the flow of history, because it’s counter-productive, and because it smacks of “I had to suffer through this stuff, walk barefoot to school in the snow, and therefore you must too.”

Some of the above will probably sound a bit abstract, but to me it’s not. I think it’s important to realize and accept. The fact that your kid can’t tie their shoelaces because they have velcro and have never owned a shoe with a lace is probably a good thing. You don’t know how to hunt your own food or start a fire, and it just doesn’t matter. The same goes for programming. The collective brilliance of generations of programmers is now built in to languages like Java, Python and Ruby, and into operating systems, graphics libraries, etc. etc., and it really doesn’t matter a damn if young people who are using those tools don’t have a clue what’s going on at the lower levels (as I said above, that’s probably a good thing). One day very few people will. The knowledge wont be lost. It’s just encapsulated into more modern environments and tools.

I’m writing all this down because I’ve been thinking about it on and off since FOWA, but also because of what I’m working on right now. I’m trying to modify 12K lines of synchronous Python code to use Twisted (an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers). The work is a bit awkward and three times I’ve not known how best to proceed in terms of design. Each time, Esteve has taken a look at the problem and quickly suggested a fairly clean way to tackle it. Desperate to cook up a way to think that he might not be that much smarter than I am, I’m forced into a corner in which I conclude that he has spent more time working with new tools (patterns, OO, a nice language like Python). So he looks at the world in a different way and naturally says “oh, you just do that”. Then I go do the routine work of making his ideas work – which is great by me, I get to learn in the best way, by doing. How nice to hire people who are better than you are.

That’s it. Encapsulation is inevitable. So you either have to embrace it or become a hand-wringing dinosaur moaning about the kids of today and how they no longer know the fundamentals. It’s not as though any of us could survive if we suddenly had to do everything from first principles (hunt, rub sticks together to make fire, etc). So relax. Enjoy it. The young are much better than we are because they grow up with better tools and they spend more time using them. It’s not enough to learn them when you’re older, even if you can do that really fast. You’ll never catch up on the experience front.

But it sure is fun to try.

[Listening just now to the Tiger Lillies song Bankrobber Blues I decided to post another little fictional story I wrote some years ago. The first in the series was Lucky Streak. As a kid I had a head full of criminal schemes and at times was convinced I'd inevitably grow up to be a cat burglar. Mathematician by day, cat burglar by night. It didn't pan out that way.]

Manhole

I recently became a photographer. Before high-quality digital cameras became available, I had never had an interest. Sure, I liked photos as much as the next person, but it was all so messy. So imprecise. So… analog. I mean why bother? I’m into digital music in a way I could never have been with tapes or vinyl. Why do anything else? Nostalgia? Tradition? Gimme a break. Analog devices with linear access? Puhlease. I keep an eye on things, I have a feel for this stuff. When technology crosses some line, it’s time to buy in. Before that it’s just too messy.

I recently became a criminal. Historically, crime at a distance was always a challenge. Physical events took place, objects were exchanged, people met or talked on the phone. Messy, way too messy. There was always a chain, a trail: someone had met the guy, someone had given someone else a brown paper bag stuffed with cash and a pistol. With time and money, the feds were eventually going to catch you. You couldn’t find good help, couldn’t recruit and train them, had to trust them, were betrayed or sold out, either to the cops or a rival. Whatever. It made no sense, unless you were the mob and you paid everyone off. Or bumped them off. Messy.

The internet changed all that. No government wants to allow anonymous action, but the net was too big. For nearly twenty years I have watched and been amazed that they didn’t do something to stop it. Now it’s too late, far too late. Anonymous action is possible. It’s not straightforward, one must be careful, but it can be done. Tools like mixmaster (anonymous remailer), spamgourmet (similar), the anonymizer (anonymous browsing), ssh tunnels, cheap and disposable machines for rent on the network running simple forwarding servers, paypal for anonymous transfers of cash, public auction sites for posting messages to those who know what to look for, steganography, instant messaging, strong crypto, wide open windows boxes (with broadband connections and fixed IP addresses) just waiting to be commandeered, etc. The tools are all out there and in most cases they’re freely available. With some knowledge, one can carefully build an action network that no-one’s ever going to trace. The whole point of some of these tools is to preserve anonymity. No logs are kept. Subpoenas are useless when there simply are no records.

Just to warm up, I robbed a bank. Not a white-collar electronic robbery. I’m talking blue-collar physical bank robbery. It’s hardly a well-paying profession, but it’s a good lark. In fact, it pays rather badly. It’s also increasingly common and easier, as tellers and other bank staff are instructed to do exactly as told, not to make a fuss, etc. So why rob a bank? Well, why take a dumbass picture of a tree or your pet dog with your new digital camera? Why? Because you have the technology and you want to try it out. Because you can. Because you’ve gone digital and you can take a no-cost snap of whatever you damned well please.

I, physically, robbed a bank. That’s messy, true. But at some point I wanted skin in the game, wanted the whole thing to be real, to prove that it all worked. Plus, I can’t solve everything at once: I didn’t want someone else actually robbing the bank. That way, I’d either never see the money or I’d leave a trail that could be followed. So I planned carefully and I did it myself.

I recruited a dozen people online. That was actually the easy part. There’s any number of bored and broke college kids out there on the net. I made two roles: watchers and actors. I had actors do various silly extroverted tasks and each time had a watcher turn up to verify that the actor showed up. Everyone got different instructions and everyone was told enough and reported enough back to me so that a) actors knew they were being watched and b) I knew the test tasks were in fact getting done. I paid on task completion and verification. An actor would be instructed to do something oddball and very specific, like walk into a 7-eleven at a certain time in the middle of summer, buy six dozen hotdogs and a case of Bud Lite and comment to the cashier that it sure did look like it might snow. I’d schedule a watcher, and tell them to look out for the person who bought six dozen hotdogs and said how it looked like snowing. The watcher would be told to pay special attention to the other purchases and to the shoes of the purchaser. Afterwards, when I heard back from both, I’d ask the watcher what the other purchase was and tell the actor what sort of shoes they were wearing. So the actor knew that I was in the loop and I could verify that the watcher had been there to verify. Everyone had to do their part or I’d know. In this way, I built up the confidence of my players: they knew that the game was for real and that, for whatever reason, if they just did some seemingly random and simple acts from time to time, they’d be paid. I let the actors and watchers take turns at both roles. If they ever asked what the deal was I just told them I was a university psych professor experimenting on social norms.

I admit this took some time. Two months. I had a few dropouts: people who found it too weird, people who tried to cheat by not turning up while claiming they did, people who just dropped out of contact.

When I was ready, I gave all my little helpers an identical task, each at a different bank. Each was to walk into their appointed bank, get in line for a teller, hand the teller a note that said they had a gun and to hand over all the cash from the drawer. Having done that, the actor was to turn and walk straight out of the bank. No watchers, just twelve people playing actors. Everyone dressed the same. I figured several would drop out at this stage, but on the other hand I’d had them doing some pretty silly stuff, pushing the edge, and I knew at least half of them would be up for it. I also know a thing or two about banks. When the note gets to the teller, the first thing they’ll do is trigger the silent alarm. The security guard in the bank will still be clueless, but the bank manager will see it and so will the cops, who are directly connected.

The banks are dotted around midtown Manhattan. The area’s thick with banks and crazy with traffic. Between 11:00 and 11:05 on July 2, the day before the holiday weekend break, the direct-connect police alarm systems of the 5th and 6th precincts go nuts. Cops cars are scrambled and routed to what turns out to be nine simultaneous false-alarm bank robberies. And one real robbery.

I walk into my bank at ten minutes after eleven. I’m dressed in a red shirt, just like my actors who are busy triggering the alarms in a dozen other midtown banks. After a few minutes I reach the teller and hand her a note: “I have a gun. Keep your mouth shut. Pass me all your cash.” After a pause, she does this without a word. That’s the training, the banks learned that one a long time ago. A hero teller or guard taking a bullet winds up costing more than dozens of unsophisticated heists. Plus the banks are fully insured. So teller training is explicit: trigger the silent alarm and do anything the nasty bank robber tells you to.

I sweep the cash into my bag, there couldn’t have been more than $10,000 probably closer to $5,000. I hand her two more notes: “The client at my window has a gun. Keep your mouth shut. Pass all the cash from your drawer to your right.” The other is identical, for the other direction. I indicate that she should pass them the notes, which she does. Handfuls of cash are passed from one to another between five tellers, under the glass to me, where I drop them into my bag. Other customers look on, and I guess they figure something odd is happening, but of course no-one says anything.

Start to finish, I’m there putting money into my bag for less than two minutes. Up until the point the first cash hit my bag, I had a pretty good story: some random person on the internet put me up to it, got me to do all this weird shit, watch others do weird random stuff, paid me through paypal. With the money in my bag though, the game has changed. I know the silent alarm was probably tripped in the first thirty seconds. With your typical mid-morning midtown traffic chaos and the cops already dispatched to at least half a dozen banks apparently being robbed simultaneously, I figure they may not get here for twenty minutes. I’m not counting on the delay any more than I have to, but I figure my margin for error is pretty healthy.

I walk out the front door. Fifteen steps from the bank entrance is a convenient manhole. And look… the cover is already off. The hole is surrounded by four orange cones connected by official-looking Men At Work tape. Of course this is my doing: started and finished a few minutes before entering the bank. From my bag I take a hard hat, some old gloves, and an orange reflective sleeveless vest. Standard issue street crew garb, bought for a few bucks here and there at thrift stores. I pick up the steel tool used for removing manhole covers and drop it into the hole. The rest of my props I leave for the cops. I climb down into the hole and pull the cover across on top of me. Although the covers are heavy, putting one in place is actually a surprisingly easy job when done from below.

The sewers of Manhattan are pretty ugly, despite large-scale efforts to sanitize and modernize them in the 90s. But there’s no need to stay down for long in order to simply disappear. I’ve walked this route before a few times. It’s no big deal. How do I know all this? The plans are on the net: obtained and published online by some urban spelunking group. Along the way I remove the red shirt and put it in my bag. A little over six minutes later I emerge from another manhole in a lane behind a hotel. A couple of suits are standing outside having a smoke. Nothing if not brazen, I bludge a cigarette and a light from one of them. He’s only too happy to help a working man in a hard hat. Show his mates he’s on good terms with blue collar America. With a false mustache, non-prescription glasses, a vest, and the hard hat, there’s not going to be much to go on later, even if these guys do wind up talking to the cops. Besides, as is well known, witnesses are wonderfully unreliable, so the more the merrier as far as I’m concerned. You’ve got to know how these things work and use them to your advantage. I tell the suits I’m on a break from the sewer, got to go to my bank, and off I slouch. Around the first corner I remove the working gear and it goes into my bag too. By my reckoning, the cops wont have even gone down the manhole outside the bank. At most they’ll have a call in for someone to remove the cover. They’ll also know that I’m probably no longer underground.

I walk down into the subway and head home to count my cash ($47,000 and change) and to catch the headline news: Daring daylight robbery! Criminal mastermind! Nine (only nine) banks used to create a distraction while one was well and truly robbed. The police chief is interviewed, indignant: we’ll hunt these men down and put them where they belong. A sophisticated operation like this isn’t done in a vacuum.

Well, good luck chief. For me this is just the beginning, just a proof of concept really. No-one should have to rob banks for a living. That kind of crime doesn’t pay well enough, even if you never get caught. Of course, I have other plans. As the world becomes more digital, less messy, things will only get easier for those who have the balls.