Because it’s emacs, it wasn’t hard to write a function to display a buffer mode histogram. Here’s mine:

235 buffers open, in 23 distinct modes

91               python +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
47          fundamental +++++++++++++++++++++++++++++++++++++++++++++++
24                  js2 ++++++++++++++++++++++++
21                dired +++++++++++++++++++++
16                 html ++++++++++++++++
 7                 text +++++++
 4                 help ++++
 4           emacs-lisp ++++
 3                   sh +++
 3       makefile-gmake +++
 2          compilation ++
 2                  css ++
 1          Buffer-menu +
 1                 mail +
 1                 grep +
 1      completion-list +
 1                   vm +
 1                  org +
 1               comint +
 1              apropos +
 1                 Info +
 1           vm-summary +
 1      vm-presentation +

Tempting as it is, I’m not going to go on about the heady delights of having a fully programmable editor. You either already know, or you can just drool in slack-jawed wonder.

Unfortunately I’m a terrible emacs lisp programmer. I can barely remember a thing each time I use it. But the interpreter is of course just emacs itself and the elisp documentation is in emacs, so it’s a really fun environment to develop in. And because emacs lisp has a ton of support for doing things to itself, code that acts on emacs and your own editing session or buffers is often very succinct. See for example the save-excursion and with-output-to-temp-buffer functions below.

Various things about the formatting could be improved. E.g., not use fixed-width fields for the count and the mode names, and make the + signs indicate more than one buffer mode when there are many.

Below is a section I just ran across that I imagine will resonate strongly with the people involved in Occupy Wall Street. I’ve long been fascinated to watch how power tries to maintain itself by attempting to enforce isolation and to restrict information flow, and, on the contrary, how increased information flow between the subjects of power naturally undermines this basis. Awareness of these opposing forces, even if not explicitly understood, is what I think accounts for the tenacity and ferocity on both sides of the OWS (and many other) movements, even (especially) when the movements are still only tiny. The occupiers experience the surge of energy and determination and self-identification that comes from solidarity, while those in power recognize the danger and act in heavy-handed ways to try to crush it, usually after trying to ignore and then ridicule. The consistent characteristic of the reaction against these movements, as Steinbeck notes, is that those in power do not understand what’s going on. So in their efforts to snuff out the protests they instead fan the flames, which they then have to react even more violently to. It seems an extraordinarily difficult task for power to successfully manage to defuse a popular movement without resorting to extremes. Hence the absurd justifications of needing to clean (often already cleaned – by the protesters) public spaces, to make the public spaces once again available to the public, etc. Disperse, ridicule, isolate. If the gentle pretenses do not work, then we’ll do what we can to get rid of or evade the media (in all its forms), and then come in and beat the shit out of you.

So for all those out there in the OWS camps around the world (don’t forget there were protests in almost one thousand cities worldwide), and especially for those in the US, here’s some beautiful Steinbeck:

One man, one family driven from the land; this rusty car creaking along the highway to the West. I lost my land, a single tractor took my land. I’m alone and I am bewildered. In the night one family camps in a ditch and other family pulls in and the tents come out. The two men squat on their hams and the women and children listen. Here’s the node, you who hate change and fear revolution. Keep these two squatting men apart; make them hate, fear, suspect each other. Here is the anlage of the thing you fear. This is the zygote. For here “I lost my land” is changed; a cell is split and from its splitting grows the thing you hate — “we lost our land.” The danger is here, for two men are not as lonely and perplexed as one. And from his first “we” there grows a still more dangerous thing; “I have a little food” plus “I have none”. If from this problem the sum is “we have a little food”, the thing is on its way, the movement has direction. Only a little multiplication now, and this land, this tractor are ours. The two-men squatting in a ditch, the little fire, the side-meat stewing in a single pot, the silent, stone-eyed women; behind, the children listening with their souls to words their minds do not understand. The night draws down. The baby has a cold. Here, take this blanket. It’s wool. It was my mothers blanket — take it for the baby. This is the thing to bomb. This is the beginning — from “I” to “we”.

If you who own the things people must have could understand this, you might preserve yourself. If you could separate causes from results, if you could know that Paine, Marx, Jefferson, Lenin were results, not causes, you might survive. But that you cannot know. For the quality of owning freezes you forever into “I”, and cuts you off forever from the “we”.

The Western states are nervous under the beginning change. Need is the stimulus to concept, concept to action. A half-million people moving over the country; one million more restive, ready to move; 10 million more feeling the first nervousness.

And tractors turning the multiple furrows in the vacant land.

• Cheap ironing board
• Braun iron
• Vacuum cleaner
• Dell DN1815 multi-function networked laser printer (black & white). Fax, copy, scan, print. 5 years old, works great.
• 20" Miyata deluxe (48 spoke) unicycle
• 26" Semcycle unicycle
• 6 Renegade juggling clubs
• Bag of about 15 silicone juggling balls
• 2 minitorre computers (from about 2002) without hard drives
• 3 Ikea CD shelves (each holds about 200 CDs)
• 7 60cm wide x 2.5m tall white Ikea (Billy) bookshelves
• 1 40cm wide x 2.5m tall white Ikea (Billy) bookshelf
• 1 30cm wide x 2.5m tall white Ikea (Billy) bookshelf
• 19" CRT monitor
• 2 100Mbit ethernet hubs (5 port, 8 port)
• 5 cable modems (DLink, Cisco, 3Com)
• 2 Siemens Gigaset AS29H DECT phones, like new, white
• White wooden Ikea TV/DVD table
• Massive (3m by 1.2m) wall-mounted whiteboard
• White Ikea filing cabinet (2 wide roll-out shelves)
• Green wooden 6-drawer small rolling shelves
• DVD player with sub-woofer & 5 external speakers
• Sony CD player with sub-woofer & 2 external speakers
• Panasonic NVGS230 hand-held video recorder, perfect condition
• K2 rollerblades 6000 series, good condition, size 41/42
• 5-wheel speed skates, size 41/42
• Philips toaster
• Large wooden Ikea cutting board
• Kettle
• Electric juice extractor
• Hand-held electric blender
• Barcelona apartment floor tiles. I have about 20 that I’ve accumulated over the years.

Image: Villa San Michele

In 1928, Axel Munthe, a Swedish physician living on the isle of Capri, published The Story of San Michele. Munthe’s villa on the slopes of Mount Barbarossa stands on a site chosen almost two thousand years earlier by the emperor Tiberius, who from tiny Capri held sway over the entire Roman empire. Extraordinarily beautiful, the island passed at various times through the hands of the Greeks, the Romans (Caesar Augustus was captivated), the Dutchy of Naples, the Saracens, the Longobards, the Normans, the Angevins, the Aragonese, the Spanish, and the Bourbons.

On completing his medical studies, Munthe was the youngest physician in Europe. The Story of San Michele describes his time in Paris and Rome, his years as the physician to the Swedish Royal family and later his years as private physician to the queen of Sweden, who had also taken a liking to Capri. Written in English, The Story of San Michele, which remains in print, was an instant success, becoming the best-selling non-fiction book in the U.S. in 1930. Munthe’s novel approach to medicine and the book’s mixture of adventure, treasure, and royalty continue to inspire. The Story of San Michele was the mysterious target of one Henry Arthur Harrington, a petty thief who crisscrossed the UK, stealing 1,321 copies from second-hand bookstores before his eventual arrest in 1982. Even in 2003, Munthe’s contributions are the subject of learned attention: the Second International Symposium on Axel Munthe’s life and work will be held in Sweden tomorrow (September 13).

With the rapid success of The Story of San Michele, the book was a natural target for would-be translators. Editions in several languages were soon completed. Given its origin, it was odd that such a popular book was not more quickly translated into Italian.

Patricia Volterra, living in Florence, was fascinated by the book and was eager for her husband Gualti to read it too. A minor obstacle: Gualti did not speak English. Undeterred, Volterra set out to translate the book into Italian. She wrote to John Murray, the publisher, requesting permission. To her surprise, she received a reply directly from Munthe. According to her diary, he wrote that

the book had already been translated into several languages and was selling like wildfire. To date he had refused offers for it to be translated into Italian as, he wrote, this language, when written, was apt to become too flowery and overloaded and that he had written the book in an extremely simple style which he wished to retain. However, he continued, he suggested I should translate the last chapter, which he considered the most difficult, and send it to him to the Torre Matterita at Anacapri. He would then let me know whether he thought he could permit me to translate the rest.

Volterra sent off her translation of the final chapter and spent several weeks waiting for an answer. Finally her manuscript was returned “with an extremely complimentary letter from Munthe, telling me to proceed to do the rest.” Later she wrote that at that time nothing seemed impossible to her but that now she wouldn’t have even considered the translation.

While working on the translation, she had lunch with Munthe in Rome when Gualti, an Italian concert pianist, was playing at the Augusteum. Munthe was staying at Villa Svezia, the Queen of Sweden’s residence on the Via Aldovrandi. When Munthe saw her he exclaimed ‘My goodness, how old are you?’ She: ‘Twenty three.’ He: ‘And you are translating San Michele!’ Munthe was over 70 at the time.

Volterra sent the work to an Italian publisher, Mondadori, who refused her. “Their great loss,” she wrote. Another, Treves, accepted. Munthe “had decreed that the entire royalties should go to the Society for the Protection of Animals in Naples.” Volterra was to sell her translation for whatever she could get for it. This amounted to the equivalent of 50 pounds sterling for 8 months work.

Later that spring, Volterra traveled to Capri. In a horse-drawn cab they drove to Anacapri where they visited San Michele. From there on foot through the olives to the Torre di Materita to have lunch with Munthe. A variety of his dogs scampered round his heels as he showed them the old tower which was then his home. They had a vegetarian lunch served by Rosina, so affectionately mentioned by Munthe in his book.

The Volterra translation ran quickly into 35 editions and was still selling well when she left Italy in 1938. Mussolini was so impressed by La Storia di San Michele that he passed a law prohibiting the shooting of migratory birds on Capri.

Volterra saw Munthe one final time, in Jermyn Street, London. Munthe died in 1949, leaving the villa of San Michele to Sweden. Owned today by the Swedish Munthe Foundation, it is home to an ornithological research center and is open to the public.

[Continued in part two, "Bob Arno".]

Image: ABC Tasmania

For the last seven years, I’ve kept a web page full of people’s email about street scams they’ve been involved in (as victims) in Barcelona.

In the beginning I just wrote down brief descriptions of things that I saw or was involved in soon after moving to Spain. I’d seen hardly any street crime in my (then) 33 years and I found it fascinating to watch for. It certainly wasn’t hard to find. Often it came right to my door or to the street under my balcony. Before long I began to receive email from others who had visited or lived in Barcelona, each with their own story to tell. I put the stories onto the web page and they soon outnumbered my own. I continue to receive a few emails a month from people who’ve read the web page (generally after being robbed, though sometimes before leaving on a trip). I don’t often reply to these emails, apart from a line or two to say thanks when I put their messages on the web page, often months after they mail me.

For whatever reason, I’ve never been very interested to meet these people, though I’ve had plenty of chances to. In general I don’t seem to have much interest in meeting new people – it’s quite rare that I do. I should probably be more sociable (or something) because once in a while the consequences are immediately extraordinary.

Among my email, I get occasional contacts from people in the tourism industry. Lonely Planet, Fodor’s, people writing books or running travel services or web sites. Mainly they want to know if they can link to the web page, or to use some of the content in their own guides. I always agree without condition. After all, the main (but not the only) point is to help people be more aware, and besides, the majority of the content was written by other people who clearly share the same advisory aim. With this attention from various professionals who are trying to pass on the information, I began to wonder how many such people there were. Maybe there were other people with web sites devoted to street crime. So once in a while I’d do a web search on “street scams”, or something similar, just to see what came up. It’s usually interesting.

On July 30th 2001, I went looking around for similar web sites and ran across Bob Arno. I took a quick look around and fired off an email to say hello, and offered to buy him a beer the next time he was in Barcelona:

    Hi Bob

    I was just having a wander around the web when I ran into your
    pages about pickpockets. They look good, very useful.

    You might be interested to see a page of my own: http://jon.es/barna/scams.html

    All about things that have happened to people in Barcelona. It's
    not too well organized, but there's a lots of it. Most of it falls
    into well known classes of petty crime. Things are getting worse
    here, with the most recent tactics being strangulation from behind
    and squirting a flammable liquid onto people's backs and then, you
    guessed it, setting them on fire.

    Let me know next time you're in Barcelona and I'll buy you a
    beer. I'm also in Manhattan very often.

    Regards,
    Terry Jones.

Bob looked very interesting, and we seemed to have the same point of view on street crime. He’s a seasoned professional, a Vegas showman, and is constantly traveling the world studying many forms of crime and passing on his knowledge. Check out his website.

I sent mail to Derek, passing on Bob Arno’s URL. I said a little of how funny and random it seemed to me, of how over all the years of doing different things and meeting any number of famous and high-powered academics and intellectuals etc., and not really having much interest in any of them, that I’m sending email to this Bob Arno guy suggesting we meet up.

The next day I read more about Bob’s exploits and interests and I guessed that we would probably get on really well. I sent off a longer email with some more of my observations about Barcelona:

    Hi again.

    I sent off that first email without having looked at more than a
    page or two of your web site.

    It's very interesting to read more. I spend far too much time
    thinking about and watching for petty thieves in Barcelona. I've
    thought about many of the issues touched on in the interview with
    you by your own TSJ. The whole thing is very intriguing and lately
    I've begun to wonder increasingly what I can do about it, and if I
    want to do anything about it. I have tended to act to try to stop
    pickpockets, but I've also seen things many times from a distance
    or a height, read many things, seen freshly robbed people weeping,
    talked to many people who have been robbed, thought of this as an
    art (I'm interviewed in a Barcelona newspaper under the headline
    "Some crimes are a work of art" - I'm not sure if they understood
    what I meant), etc. I've never tried filming these people. But I
    know how they look at you when they know they have been spotted,
    how their faces look when the wallet hits the floor, how they prey
    on Western or "rich" psychology, and so many other things.  My
    focus has been Barcelona, after coming to live here 5 years ago
    and (at that time) having an apartment 1 floor up about 100 meters
    from Plaza Real. If I had had a net I could have caught people
    several times a day.

    I recently got a video camera and was thinking of interviewing the
    woman on my web site who was strangled here earlier this month. By
    the way, the papers reported up to 9 cases of such stranglings in a
    single day. I wasn't quite sure what to do with the tape. It hadn't
    occurred to me to film the thieves, but it would be so easy.  In
    Barcelona it's trivial to spot these people, and also feels very
    safe since many of them have been arrested literally hundreds of
    times.  There is basically no deterrent. There are undoubtedly more
    sophisticated pickpockets here too, but there is little in the way
    of evolutionary pressure to make them improve their methods. The
    tourists are too many and too unaware, the police are too few, and
    the laws are too slack. Why would you even bother to improve or
    think?

    I also know the boredom that comes with professional acts. I used to
    do a lot of juggling and unicycling, practicing 6 hours a day for a
    long time. But I could never stand to have a canned show that I did
    time after time - it was just too routine to have a routine. So I
    refused and eventually drifted into other things.

    How can I get a copy of your book? It doesn't seem to say on the web
    site. Also, the menu of links at the top left of your pages looks
    extremely garbled under my browser (Opera).

    Terry

As it turned out, my timing was perfect. I got a mail back the next day from Bob’s wife Bambi (yes, really). She said they’d be in Barcelona in just 5 days time and that they’d love to meet up.

And meet up we did!

They came to our apartment and we all hit it off immediately. As I’d thought, we did have a lot in common, both in terms of what we had done and in outlook. They told me they also get lots of email through their web site and hardly ever reply. Ana and I took them out for food. We sat outside at the nearby Textile Museum. Later, Ana went home to look after Sofia, and I stayed with Bob and Bambi. In the end I was with them about five hours and I had a really good time. We arranged to meet the next day to go hunting for thieves on the Ramblas. In one sense, “hunting” isn’t at all the right word: the thieves are typically very obvious to anyone who’s actually paying attention. But there’s a lot of subtlety in tracking and filming them, so it really is something like a hunt. I’ve since spent many hours, on several occasions, in action with Bob and Bambi in Barcelona. But that’s another story.

After getting home that first night, I went back to Bob’s web site and read more of his pages. He’s had a pretty colorful life. Actually, it’s extraordinarily colorful by almost any measure. “Who is this Bob Arno?” I wondered. Fortunately, Bob has a “Who is Bob Arno?” page, which I finally got around to reading.

Halfway down… unbelievable… I want to cry.

    Born in Sweden, Bob Arno is a great-grandson of Dr. Axel Munthe,
    who is most famous for his novel The Story of San Michele.

Patricia Volterra was my great aunt.

While thinking about Deferreds in the days before the talk, I think I put my finger on one of the reasons I find Deferreds so attractive and elegant. The thought can be neatly summarized: Deferreds let you replace synchronous data structures with elegant asynchronous ones.

There are various ways that people get exposed to thinking about programming in an asynchronous style. The great majority do so via building user interfaces, usually with widget libraries wherein one specifies widgets and layouts, sets up event handlers, and then surrenders control to a main loop that thereafter routes events to handlers. Exposure to asynchronous programming via building UIs is becoming much more common as Javascript programmers build client-side web apps that operate asynchronously with web servers (and the “A” in AJAX of course stands for asynchronous). Others become aware of asynchronous programming via writing network code (perhaps using Twisted). Relatively few become aware of asynchronous programming via doing async filesystem I/O.

Because Twisted’s Deferreds don’t have anything to do with UIs or networking or filesystems, you can use them to implement other asynchronous things, like an asynchronous data structure. To show you what I mean, here’s a slightly simplified version of Twisted’s DeferredQueue class, taken from twisted/internet/defer.py:

I find this extremely elegant, and I’m going to explain why.

But first, think about code for a regular synchronous queue. What happens if you call get on a regular queue that’s empty? Almost certainly one of two things: you’ll get some kind of QueueEmpty error, or else your code will block until some other code puts something into the queue. I.e., you either get a synchronous error or you get a synchronous non-empty response.

If you look at the get method in the code above, you’ll see that if the queue is empty (i.e., the pending list is empty), a new Deferred is made, added to self.waiting, and is immediately returned to the caller. So code calling get on an empty queue doesn’t get an error and doesn’t block, it always gets a result back essentially immediately. How can you get a result from an empty queue? Easy: the result is a Deferred. And because we’re in the asynchronous world, you just attach callbacks (like event handlers in the UI world) to the Deferred, and go on your merry way.

If you can follow that thinking, the rest of the code in the class above should be easy to grok. In put, if there are any outstanding Deferreds (i.e., earlier callers who hit an empty queue and got a Deferred back), the incoming object is given to the first of these by passing it to the callback function of the Deferred (and popping it out of the waiting list). If there are no outstanding Deferreds expecting a result, the incoming object is simply appended to self.pending. On the get side, if the queue (i.e., self.pending) is non-empty, the code creates a Deferred that has already been fired (using the succeed helper function) and returns it.

By now, this kind of code seems routine and straightforward to me. But it certainly wasn’t always that way. So if the above seems cryptic or abstract, I encourage you to think it over, maybe write some code, ask questions below, etc. To my mind, these kinds of constructs – simple, elegant, robust, practical, obviously(?) bug-free, single-threaded, etc. – are extremely instructive. You too can solve potentially gnarly (at least in the threaded world) networking coding challenges in very simple ways just by building code out of simple Twisted Deferreds (there’s nothing special about the DeferredQueue class, it’s just built using primitive Deferreds).

For extra points, think about the concept of asynchronous data structures. I find that concept very attractive, and putting my finger on it helped me to see part of the reason why I think Deferreds are so great. (In fact, as you can see from the use of succeed, Deferreds are not specifically asynchronous or synchronous: they’re so simple they don’t even care, and as a consumer of Deferreds, your code doesn’t have to either). That’s all remarkably elegant.

I sometimes hope O’Reilly will publish a second volume of Beautiful Code and ask me to write a chapter. I know exactly what I’d write about :-)

The new approach is simply to divide the number of hours that have passed since In Search of Lost Time was published by the number of people who’ve ever finished it. That average is a crude measure, but it may be nevertheless quite accurate and it’s irresistibly interesting to me to see how it compares to my original 2006 estimate of 2.19 hours.

So, assume 2B people were alive in 1927 when the final volume was published, and 6.4B alive at the end of 2006 (source).

Assume that no-one alive in 1927 was still alive in 2006 (obviously not the case, but not unreasonable and not a significant error). I.e., there were 4.4B births in those 79 years. Note: This is ignoring a significant number of people who were born after 1927 and who died before 2006. But it is including everyone born from 1990 onwards, essentially zero of whom would have read Proust by 2006.

In my original post I estimated that one person in 10K actually finishes the whole book. So that’s 4.4B/10K = 440K people who read the book during the 79 years.

79 years is 28,835 days, or 692,040 hours. Doing the division, 692,040 / 440,000 = 1.57 hours.

I.e., by the above rough reasoning, someone, somewhere on earth, finishes Proust every 1.57 hours, on average.

I find the closeness of the two estimates quite remarkable. There’s only one shared assumption (1 in 10,000 finishes). Both estimates are quite crude, yet there’s only about a 30% difference in the answers. I was expecting them to be much more divergent.

We’re all exhausted and thrilled to have FluidDB out the door. I wont try to describe the feelings, except to say that it’s all incredibly exciting, and that I haven’t been getting much sleep recently. The reaction in the programmer community has been astounding: there are 9 client-side libraries already written (with more on the way), there are tools, there’s a FluidDB Explorer, and little apps are now starting to pop up. We couldn’t be happier. You can see a list of those things here.

To find out more about FluidDB, here are your best choices:

• The FluidDB blog. Recent posts describing FluidDB are:
  • Truly social data.
  • Cloud storage: the real opportunity.
  • Kaleidoscope: 10 takes on FluidDB.
• The FluidDB high-level overview goes into the information model of FluidDB, its permissions system, and the query language.
• The FluidDB-discuss mailing list is a great place to ask questions, and has already seen some detailed discussion on spam and the FluidDB about tag.
• If you’re a programmer looking to learn about FluidDB, the #1 thing you should do is join the #fluidDB IRC channel on irc.freenode.net. There are many people there learning about FluidDB, exchanging tips, and writing new libraries, tools, and apps. You can also read the API documentation.

Thanks for reading along! The real journey is probably only just beginning…

I could have added the subtitle “How someone who knows nothing about databases wound up in a project to build a database.”

I would like again to emphasize that I’m not an authority and I’m not trying to pass myself off as one.

I’ve already been accused of deliberate fear-mongering. That’s the opposite of my purpose. On the contrary, it’s important to stay calm and there are good reasons for doing so. If you don’t want to know a bit of history and to have some sense of things that have happened in previous pandemics, then you don’t have to read what follows. There’s no harm in staying calm via not knowing. On the other hand, there is harm in being gripped by fear due to ignorance.

If you do read, try to keep in mind that the main point here is that you shouldn’t be overwhelmed by fear or begin to panic. There’s no reason to. Plus it will only make things worse.

On the subject of being an authority and fear-mongering, after I wrote up the first set of thoughts I was invited to be a guest commentator on a radio show. I declined. It feels very irresponsible to say anything about influenza given that I am not an expert and don’t even work in the field anymore, but OTOH it feels irresponsible to remain totally silent given that I know at least some historical things fairly well.

To illustrate the conflict: On April 26 it seemed crystal clear to me that the virus was going worldwide. You only had to have seen the Google map that I twittered about the day before to see that it was going to be all over the place in days. But I didn’t want to point that out, and when I was asked I told the asker to be his own judge. I linked to a map showing Mexico and the US earlier and said “hopefully we wont have to zoom out” – trying to get people to consider that we would probably soon have to zoom out.

So I think I’ve been quite restrained. This post is also restrained. As I said above, there are good reasons not to sensationalize things or to create the impression that people should panic.

So here we go again, a few more thoughts as they come to mind. These are things that I find interesting, with a few scattered opinions (all of which are just guesses). There’s no real structure to this post.

The WHO have announced today that we’re officially in a pandemic. That doesn’t really mean much, but it’s good to have a candid and early declaration – part of the problem historically has been slowness to even admit there’s a problem. The WHO didn’t even exist in 1918.

In case you don’t know, there’s pretty good evidence that humans have been fighting influenza for thousands of years.

The most interesting thing to me in reading about the 1918 pandemic is the social impact of the disease.

One thing to make clear is that the current pandemic is not the 1918 pandemic. I tend to agree with those who say that a pandemic of that nature could not take place today – but note that people, perhaps especially scientists – would have said that at all times prior to and after 1918. We often under-estimate the forces of nature and over-estimate our own knowledge and level of control.

BTW, something like 75% of people who died during World War 1 did so because of the flu pandemic, which didn’t really take off until November of 1918. Amazing.

As I mentioned in my earlier post, under normal circumstances (even in a pandemic), flu doesn’t kill you. It leaves you susceptible to opportunistic follow-on disease. The good news is that we are vastly more informed now than we were in 1918 about the nature of infectious diseases. For example, we know a lot about pneumonia, which we did not in 1918. See the moving story of the amazing Oswald Avery, who dedicated his life to the disease and along the way fingered DNA as the vehicle of genetic inheritance – and never won a Nobel prize.

So the care of people who have been struck down by flu is going to be much more informed this time around. And it will probably be better in practice too. I put it that way because odd societal things happen in a pandemic. I hesitate to go into detail, because some people will assume that things that happened way back when will necessarily happen again this time around.

One of those things is that medical systems get overrun by the sick. Plus, doctors and nurses understandably decide that their jobs have become too dangerous and they stop showing up for work. So there can be a sharp drop-off in the availability of medical help.

So much so that there were reports of doctors and nurses being held hostage in houses in 1918. I.e., if you could get a doctor to visit to attend to your family, the situation was so dire you might consider pulling a gun on him/her and suggesting they make themselves comfortable for the duration.

The problem is not so much that many people are dying, it’s that a much larger number are simultaneously extremely ill and that panic grips them and everyone else. Roughly 30% of all people caught the 1918 flu. I have another post I may write up on that. Normal (epidemic) flu catches 10-15% of people in any give year.

Many of our systems are engineered to provide just-in-time resources, to cut the fat in order to maximize profitability, etc. That means that we’re closer to collapse that would seem apparent. How many days of fresh food are there on hand in a major city?

None of this is meant to be alarmist. But the reality is that alarming things have happened in the past.

Most interesting and revealing to me is that our cherished notions of politeness, of our generosity, our goodwill towards our neighbors, etc., can all go out the window pretty quickly. I’ve long held that all those things are the merest veneer on our underlying biological / evolutionary reality. We’re very fond of the ideas that we’re somehow no longer primates, that we’re not really the product of billions of years of evolutionary history, that somehow the last centuries of vaunted rationality have put paid to all those primitive lower impulses. I think that’s completely wrong. Behavior during a full-scale pandemic is one of the things that makes that very clear.

In a pandemic, if things get ugly, you can expect to see all manner of anti-social behavior. If you read John Barry’s book The Great Influenza or Crosby’s America’s Forgotten Pandemic you’ll get some graphic illustrations.

If I had a supply of tamiflu (which I don’t), I wouldn’t tell anyone. That’s deliberately anti-social. Ask yourself: What would you do if you had kids who were still healthy and your neighbor called you in desperation to tell you that his/her kids seemed to have come down with influenza? Get out your family’s tamiflu supply and hand it over? Lie? What if they knew you had it and you refused to give it or share? What if your neighbor’s kid died and yours never even got the flu? What kind of relationship would be left after the pandemic had passed?

This may all seem a bit extreme and deliberately provocative of me, and yet those sorts of dilemmas (sans tamiflu, naturally) were commonplace in 1918. As you might expect, they don’t always get resolved in ways that accord well with our preferred beliefs about our own natures in easier times. Crosby speculates that the reason the pandemic of 1918 is “forgotten” is due largely to the fact that it coincided with the war, and that people were generally exhausted and dispirited and wanting to move on. I’d speculate further that people en masse frequently behaved in ways that they weren’t proud of, and wanted to forget about it and act as if it hadn’t happened ASAP. That’s just a guess, of course.

In any case, if there’s a full-blown pandemic, societal structures that we take for granted are going to be hugely transformed. Medical services, emergency services, food supply, child care and education, job absenteeism, large numbers of the people who would normally be in charge of things coming down sick and being unable to do their normal jobs, etc. All sorts of things are impacted and lots of them are interconnected. The system breaks down in many unanticipated ways as all sorts of things that “could never happen” are all happening at once.

You might think I’m fear-mongering here, but I’m not. In fact I’m refraining from going into detail. Go read John Barry’s book, or any of the others, and see for yourself.

The important thing to remember in all this is that we are no longer in 1918. BTW, there were also influenza pandemics in 1957 (killing just a couple of million people) and 1968 (killing a mere million).

Apart from the fact that we’ve advanced hugely in medical terms, we are also much better connected. I can sit in the safety of my apartment in Barcelona and broadcast calming information like this blog post to thousands of people. We are better informed. We know that panic and fear greatly compound the impact of a pandemic. They feed on one another and prolong the systemic societal collapse. Because we can communicate so easily via the internet – provided our ISPs stay online – we can help keep each other calm. That’s an important advantage.

So my guess is that this wave isn’t going to be so bad, certainly not in terms of mortality. One thing to keep in mind though is that the virus isn’t going away. It will likely enjoy the Southern hemisphere winter, and we’ll see it again next Northern hemisphere winter. And yes, those are guesses. Because influenza is a single-stranded RNA virus it mutates rapidly (you don’t get the copy protection of a double strand). So this is the beginning, not the end, even if the pandemic fizzles out in the short term. It will be back – probably in less virulent form – but by then we’ll also have a good leg up on potential vaccines, and we’ll also know it’s coming.

OK, I’ll stop there for now. I have tons of other things I could write now that I’m warmed up. You can follow me on Twitter if you like, though I doubt I’ll be saying much about influenza.

If you truly believe I’m fear mongering, please send me an email or leave your email address in the comments. I’ll send you mail with some truly shocking and frightening stuff, or maybe fax you a few pages from some books. Believe me, it gets a lot nastier than anything I’ve described. Things are not that bad, certainly not yet. We’re not in 1918 anymore

So, stay calm, and do the simple things to keep yourself relatively safe. If everyone follows instructions like those, the virus wont have a chance to spread the way it could otherwise. That may sound like pat concluding advice, but there’s actually a lot to it – the epidemiology of infectious disease – in part the mathematics of infection – can be hugely altered depending on the behavior the typical individual. Following basic hygiene and getting your kids to too will make a big difference. There’s no denying that this is going to get worse before it gets better, but we can each do our part to minimize its opportunities.

I am both unqualified and qualified to make a few comments. I’m unqualified because I no longer work on influenza virus, because I’m not a virologist, because I have no inside information at all about the current outbreak. OTOH, I have some claim to know what I’m talking about. I worked on influenza virus as part of the Antigenic Cartography team at the University of Cambridge for a few years. We helped the WHO choose the H3N2 strain for the human vaccine. I’ve met the heads of the 4 international flu centers and even been in the WHO Situation Room in Geneva – a self-contained underground fortress. I spent a lot of time hanging out and talking to influenza virologists, many from the Erasmus Medical Centre in Rotterdam. I was even an author on a Science paper on the global spread of epidemic influenza. Plus I’ve read all the books on the 1918 pandemic, which gives some (largely retrospective) insight into what happened back then, and perhaps some insight into what could be about to happen.

I also feel it’s good for someone like me to comment because I’m outside the flu world and the people inside it will be unlikely to say much. Flu is a highly political issue, to put it mildly. People working in the flu research community will be reluctant to speak up. So I should make it very clear that the comments below are just my opinions, and don’t represent anyone else’s thoughts.

I’ll try to just make a few points that I think are fairly sober – neither alarmist, nor dismissive – and to keep speculation out of it.

Apart from the details of the actual virus, the social side of a potential pandemic is extraordinarily interesting. Very few people will have really concrete information, and those that do will still only be making their best guesses.

In a pandemic, or something that looks like it might be one, wild rumors sweep through the population. That will happen on an unprecedented scale this time round.

The virus has, as far as we know, not spent much time in humans yet. Once it does, it will begin to adapt itself in unpredictable ways. It may become more virulent, or less virulent. It may develop resistance to the antivirals that are currently effective. Antiviral resistance has been a topic of great concern for at least a couple of years. The current virus is already known to be resistant to both amantadine and rimantadine, though oseltamivir is still effective.

If you ask virologists what the probability is that there will be another pandemic, they’ll tell you it’s 1.0. It’s just a matter of time until it happens. it’s like a non-zero probability state in a Markov process. When it does happen, what you do in the first phase is critically important. In the case of the avian influenza they would try to immediately cull all potentially infected birds, to stop the virus spreading and mutating and becoming more likely to enter the human population. When it did get into the human population, there would be swift action to isolate it, again to reduce the spread and the time the virus has to adapt. In the case of the avian influenzas in humans, there has been very little airborne transmission, and we’re lucky for that. But the current virus seems to already have that property, which is of great concern.

It would be a miracle if the current epidemic vaccine provided any protection against this virus. The human vaccine does contain a strain against H1N1, but that’s a strain picked based on sampled human viruses from many months ago. The epidemic vaccine is aimed at thwarting what’s known as antigenic drift – the relatively slow accumulation of point mutations in the virus. Pandemic strains arise through antigenic shift in which large chunks of viral genetic material, sometimes whole genes, are mixed between influenza viruses from different species. In a pandemic strain some of the genetic material and the proteins it expresses will very likely never have been seen by a human immune system.

The current WHO standard influenza test kit is not very useful in identifying this strain. They have issued instructions warning against false negatives.

Some aspects of the current outbreak are, to my mind, cause for great concern.

The acting-director of the CDC has already said: “There are things that we see that suggest that containment is not very likely.” That is a remarkably candid statement. I think it’s very clear that the cat is out of the bag. The question is how bad is it going to be. That’s impossible to tell right now, because we do not know what the virus will look like in the future, after it has had time to mutate and adapt inside humans.

In normal circumstances it takes about 6 months to make the world’s supply of epidemic vaccine. It’s a long and difficult process requiring tons of virus to be grown in chicken eggs. A canidate vaccine strain has to be identified, it has to be one that grows well in the chicken egg (including not killing the chick). Even under the high pressure of a potential pandemic, making a new vaccine is going to take months. By then the virus may have moved on (via mutation) and the vaccine’s efficacy may be less. Note that the 1918 virus killed tens of millions of people over a period much shorter than this.

Diverting the world’s influenza resources to covering a pandemic threat necessarily diverts them from work on epidemic vaccines. Epidemic flu kills roughly 0.5M people a year as it is. Not being able to pay due attention to the epidemic strains is also a bad thing.

The new virus has been popping up in various places in the US in the last days. I expect it will go global in the next couple of days, maximum. What’s to stop it? The virus has been isolated in several diverse areas and in many cases is genetically identical. The 1918 virus also popped up, in many cases inexplicably, across the US. The book America’s Forgotten Pandemic is worth a read.

There were 3 waves of the 1918/19 pandemic. The first was in summer of 1918 – very unusual, as influenza normally falls to extremely low rates during summer. Note that the current outbreak is also highly unseasonal.

The 1918 pandemic killed with a very unusual age pattern. Instead of peaks in just the very young and the very old, there was a W shape, with a huge number of young and healthy people who would not normally die from influenza. There are various conjectures as to the cause of this. The current virus is also killing young and healthy adults.

The social breakdown in a pandemic is extraordinary. If you read The Great Pandemic by John Barry, you’ll get some sense of it. America’s Forgotten Pandemic also helps give some idea of what it must have been like.

No-one knows just how many people died as a result of the 1918 pandemic. Estimates generally range between 40M and 100M, and have trended upwards over the years. Influenza is not the easiest to diagnose (hence the category ILI – influenza-like illnesses). It also strips the throat of protective epithelial cells, leaving you susceptible to opportunistic follow-on infections, such as pneumonia, which often do the killing.

No-one knows how bad another pandemic might be in terms of mortality. Low estimates are in the single digit millions. Someone from the WHO suggested a significantly higher number about 4 years ago in the context of avian influenza and that number was quickly retracted. Jeff Taubenberger, who was responsible for resurrecting and sequencing the 1918 virus (an extraordinary story, related in a couple of books) has published work saying 100M might be possible. No-one knows, and it depends on many factors, including the characteristics of the virus, how early it is detected, how easily it spreads, how virulent it is (obviously), the social measures taken to combat it, antiviral resistance, and many other factors.

I don’t think anyone knows how the balance between vastly increased medical knowledge and vastly increased national and international travel will play out. If this virus is not popping up all over the world within a week’s time, I’ll be surprised. Airports are already screening people arriving from Mexico, but I imagine it’s too late and it’s certainly not being done globally.

History dictates that you should probably not believe anything any politician says about pandemic influenza. There has been a strong tendency to downplay risks. All sorts of factors are at work in communicating with the public. You can be sure that everything officially said by the WHO or CDC has been very carefully vetted and considered. There’s no particular reason to believe anything else you hear, either :-)

Facemasks have an interesting history, and have made it into law several times. In 1918 we didn’t even know what a virus was, let alone how tiny they are, so the gauze on the masks was likely totally ineffective.

In conclusion, I’d say that the thing is largely out of our hands for the time being. We’re going to have to wait and see what happens, and make our best guesses along the way.

The influenza people at the CDC and the other international labs are an amazing team of experts. They’ve been at this game for a very long time and they work extremely hard and generally get a bad rap. It’s no wonder flu is such a political issue, the responsibility is high and the tendency towards opaqueness is understandable. Despite all the expertise though, at bottom you have an extremely complex virus – much of whose behavior is unknown, especially in the case of antigenic shift, especially when it is so young, and especially when you don’t know what nearby mutational opportunities may exist for it in antigenic space – spreading in a vastly more complex environment (our bodies), and with us moving and interacting in odd ways in a complex and extremely interconnected world. It’s a wonder we know as much as we do, but in many ways we don’t know much at all.

Below is one I encountered a few days ago. Can you place it? You can find the answer on Google in a flash.

The glamour of youth enveloped his parti-coloured rags, his destitution, his loneliness, the essential desolation of his futile wanderings. For months — for years — his life hadn’t been worth a day’s purchase; and there he was gallantly, thoughtlessly alive, to all appearances indestructible solely by the virtue of his few years and of his unreflecting audacity. I was seduced into something like admiration — like envy. Glamour urged him on, glamour kept him unscathed. He surely wanted nothing from the wilderness but space to breathe in and to push on through. His need was to exist, and to move onwards at the greatest possible risk, and with a maximum of privation. If the absolutely pure, uncalculating, unpractical spirit of adventure had ever ruled a human being, it ruled this be-patched youth. I almost envied him the possession of this modest and clear flame. It seemed to have consumed all thought of self so completely, that even while he was talking to you, you forgot that it was he— the man before your eyes— who had gone through these things.

We were graduate students together at The University of Waterloo in Canada in 1986-88. I met him on my first day there and we spent many hours together on a daily basis over the next 2.5 years. I don’t want to try to say too much about him now. It occurred to me a few days ago that I might post a few stories here. We did lots of crazy things. At one point I had wanted to write something titled “100 things to a Hensel” and I made a bunch of notes, but it went no further.

I wrote about him in my Ph.D. acknowledgments in 1995:

Andrew Hensel, with whom I shared so much of my two and a half years at Waterloo, was the most original and creative person I have ever known well. Together, we dismantled the world and rebuilt it on our own crazy terms. We lived life at a million miles an hour and there was nothing like it. Five years ago, Andrew killed himself. There have been few days since then that I have not thought of him and the time we spent together.

I still think about him frequently. Today I was remembering one of his many, many oddball projects (most of which went unfinished), which he called “10,000 things”. It was to be a list of 10,000 things that he thought of. By the time he started sending them to me we had both dropped out of Waterloo. He was back in Australia and I was in Munich.

He only sent me 300 of the to-be 10,000. Of course I still have them. They’re all very short. At the risk of being thought macabre I’ve decided to bring Andrew back a very little and post them to Twitter, chosen at random, one a day. You can follow adhensel to get just a glimpse of his mind. The first tweet, people being planted into earth, is already up.

There are at least half a dozen twitterers who knew Andrew, including one who knew him probably better than anybody. Once in a while I get email from someone who finds my online mentions of him. Invariably they also found him extraordinary.

What would Andrew have made of Twitter? I have no doubt at all that he’d have immediately dismissed it as “weak”. That was one of his favorite adjectives. Almost everything was weak. It’s a small miracle to me to partly bring him back to life 18 years after he died, by posting just some of his 10,000 things to Twitter.

And… my apologies to anyone who knew Andrew and who finds this upsetting.

First off, Tim is very generous in doing this. He’s ultra connected and he spends a significant amount of his time in Twitter pointing things out, connecting people, and re-tweeting stuff he finds interesting. Re-tweeting is really important because when you tweet you only reach the people who are already following you. But when someone re-tweets you, you reach new people who likely have no idea of your existence. And when Tim does the re-tweeting there can be a big impact. 24 hours after his message to Robert I had 50 new followers. Tim explicitly tries to help people doing things he finds interesting, but who have just a small number of Twitter followers. He filters and amplifies information, broadcasting it out to his 16,000+ followers. Robert was in a hotel about 10 minutes’ walk from my place and I had no idea. A mutual friend in California noticed and took a minute to connect us. That’s really something, and it perfectly illustrates some of the value of Twitter.

I met Robert yesterday afternoon and we spent 6 hours together. It was great. You can see at once why he’s been so successful: he’s smart, he’s thoughtful, he’s sympathetic, and he’s a careful listener. I had no idea what to expect, and seeing as what we’re building can take some time to sink in, I wondered what sort of an audience he’d be.

After we’d climbed around up in the Sagrada Familia (official site, wikipedia), Robert came back to my place to see a demo of the things I’d been describing. We sat down and he pulled out his cell phone and asked if he could film me. I didn’t really think about it and said of course. It didn’t dawn on me that we were doing an informal interview, and I was totally unprepared – which is probably a good thing.

In the end we filmed 4 segments: parts one, two, three, and four. There’s also been some discussion here on Robert’s FriendFeed page.

So if you’ve been wondering what we’re building in here, go watch the videos.

I had no idea all this was about to come down. The Fluidinfo web site (a generous word) was a single page with no contact information, no nuthin’. We simply haven’t needed a web site of any description yet. I went and added a box so you can sign up to receive news of the alpha launch.

And then there was this, posted on Twitter, and which I have absolutely no shame in reproducing (this is a blog, after all):

Wow, what @terrycojones showed me last night (a new kind of database that he’s been workng on for 11 years) blew me away. Uploading vids now

Now I have to put my head back down with Esteve to get the alpha out the door ASAP.

I started to write the news:

Today the US Department of Homeland Security announced that the US is insisting that countries install the new Passenger Mood Assessment Security Screening (PMASS) in all airports with direct or connecting flights to the US. The system, developed by Kellogg Brown & Root (a wholly-owned subsidiary of Halliburton, an oil-services company), is based on techniques borrowed from functional magnetic resonance imaging and, most controversially, the mood ring industry.

Ashcroft, Bush and Rumsfeld flanked Homeland Security Director Tom Ridge today as he announced the new measures. "People have no right to turn up at the airport with a bad attitude."

Halliburton wins a no-bid contract, shortly after moving to buy up all known mood ring manufacturers, mystifying investors (stock falls sharply, then rises sharply on news of contract). To fly people to the US, other countries are forced install giant mood rings and to hand over the mood data of all embarking passengers to US authorities.

You get the picture. I sent an outline of this to the folks at The Onion, but never heard back.

It’s a joke, of course. Couldn’t possibly happen. Right? Think again.

On July 29, 2007 we noticed that both our machines had been hacked into. We’re both very security conscious and we didn’t know how this had happened. Both machines had been broken into in the same way, from the same IP address, leaving behind the same evidence. It was pretty clearly a root kit.

I immediately contacted Serverpronto and asked for help. They offered to do a full system restore for $69 and were otherwise completely passive. I suggested that they might want to take a bit more interest as perhaps their other clients had also been hacked. After a couple of mails I got an email saying that they would make their best effort. But it was too late and I replied asking them to cancel service.

From terry Mon Jul 30 02:32:59 +0200 2007
Message-ID: <18093.12731.650708.403211@terry-jones-computer.local>
Date: Mon, 30 Jul 2007 02:32:59 +0200
From: Terry Jones <terry@XXX.es>
To: "SP1443" <support@serverpronto.com>
Subject: Re: [WYL-89853]: Machines broken into?
In-Reply-To: Your message at 18:34:08 on Sunday, 29 July 2007
References: <jlypcw.k87@serverpronto.infolink.com>

Thanks, but no. I would simply like to cancel service. What more
do I need to do?

Terry

====== Please reply above this line ======
Machines broken into?

Mr. Jones. I will transfer to the abuse team, and the Customer
Service your issue.

We will find the best solution.

They replied:

From support@serverpronto.com  Tue Jul 31 00:00:10 2007
Message-ID: <jlzkyj.p6l@serverpronto.infolink.com>
From: "SP1422" <upport@serverpronto.com>
Subject: [WYL-89853]: Machines broken into?
Date: Mon, 30 Jul 07 05:56:43 -0400
To: terry@jon.es

Machines broken into?

69.60.109.173 has been taken offline.
ServerPronto Support

Ticket Details
=========
Ticket ID: WYL-89853
Tracking URL: https://serverpronto.infolink.com/esupport/index.php?_a=tickets&_m=viewmain&emailre=terry@jon.es&ticketkeyre=dee55eec&_i=WYL-89853
Department: Technical Support
Created On: 29 Jul 2007 07:57 AM
Last Update: 29 Jul 2007 05:34 PM
Status: Closed

That’s it, right? End of story.

Unfortunately not. First of all, my friend could not get Serverpronto to stop billing after he also canceled his contract. They just kept billing. After many phone calls and emails, they eventually had to get rid of Serverpronto via getting the credit card company to refuse all charges.

I thought I had gotten off their books. But I got a phone call from Deutsche Bank a couple of months ago telling me that a company called Serverpronto was trying repeatedly to charge my now-expired credit card. I was completely surprised and told them that I had nothing to do with Serverpronto and that I had canceled service with them. I had been receiving mail from Serverpronto telling me that my credit card expiry date was coming up. I mailed them to tell them there was a mistake and that I was no longer a customer, and they promised to look into it.

And now, today, I get a statement from my bank showing that they have approved two charges for a total of $356.30 from Serverpronto!!!! That means those evil assholes have somehow called Deutsche Bank and managed to charge me for a whole year of costs on an account that was closed, for a server that was not up, and on a credit card that had expired.

I am assured by my friend that I will never see a cent of that money again.

Deutsche Bank are about to hear from me.

DON’T USE SERVERPRONTO.

Go search Google for things like Serverpronto billing cancel ripoff. You’ll get tons of hits. I should have done more homework before I signed up. They actually had good support. But the people running the billing department deserve a slow painful death. This is obviously a deliberate policy designed to extract the maximum out of their trying-to-depart customers.

To whom do I complain?

I’ll first explain what I mean by “the creation of highly non-uniform value”. I’m talking about ideas that create a lot of (monetary) value for a very small number of people. If you made a graph and on the X axis put all the people in the world, in sorted order of how much they make from an idea, and on the Y axis you put value they each receive, we’re talking about distributions that look like the image above, but much more skewed.

In other words, a setting in which a very small number of people try to get extremely rich. I.e., startup founders, a few key employees, their investors, and their investors’ investors. BTW, I don’t want to talk about the moral side of this, if there is one. There’s nothing to stop the obscenely rich from giving their money away or doing other charitable things with it.

So let’s just accept that many startup founders, and (in theory) all venture investors, are interested in turning ideas into wealth distributions that look like the above.

I was partly beaten to the punch on this post by Paul Graham in his essay Why There Aren’t More Googles? Paul focused on VC caution, and with justification. But there’s another important part of the answer.

One of the most fascinating things I’ve heard in the last couple of years is an anecdote about the early Google. I wrote about it in an earlier article, The blind leading the blind:

…the Google guys were apparently running around search engine companies trying to sell their idea (vision? early startup?) for $1M. They couldn’t find a buyer. What an extraordinary lack of.. what? On the one hand you want to laugh at those idiot companies (and VCs) who couldn’t see the huge value. OK, maybe. But the more extraordinary thing is that Larry Page and Sergei Brin couldn’t see it either! That’s pretty amazing when you think about it. Even the entrepreneurs couldn’t see the enormous value. They somehow decided that $1M would be an acceptable deal. Talk about a lack of vision and belief.

So you can’t really blame the poor VCs or others who fail to invest. If the founding tech people can’t see the value and don’t believe, who else is going to?

I went on to talk about what seemed like it might be a necessary connection between risk and value.

Following on…

After more thought, I’m now fairly convinced that I was on the right track in that post.

It seems to me that the degree to which a highly non-uniform wealth distribution can be created from an idea depends heavily on how non-obvious the value of the idea is.

If an idea is obviously valuable, I don’t think it can create highly non-uniform wealth. That’s not to say that it can’t create vast wealth, just that the distribution of that wealth will be more widely spread. Why is that the case? I think it’s true simply because the value will be apparent to many people, there will be multiple implementations, and the value created will be spread more widely. If the value of an idea is clear, others will be building it even as you do. You might all be very successful, but the distribution of created value will be more uniform.

Obviously it probably helps if an idea is hard to implement too, or if you have some other barrier to entry (e.g., patents) or create a barrier to adoption (e.g., users getting positive reinforcement from using the same implementation).

I don’t mean to say that an idea must be uniquely brilliant, or even new, to generate this kind of wealth distribution. But it needs to be the kind of proposition that many people look at and think “that’ll never work.” Even better if potential competitors continue to say that 6 months after launch and there’s only gradual adoption. Who can say when something is going to take off wildly? No-one. There are highly successful non-new ideas, like the iPod or YouTube. Their timing and implementation were somehow right. They created massive wealth (highly non-uniformly distributed in the case of YouTube), and yet many people wrote them off early on. It certainly wasn’t plain sailing for the YouTube founders – early adoption was extremely slow. Might Twitter, a pet favorite (go on, follow me), create massive value? Might Mahalo? Many people would have found that idea ludicrous 1-2 years ago – but that’s precisely the point. Google is certainly a good example – search was supposedly “done” in 1998 or so. We had Alta Vista, and it seemed great. Who would’ve put money into two guys building a search engine? Very few people.

If it had been obvious the Google guys were doing something immensely valuable, things would have been very different. But they traveled around to various companies (I don’t have this first hand, so I’m imagining), showing a demo of the product that would eventually create $100-150B in value. It wasn’t clear to anyone that there was anything like that value there. Apparently no-one thought it would be worth significantly more that $1M.

I’ve come to the rough conclusion that that sort of near-universal rejection might be necessary to create that sort of highly non-uniform wealth distribution.

There are important related lessons to be learned along these lines from books like The Structure of Scientific Revolutions and The Innovator’s Dilemma.

Now back to Paul’s question: Why aren’t there more Googles?

Part of the answer has to be that value is non-obvious. Given the above, I’d be willing to argue (over beer, anyway) that that’s almost by definition.

So if value is non-obvious, even to the founders, how on earth do things like this get created?

The answer is passion. If you don’t have entrepreneurs who are building things just from sheer driving passion, then hard projects that require serious energy, sacrifice, and risk-taking, simply wont be built.

As a corollary, big companies are unlikely to build these things – because management is constantly trying to assess value. That’s one reason to rue the demise of industrial research, and a reason to hope that cultures that encourage people to work on whatever they want (e.g., Google, Microsoft research) might be able to one day stumble across this kind of value.

This gets me to a recent posting by Tim Bray, which encourages people to work on things they care about.

It’s not enough just to have entrepreneurs who are trying to create value. As I’m trying to say, practically no-one can consistently and accurately predict where undiscovered value lies (some would argue that Marc Andreessen is an exception). If it were generally possible to do so, the world would be a very different place – the whole startup scene and venture/angel funding system would be different, supposing they even existed. Even if it looks like a VC or entrepreneur can infallibly put their finger on undiscovered value, they probably can’t. One-time successful VCs and entrepreneurs go on to attract disproportionately many great companies, employees, funding, etc., the next time round. You can’t properly separate their raw ability to see undiscovered value from the strong bias towards excellence in the opportunities they are later afforded. Successful entrepreneurs are often refreshingly and encouragingly frank about the role of luck in their success. They’re done. VCs are much less sanguine – they’re supposed to have natural talent, they’re trying to manufacture the impression that they know what they’re doing. They have to do that in order to get their limited partners to invest in their funds. For all their vaunted insight, roughly only 25% of VCs provide returns that are better than the market. The percentage generating huge returns will of course be much smaller, as in turn will be those doing so consistently. I reckon the whole thing’s a giant crap shoot. We may as well all admit it.

I have lots of other comments I could make about VCs, but I’ll restrict myself to just one as it connects back to Paul’s article.

VCs who claim to be interested in investing in the next Google cannot possibly have the next Google in their portfolio unless they have a company whose fundamental idea looks like it’s unlikely to pan out. That doesn’t mean VCs should invest in bad ideas. It means that unless VCs make bets on ideas that look really good – but which are e.g., clearly going to be hard to build, will need huge adoption to work, appear to be very risky long-shots, etc. – then they can’t be sitting on the next Google. It also doesn’t mean VCs must place big bets on stuff that’s highly risky. A few hundred thousand can go a long way in a frugal startup.

I think this is a fundamental tradeoff. You’ll very frequently hear VCs talk about how they’re looking for companies that are going to create massive value (non-uniformly distributed, naturally), with massive markets, etc. I think that’s pie in the sky posturing unless they’ve already invested in, or are willing to invest in, things that look very risky. That should be understood. And so a question to VCs from entrepreneurs and limited partners alike: if you claim to be aiming to make massive returns, where are your necessary correspondingly massively risky investments? Chances are you wont find any.

There is a movement in the startup investment world towards smaller funds that make smaller investments earlier. I believe this movement is unrelated to my claim about non-obviousness and highly non-uniform returns. The trend is fuelled by the realization that lots of web companies are getting going without the need for traditional levels of financing. If you don’t get in early with them, you’re not going to get in at all. A big fund can’t make (many) small investments, because their partners can’t monitor more than a handful of companies. So funds that want to play in this area are necessarily smaller. I think that makes a lot of sense. A perhaps unanticipated side effect of this is that things that look like they may be of less value end up getting small amounts of funding. But on the whole I don’t think there’s a conscious effort in that direction – investors are strongly driven to select the least risky investment opportunities from the huge number of deals they see. After all, their jobs are on the line. You can’t expect them to take big risks. But by the same token you should probably ignore any talk of “looking for the next Google”. They talk that way, but they don’t invest that way.

Finally, if you’re working on something that’s being widely rejected or whose value is being widely questioned, don’t lose heart (instead go read my earlier posting) and don’t waste your time talking to VCs. Unless they’re exceptional and serious about creating massive non-uniformly distributed value, and they understand what that involves, they certainly wont bite.

Instead, follow your passion. Build your dream and get it out there. Let the value take care of itself, supposing it’s even there. If you can’t predict value, you may as well do something you really enjoy.

Now I’m working hard to follow my own advice.

I had to learn all this the hard way. I spent much of 2008 on the road trying to get people to invest in Fluidinfo, without success. If you’re interested to know a little more, earlier tonight I wrote a Brief history of an idea to give context for this posting.

That’s it for now. Blogging is a luxury I can’t afford right now, not that I would presume to try to predict which way value lies.

For example, I put considerable time and effort into writing academic papers, several of which were rejected, to my surprise. I’d never considered that the papers might not be accepted. When this happened, I wouldn’t re-submit them or try to re-publish them. By then I would usually have moved on to doing something else anyway.

When I applied for jobs, it never entered my mind that I might not be wanted. How could anyone not want me? After a couple of years working on my current ideas, I applied for a computer science faculty position at over 40 US universities. I refused to emphasize my well-received and published Ph.D. work, of which I was and am still proud, because I was no longer working in that area.

I was convinced the new ideas would be recognized as being strong.

But guess what? I was summarily rejected by all 40+ universities. I only got one interview, at RPI. No other school even wanted to meet me. I kept all the rejection letters. I still have them. (Amusingly, I was swapping emails with Ben Bederson earlier this year and it transpired that he’d had the same experience, also with 40 universities, and he too kept all his rejection letters!)

You never learn more than when you’re being humbled.

I’ve now returned to those same ideas and have been working on them for the last 3 years. In January 2007 I went and met with a couple of the most appropriately visionary VCs to tell them what I was building. I was naïve enough to think they might back me at that early point. Wrong. They suggested I come back with a demo to concretely illustrate what the system would allow people to do. That was easier said than done – the system is not simple. I spent 2007 building the core engine, a 90% fully-functional demo of the major application, several smaller demo apps (including a Firefox toolbar extension built by Esteve Fernandez), and added about 20 sample data sets to further illustrate possibilities.

That’ll show ‘em, right? I went out in November 2007 armed with all this, and began talking to a variety of potential investors. I was sure VCs would be falling over themselves to invest, especially given that we were working on some mix of innovative search, cloud computation, APIs, and various Web 2.0 concepts, and that tons of VCs claimed to be looking for the Next Big Thing in search, and for Passionate Entrepreneurs tackling Hard Problems who wanted to build Billion Dollar Companies, etc., etc.

You guessed it. Over the next year literally dozens of potential investors all said no. The demo wasn’t enough. Would people use it? Could we build the real thing? Would it scale? Where was the team? What are you doing in Barcelona? “Looks fascinating, do please let us know when you’ve released it and are seeing adoption,” they almost universally told me. The standout exception to this was Esther Dyson, who agreed to invest immediately after seeing the demo, and whose courage I hope I can one day richly reward.

What to make of all this rejection?

One thing that became clear is that if you’re smarter than average, you’ll almost by definition be constantly thinking of things too early. Maybe many years too early. Your ideas will seem strange, doubtful, and perhaps plain wrong to many people.

This makes you realize how important timing is.

Being right with an idea too early and trying to build a startup around it is similar to correctly knowing a company is going to fail, and immediately rushing out to short its stock. Even though you’re right, you can be completely wiped out if the stock’s value rises in the short term. You were brilliant, insightful, and 100% correct – but you were too early.

Getting timing right can clearly be partly based on calculation and reason. But given that many startups are driven by founder passion, I think luck in timing plays an extremely important role in startup success. And the smarter and more far-sighted you are, the greater the chance that your timing will be wrong.

So the that’s the first thing to understand: if you’re smarter than average, your ideas will, on average, be ahead of their time. Some level of rejection comes with the territory.

But I’d go much further than that, and claim that if you are not seeing a very high level of rejection in trying to get a new idea off the ground, you’re probably not working on anything that’s going to change the world or be extremely valuable.

That might sound like an outrageous extrapolation (or even wishful thinking, given my history). Later tonight I plan to explain this claim in a post on the connections between passion, value, non-obviousness, and rejection. That’s the subject I really want to write about.

For now though, I simply want to say that I’ve come to understand that having one’s ideas regularly rejected is a good sign. It tells you you’re either on a fool’s errand, or that you’re doing something that might actually be valuable and important.

If you’re not going to let rejection get you down, you might content yourself by learning to ignore it. But you can do better. You can come to regard it as positive and affirming. Without becoming pessimistic or in any way accepting defeat, you can come to expect to be rejected and even to embrace it.

If you can do that, rejection loses its potential for damage. As Paul Graham pointed out, the impact of disappointment can destroy a startup. That’s an important observation, and a part of why startups can be so volatile and such a wild ride.

I don’t mean to suggest that you don’t also do practical things with rejection too – like learn from it. That’s very important and will help you shape your product, thoughts, presentation, expectations, etc. Again, see Paul’s posting.

But I think the mental side of rejection is more important than the practical. The mental side has more destructive potential. You have to figure out how to deal with it. If you look at it the right way you can turn it into something that’s almost by definition positive, as I’ve tried to illustrate.

In a sense I even relish it, and use it for fuel. There are little tricks I sometimes use to keep myself motivated. I even keep a list of them (and no, you can’t see it). One is imagining that some day all the people who rejected me along the way will wring their hands in despair at having missed such an opportunity :-)

I’ve not been universally rejected, of course. There are lots of people who know what we’re doing and are highly supportive (more on them at a later point). If I’d been universally rejected, or rejected by many well-known people whose opinions I value, I probably would have stopped by now.

I’ve had to learn to see a high level of rejection as not just normal but a necessary (but not sufficient!) component of a correct belief that you’re doing something valuable.

Stay tuned for the real point of this flurry of blogging activity.