Terry Jones

Here, finally, are some thoughts on the creation of value. I don’t plan to do as good a job as the subject merits, but if I don’t take a rough stab at it, it’ll never happen.

I’ll first explain what I mean by “the creation of highly non-uniform value”. I’m talking about ideas that create a lot of (monetary) value for a very small number of people. If you made a graph and on the X axis put all the people in the world, in sorted order of how much they make from an idea, and on the Y axis you put value they each receive, we’re talking about distributions that look like the image on the right.

In other words, a setting in which a very small number of people try to get extremely rich. I.e., startup founders, a few key employees, their investors, and their investors’ investors. BTW, I don’t want to talk about the moral side of this, if there is one. There’s nothing to stop the obscenely rich from giving their money away or doing other charitable things with it.

So let’s just accept that many startup founders, and (in theory) all venture investors, are interested in turning ideas into wealth distributions that look like the above.

I was partly beaten to the punch on this post by Paul Graham in his essay Why There Aren’t More Googles? Paul focused on VC caution, and with justification. But there’s another important part of the answer.

One of the most fascinating things I’ve heard in the last couple of years is an anecdote about the early Google. I wrote about it in an earlier article, The blind leading the blind:

…the Google guys were apparently running around search engine companies trying to sell their idea (vision? early startup?) for $1M. They couldn’t find a buyer. What an extraordinary lack of.. what? On the one hand you want to laugh at those idiot companies (and VCs) who couldn’t see the huge value. OK, maybe. But the more extraordinary thing is that Larry Page and Sergei Brin couldn’t see it either! That’s pretty amazing when you think about it. Even the entrepreneurs couldn’t see the enormous value. They somehow decided that $1M would be an acceptable deal. Talk about a lack of vision and belief.

So you can’t really blame the poor VCs or others who fail to invest. If the founding tech people can’t see the value and don’t believe, who else is going to?

I went on to talk about what seemed like it might be a necessary connection between risk and value.

Image: Lost Tulsa

After more thought, I’m now fairly convinced that I was on the right track in that post.

It seems to me that the degree to which a highly non-uniform wealth distribution can be created from an idea depends heavily on how non-obvious the value of the idea is.

If an idea is obviously valuable, I don’t think it can create highly non-uniform wealth. That’s not to say that it can’t create vast wealth, just that the distribution of that wealth will be more widely spread. Why is that the case? I think it’s true simply because the value will be apparent to many people, there will be multiple implementations, and the value created will be spread more widely. If the value of an idea is clear, others will be building it even as you do. You might all be very successful, but the distribution of created value will be more uniform.

Obviously it probably helps if an idea is hard to implement too, or if you have some other barrier to entry (e.g., patents) or create a barrier to adoption (e.g., users getting positive reinforcement from using the same implementation).

I don’t mean to say that an idea must be uniquely brilliant, or even new, to generate this kind of wealth distribution. But it needs to be the kind of proposition that many people look at and think “that’ll never work.” Even better if potential competitors continue to say that 6 months after launch and there’s only gradual adoption. Who can say when something is going to take off wildly? No-one. There are highly successful non-new ideas, like the iPod or YouTube. Their timing and implementation were somehow right. They created massive wealth (highly non-uniformly distributed in the case of YouTube), and yet many people wrote them off early on. It certainly wasn’t plain sailing for the YouTube founders – early adoption was extremely slow. Might Twitter, a pet favorite (go on, follow me), create massive value? Might Mahalo? Many people would have found that idea ludicrous 1-2 years ago – but that’s precisely the point. Google is certainly a good example – search was supposedly “done” in 1998 or so. We had Alta Vista, and it seemed great. Who would’ve put money into two guys building a search engine? Very few people.

If it had been obvious the Google guys were doing something immensely valuable, things would have been very different. But they traveled around to various companies (I don’t have this first hand, so I’m imagining), showing a demo of the product that would eventually create $100-150B in value. It wasn’t clear to anyone that there was anything like that value there. Apparently no-one thought it would be worth significantly more that $1M.

I’ve come to the rough conclusion that that sort of near-universal rejection might be necessary to create that sort of highly non-uniform wealth distribution.

There are important related lessons to be learned along these lines from books like The Structure of Scientific Revolutions and The Innovator’s Dilemma.

Now back to Paul’s question: Why aren’t there more Googles?

Part of the answer has to be that value is non-obvious. Given the above, I’d be willing to argue (over beer, anyway) that that’s almost by definition.

So if value is non-obvious, even to the founders, how on earth do things like this get created?

The answer is passion. If you don’t have entrepreneurs who are building things just from sheer driving passion, then hard projects that require serious energy, sacrifice, and risk-taking, simply wont be built.

As a corollary, big companies are unlikely to build these things – because management is constantly trying to assess value. That’s one reason to rue the demise of industrial research, and a reason to hope that cultures that encourage people to work on whatever they want (e.g., Google, Microsoft research) might be able to one day stumble across this kind of value.

This gets me to a recent posting by Tim Bray, which encourages people to work on things they care about.

It’s not enough just to have entrepreneurs who are trying to create value. As I’m trying to say, practically no-one can consistently and accurately predict where undiscovered value lies (some would argue that Marc Andreessen is an exception). If it were generally possible to do so, the world would be a very different place – the whole startup scene and venture/angel funding system would be different, supposing they even existed. Even if it looks like a VC or entrepreneur can infallibly put their finger on undiscovered value, they probably can’t. One-time successful VCs and entrepreneurs go on to attract disproportionately many great companies, employees, funding, etc., the next time round. You can’t properly separate their raw ability to see undiscovered value from the strong bias towards excellence in the opportunities they are later afforded. Successful entrepreneurs are often refreshingly and encouragingly frank about the role of luck in their success. They’re done. VCs are much less sanguine – they’re supposed to have natural talent, they’re trying to manufacture the impression that they know what they’re doing. They have to do that in order to get their limited partners to invest in their funds. For all their vaunted insight, roughly only 25% of VCs provide returns that are better than the market. The percentage generating huge returns will of course be much smaller, as in turn will be those doing so consistently. I reckon the whole thing’s a giant crap shoot. We may as well all admit it.

I have lots of other comments I could make about VCs, but I’ll restrict myself to just one as it connects back to Paul’s article.

VCs who claim to be interested in investing in the next Google cannot possibly have the next Google in their portfolio unless they have a company whose fundamental idea looks like it’s unlikely to pan out. That doesn’t mean VCs should invest in bad ideas. It means that unless VCs make bets on ideas that look really good – but which are e.g., clearly going to be hard to build, will need huge adoption to work, appear to be very risky long-shots, etc. – then they can’t be sitting on the next Google. It also doesn’t mean VCs must place big bets on stuff that’s highly risky. A few hundred thousand can go a long way in a frugal startup.

I think this is a fundamental tradeoff. You’ll very frequently hear VCs talk about how they’re looking for companies that are going to create massive value (non-uniformly distributed, naturally), with massive markets, etc. I think that’s pie in the sky posturing unless they’ve already invested in, or are willing to invest in, things that look very risky. That should be understood. And so a question to VCs from entrepreneurs and limited partners alike: if you claim to be aiming to make massive returns, where are your necessary correspondingly massively risky investments? Chances are you wont find any.

There is a movement in the startup investment world towards smaller funds that make smaller investments earlier. I believe this movement is unrelated to my claim about non-obviousness and highly non-uniform returns. The trend is fuelled by the realization that lots of web companies are getting going without the need for traditional levels of financing. If you don’t get in early with them, you’re not going to get in at all. A big fund can’t make (many) small investments, because their partners can’t monitor more than a handful of companies. So funds that want to play in this area are necessarily smaller. I think that makes a lot of sense. A perhaps unanticipated side effect of this is that things that look like they may be of less value end up getting small amounts of funding. But on the whole I don’t think there’s a conscious effort in that direction – investors are strongly driven to select the least risky investment opportunities from the huge number of deals they see. After all, their jobs are on the line. You can’t expect them to take big risks. But by the same token you should probably ignore any talk of “looking for the next Google”. They talk that way, but they don’t invest that way.

Finally, if you’re working on something that’s being widely rejected or whose value is being widely questioned, don’t lose heart (instead go read my earlier posting) and don’t waste your time talking to VCs. Unless they’re exceptional and serious about creating massive non-uniformly distributed value, and they understand what that involves, they certainly wont bite.

Instead, follow your passion. Build your dream and get it out there. Let the value take care of itself, supposing it’s even there. If you can’t predict value, you may as well do something you really enjoy.

Now I’m working hard to follow my own advice.

I had to learn all this the hard way. I spent much of 2008 on the road trying to get people to invest in Fluidinfo, without success. If you’re interested to know a little more, earlier tonight I wrote a Brief history of an idea to give context for this posting.

That’s it for now. Blogging is a luxury I can’t afford right now, not that I would presume to try to predict which way value lies.

When I was younger, I didn’t know what to make of it when people rejected my ideas. Instead of fighting it, trying again, or improving my delivery, I’d just conclude that the rejector was an idiot, and that it was their loss if they didn’t get it.

For example, I put considerable time and effort into writing academic papers, several of which were rejected, to my surprise. I’d never considered that the papers might not be accepted. When this happened, I wouldn’t re-submit them or try to re-publish them. By then I would usually have moved on to doing something else anyway.

When I applied for jobs, it never entered my mind that I might not be wanted. How could anyone not want me? After a couple of years working on my current ideas, I applied for a computer science faculty position at over 40 US universities. I refused to emphasize my well-received and published Ph.D. work, of which I was and am still proud, because I was no longer working in that area.

I was convinced the new ideas would be recognized as being strong.

But guess what? I was summarily rejected by all 40+ universities. I only got one interview, at RPI. No other school even wanted to meet me. I kept all the rejection letters. I still have them. (Amusingly, I was swapping emails with Ben Bederson earlier this year and it transpired that he’d had the same experience, also with 40 universities, and he too kept all his rejection letters!)

You never learn more than when you’re being humbled.

I’ve now returned to those same ideas and have been working on them for the last 3 years. In January 2007 I went and met with a couple of the most appropriately visionary VCs to tell them what I was building. I was naïve enough to think they might back me at that early point. Wrong. They suggested I come back with a demo to concretely illustrate what the system would allow people to do. That was easier said than done – the system is not simple. I spent 2007 building the core engine, a 90% fully-functional demo of the major application, several smaller demo apps (including a Firefox toolbar extension built by Esteve Fernandez), and added about 20 sample data sets to further illustrate possibilities.

That’ll show ’em, right? I went out in November 2007 armed with all this, and began talking to a variety of potential investors. I was sure VCs would be falling over themselves to invest, especially given that we were working on some mix of innovative search, cloud computation, APIs, and various Web 2.0 concepts, and that tons of VCs claimed to be looking for the Next Big Thing in search, and for Passionate Entrepreneurs tackling Hard Problems who wanted to build Billion Dollar Companies, etc., etc.

You guessed it. Over the next year literally dozens of potential investors all said no. The demo wasn’t enough. Would people use it? Could we build the real thing? Would it scale? Where was the team? What are you doing in Barcelona? “Looks fascinating, do please let us know when you’ve released it and are seeing adoption,” they almost universally told me. The standout exception to this was Esther Dyson, who agreed to invest immediately after seeing the demo, and whose courage I hope I can one day richly reward.

What to make of all this rejection?

One thing that became clear is that if you’re smarter than average, you’ll almost by definition be constantly thinking of things too early. Maybe many years too early. Your ideas will seem strange, doubtful, and perhaps plain wrong to many people.

This makes you realize how important timing is.

Being right with an idea too early and trying to build a startup around it is similar to correctly knowing a company is going to fail, and immediately rushing out to short its stock. Even though you’re right, you can be completely wiped out if the stock’s value rises in the short term. You were brilliant, insightful, and 100% correct – but you were too early.

Getting timing right can clearly be partly based on calculation and reason. But given that many startups are driven by founder passion, I think luck in timing plays an extremely important role in startup success. And the smarter and more far-sighted you are, the greater the chance that your timing will be wrong.

So the that’s the first thing to understand: if you’re smarter than average, your ideas will, on average, be ahead of their time. Some level of rejection comes with the territory.

But I’d go much further than that, and claim that if you are not seeing a very high level of rejection in trying to get a new idea off the ground, you’re probably not working on anything that’s going to change the world or be extremely valuable.

That might sound like an outrageous extrapolation (or even wishful thinking, given my history). Later tonight I plan to explain this claim in a post on the connections between passion, value, non-obviousness, and rejection. That’s the subject I really want to write about.

For now though, I simply want to say that I’ve come to understand that having one’s ideas regularly rejected is a good sign. It tells you you’re either on a fool’s errand, or that you’re doing something that might actually be valuable and important.

If you’re not going to let rejection get you down, you might content yourself by learning to ignore it. But you can do better. You can come to regard it as positive and affirming. Without becoming pessimistic or in any way accepting defeat, you can come to expect to be rejected and even to embrace it.

If you can do that, rejection loses its potential for damage. As Paul Graham pointed out, the impact of disappointment can destroy a startup. That’s an important observation, and a part of why startups can be so volatile and such a wild ride.

I don’t mean to suggest that you don’t also do practical things with rejection too – like learn from it. That’s very important and will help you shape your product, thoughts, presentation, expectations, etc. Again, see Paul’s posting.

But I think the mental side of rejection is more important than the practical. The mental side has more destructive potential. You have to figure out how to deal with it. If you look at it the right way you can turn it into something that’s almost by definition positive, as I’ve tried to illustrate.

In a sense I even relish it, and use it for fuel. There are little tricks I sometimes use to keep myself motivated. I even keep a list of them (and no, you can’t see it). One is imagining that some day all the people who rejected me along the way will wring their hands in despair at having missed such an opportunity :-)

I’ve not been universally rejected, of course. There are lots of people who know what we’re doing and are highly supportive (more on them at a later point). If I’d been universally rejected, or rejected by many well-known people whose opinions I value, I probably would have stopped by now.

I’ve had to learn to see a high level of rejection as not just normal but a necessary (but not sufficient!) component of a correct belief that you’re doing something valuable.

Stay tuned for the real point of this flurry of blogging activity.

In 1997 I had a fairly simple idea.

I spent a year thinking about it obsessively: where it might lead, what it would enable, how to implement it, etc. As I considered and refined it during this time, I began to appreciate its generality and power. I soon realized you could use it computationally to do anything and everything.

By day I was running Teclata S.L. a too-early web company in Barcelona. By night, very often until 4 or 5am, I was up thinking and writing, trying to get a better grip on what I’d dreamed up. I wrote hundreds of pages of notes. That’s a year of thinking, with no coding at all.

Teclata was killing me. I was burning to work on my ideas instead. I was exhausted, and several times literally in tears with the frustration. So in 1998 we sold the company and I started as a postdoc in Jim Hollan‘s Distributed Cognition and Human Computer Interaction lab in the Department of Cognitive Science at UCSD. I spent a year there writing code to build a prototype. I worked typically 15 hours a day for a whole year, producing roughly 30K lines of C code. During that year I think I went out only a few nights, saw exactly one movie, and made no new friends apart from Al Davis, a brilliant homeless guy who hung around campus. The prototype implementation worked, and I built some trivial applications on top of it.

I’ve now spent most of 2006-2008 back working on the same ideas, founding Fluidinfo in the process.

From March 2006 I began re-coding my UCSD work in Python. During 2007 I built a demo version, writing the core engine myself. One other person, Daniel Burr, worked on building a web front-end for a particularly important application. That was another 30K lines of code. To fund that effort, we sold a small apartment we owned in Barcelona and I took a small investment from my parents. During 2008, with funding from Esther Dyson, I’ve worked on a full distributed implementation, with Esteve Fernandez. We’re getting there, and are planning to release something in early 2009. I can’t wait to be able to talk about it all more openly.

That’s 5 years of my life working to reduce this idea to practice.

The opportunity cost in the last few years has been huge. The career risk feels huge. The salary cost has definitely been huge. I’m 45, and my most expensive personal possession is either my CD player or a €200 bicycle I bought about 5 years ago (I have a few computers – but they’re all company/job bought). Plus, I have 3 kids.

I have many other blog posts I’d love to write about this journey (e.g., pond scum), but blogging is a low priority right now. I’ve just jotted down these notes in order to point to them from an upcoming post on the importance of passion, innovation, and the creation of value.

I’ve had plenty of time to think about the subject.

[Update: here are the follow-on posts: one on rejection and one on passion and value.]

OK, I admit, this is geeky.

But we’ve all run into the situation in which you’re using Python and Twisted, and you’re writing a new class and you want to call something from the __init method that returns a Deferred. This is a problem. The __init method is not allowed to return a value, let alone a Deferred. While you could just call the Deferred-returning function from inside your __init, there’s no guarantee of when that Deferred will fire. Seeing as you’re in your __init method, it’s a good bet that you need that function to have done its thing before you let anyone get their hands on an instance of your class.

For example, consider a class that provides access to a database table. You want the __init__ method to create the table in the db if it doesn’t already exist. But if you’re using Twisted’s twisted.enterprise.adbapi class, the runInteraction method returns a Deferred. You can call it to create the tables, but you don’t want the instance of your class back in the hands of whoever’s creating it until the table is created. Otherwise they might call a method on the instance that expects the table to be there.

A cumbersome solution would be to add a callback to the Deferred you get back from runInteraction and have that callback add an attribute to self to indicate that it is safe to proceed. Then all your class methods that access the db table would have to check to see if the attribute was on self, and take some alternate action if not. That’s going to get ugly very fast plus, your caller has to deal with you potentially not being ready.

I ran into this problem a couple of days ago and after scratching my head for a while I came up with an idea for how to solve this pretty cleanly via a Python metaclass. Here’s the metaclass code:

from twisted.internet import defer

class TxDeferredInitMeta(type):
    def __new__(mcl, classname, bases, classdict):
        hidden = '__hidden__'
        instantiate = '__instantiate__'
        for name in hidden, instantiate:
            if name in classdict:
                raise Exception(
                    'Class %s contains an illegally-named %s method' %
                    (classname, name))
        try:
            origInit = classdict['__init__']
        except KeyError:
            origInit = lambda self: None
        def newInit(self, *args, **kw):
            hiddenDict = dict(args=args, kw=kw, __init__=origInit)
            setattr(self, hidden, hiddenDict)
        def _instantiate(self):
            def addSelf(result):
                return (self, result)
            hiddenDict = getattr(self, hidden)
            d = defer.maybeDeferred(hiddenDict['__init__'], self,
                                    *hiddenDict['args'], **hiddenDict['kw'])
            return d.addCallback(addSelf)
        classdict['__init__'] = newInit
        classdict[instantiate] = _instantiate
        return super(TxDeferredInitMeta, mcl).__new__(
            mcl, classname, bases, classdict)

I’m not going to explain what it does here. If it’s not clear and you want to know, send me mail or post a comment. But I’ll show you how you use it in practice. It’s kind of weird, but it makes sense once you get used to it.

First, we make a class whose metaclass is TxDeferredInitMeta and whose __init__ method returns a deferred:

class MyClass(object):
    __metaclass__ = TxDeferredInitMeta
    def __init__(self):
        d = aFuncReturningADeferred()
        return d

Having __init__ return anything other than None is illegal in normal Python classes. But this is not a normal Python class, as you will now see.

Given our class, we use it like this:

def cb((instance, result)):
    # instance is an instance of MyClass
    # result is from the callback chain of aFuncReturningADeferred
    pass

d = MyClass()
d.__instantiate__()
d.addCallback(cb)

That may look pretty funky, but if you’re used to Twisted it wont seem too bizarre. What’s happening is that when you ask to make an instance of MyClass, you get back an instance of a regular Python class. It has a method called __instantiate__ that returns a Deferred. You add a callback to that Deferred and that callback is eventually passed two things. The first is an instance of MyClass, as you requested. The second is the result that came down the callback chain from the Deferred that was returned by the __init__ method you wrote in MyClass.

The net result is that you have the value of the Deferred and you have your instance of MyClass. It’s safe to go ahead and use the instance because you know the Deferred has been called. It will probably seem a bit odd to get your instance later as a result of a Deferred firing, but that’s perfectly in keeping with the Twisted way.

That’s it for now. You can grab the code and a trial test suite to put it through its paces at http://foss.fluidinfo.com/txDeferredInitMeta.zip. The code could be cleaned up somewhat, and made more general. There is a caveat to using it – your class can’t have __hidden__ or __instantiate__ methods. That could be improved. But I’m not going to bother for now, unless someone cares.

I just followed a link to a site that shows the date of the first tweet of 50 early Twitter users. I wondered how many of these early users were still active users, and guessed many would be.

Instead of going and fetching each user’s last tweet by hand, I wrote a little shell script to do all the work:

for name in \
  `curl -s http://myfirsttweet.com/oldest.php | 
   perl -p -e 's,,\nNAME:\t$1\n,g' |
   egrep '^NAME:' |
   cut -f2 |
   uniq`
do
    echo $name \
      `curl -s "http://twitter.com/statuses/user_timeline/$name.xml?count=1" |
       grep created_at |
       cut -f2 -d\> |
       cut -f1 -d\<`
done

This is just a quick post. I have a ton of things I could say about this, but they’ll have to wait – I need to do some real work.

Last night and today I wrote some Python code to dig into the follower and following sets of Twitter users.

I also think I understand better why Twitter is so compelling, but that’s going to have to wait for now too.

You give my program some Twitter user names and it builds you a table showing numbers of followers, following etc. for each user. It distinguishes between people you follow and who don’t follow you, and people who follow you but whom you don’t follow back.

But the really interesting thing is to look at the intersection of some of these sets between users.

For example, if I follow X and they don’t follow me back, we can assume I have some interest in X. So if am later followed by Y and it turns out that X follows Y, I might be interested to know that. I might want to follow Y back just because I know it might bring me to the attention of X, who may then follow me. If I follow Y, I might want to publicly @ message him/her, hoping that he/she might @ message me back, and that X may see it and follow me.

Stuff like that. If you think that sort of thing isn’t important, or is too detailed or introspective, I’ll warrant you don’t know much about primate social studies. But more on that in another posting too.

As another example use, I plan to forward the mails Twitter sends me telling me someone new is following me into a variant of my program. It can examine the sets of interest and weight them. That can give me an automated recommendation of whether I should follow that person back – or just do the following for me.

There are lots of directions you could push this in, like considering who the person had @ talked to (and whether those people were followers or not) and the content of their Tweets (e.g., do they talk about things I’m interested or not interested in?).

Lots.

For now, here are links to a few sample runs. Apologies to the Twitter users I’ve picked on – you guys were on my screen or on my mind (following FOWA).

I’d love to turn these into nice Euler Diagrams but I didn’t find any decent open source package to produce them.

I’m also hoping someone else (or other people) will pick this up and run with it. I’ve got no time for it! I’m happy to send the source code to anyone who wants it. Just follow me on Twitter and ask for it.

Example 1: littleidea compared to sarawinge.
Example 2: swardley compared to voidspace.
Example 3: aweissman compared to johnborthwick.

And finally here’s the result for deWitt, on whose Twitter Python library I based my own code. This is the output you get from the program when you only give it one user to examine.

More soon, I guess.

Here’s a short summary of a failed experiment using the Principle of Inclusion/Exclusion to estimate how many users Twitter has. I.e., there’s no answer below, just the outline of some quick coding.

I was wondering about this over cereal this morning. I know some folks at Twitter, and I know some folks who have access to the full tweet database, so I could perhaps get that answer just by asking. But that wouldn’t be any fun, and I probably couldn’t blog about it.

I was at FOWA last week and it seemed that absolutely everyone was on Twitter. Plus, they were active users, not people who’d created an account and didn’t use it. If Twitter’s usage pattern looks anything like a Power Law as we might expect, there will be many, many inactive or dormant accounts for every one that’s moderately active.

BTW, I’m terrycojones on Twitter. Follow me please, I’m trying to catch Jason Calacanis.

You could have a crack at answering the question by looking at Twitter user id numbers via the API and trying to estimate how many users there are. I did play with that at one point at least with tweet ids, but although they increase there are large holes in the tweet id space. And approaches like that have to go through the Twitter API, which limits you to a mere 70 requests per hour – not enough for any serious (and quick) probing.

In any case, I was looking at the Twitter Find People page. Go to the Search tab and you can search for users.

I searched for the single letter A, and got around 109K hits. That lead me to think that I could get a bound on Twitter’s size using the Principle of Inclusion/Exclusion (PIE). (If you don’t know what that is, don’t be intimidated by the math – it’s actually very simple, just consider the cases of counting the size of the union of 2 and 3 sets). The PIE is a beautiful and extremely useful tool in combinatorics and probability theory (some nice examples can be found in Chapter 3 of the introductory text Applied Combinatorics With Problem Solving). The image above comes from the Wikipedia page.

To get an idea of how many Twitter users there are, we can add the number of people with an A in their name to the number with a B in their name, …., to the number with a Z in their name.

That will give us an over-estimate though, as names typically have many letters in them. So we’ll be counting users multiple times in this simplistic sum. That’s where the PIE comes in. The basic idea is that you add the size of a bunch of sets, and then you subtract off the sizes of all the pairwise intersections. Then you add on the sizes of all the triple set intersections, and so on. If you keep going, you get the answer exactly. If you stop along the way you’ll have an upper or lower bound.

So I figured I could add the size of all the single-letter searches and then adjust that downwards using some simple estimates of letter co-occurrence.

That would definitely work.

But then the theory ran full into the reality of Twitter.

To begin with, Twitter gives zero results if you search for S or T. I have no idea why. It gives a result for all other (English) letters. My only theory was that Twitter had anticipated my effort and the missing S and T results were their way of saying Stop That!

Anyway, I put the values for the 24 letters that do work into a Python program and summed them:

count = dict(a = 108938,
             b =  12636,
             c =  13165,
             d =  21516,
             e =  14070,
             f =   5294,
             g =   8425,
             h =   7108,
             i = 160592,
             j =   9226,
             k =  12524,
             l =   8112,
             m =  51721,
             n =  11019,
             o =   9840,
             p =   8139,
             q =   1938,
             r =  10993,
             s =      0,
             t =      0,
             u =   8997,
             v =   4342,
             w =   6834,
             x =   8829,
             y =   8428,
             z =   3245)

upperBoundOnUsers = sum(count.values())
print 'Upper bound on number of users:', upperBoundOnUsers

The total was 515,931.

Remember that that’s a big over-estimate due to duplicate counting.

And unless I really do live in a tech bubble, I think that number is way too small – even without adjusting it using the PIE.

(If we were going to adjust it, we could try to estimate how often pairs of letters co-occur in Twitter user names. That would be difficult as user names are not like normal words. But we could try.)

Looking at the letter frequencies, I found them really strange. I wrote a tiny bit more code, using the English letter frequencies as given on Wikipedia to estimate how many hits I’d have gotten back on a normal set of words. If we assume Twitter user names have an average length of 7, we can print the expected numbers versus the actual numbers like this:

# From http://en.wikipedia.org/wiki/Letter_frequencies
freq = dict(a = 0.08167,
            b = 0.01492,
            c = 0.02782,
            d = 0.04253,
            e = 0.12702,
            f = 0.02228,
            g = 0.02015,
            h = 0.06094,
            i = 0.06966,
            j = 0.00153,
            k = 0.00772,
            l = 0.04025,
            m = 0.02406,
            n = 0.06749,
            o = 0.07507,
            p = 0.01929,
            q = 0.00095,
            r = 0.05987,
            s = 0.06327,
            t = 0.09056,
            u = 0.02758,
            v = 0.00978,
            w = 0.02360,
            x = 0.00150,
            y = 0.01974,
            z = 0.00074)

estimatedUserNameLen = 7

for L in sorted(count.keys()):
    probNotLetter = 1.0 - freq[L]
    probOneOrMore = 1.0 - probNotLetter ** estimatedUserNameLen
    expected = int(upperBoundOnUsers * probOneOrMore)
    print "%s: expected %6d, saw %6d." % (L, expected, count[L])

Which results in:

a: expected 231757, saw 108938.
b: expected  51531, saw  12636.
c: expected  92465, saw  13165.
d: expected 135331, saw  21516.
e: expected 316578, saw  14070.
f: expected  75281, saw   5294.
g: expected  68517, saw   8425.
h: expected 183696, saw   7108.
i: expected 204699, saw 160592.
j: expected   5500, saw   9226.
k: expected  27243, saw  12524.
l: expected 128942, saw   8112.
m: expected  80866, saw  51721.
n: expected 199582, saw  11019.
o: expected 217149, saw   9840.
p: expected  65761, saw   8139.
q: expected   3421, saw   1938.
r: expected 181037, saw  10993.
s: expected 189423, saw      0.
t: expected 250464, saw      0.
u: expected  91732, saw   8997.
v: expected  34301, saw   4342.
w: expected  79429, saw   6834.
x: expected   5392, saw   8829.
y: expected  67205, saw   8428.
z: expected   2666, saw   3245.

You can see there are wild differences here.

While it’s clearly not right to be multiplying the probability of one or more of each letter appearing in a name by the 515,931 figure (because that’s a major over-estimate), you might hope that the results would be more consistent and tell you how much of an over-estimate it was. But the results are all over the place.

I briefly considered writing some code to scrape the search results and calculate the co-occurrence frequencies (and the actual set of letters in user names). Then I noticed that the results don’t always add up. E.g., search for C and you’re told there are 13,190 results. But the results come 19 at a time and there are 660 pages of results (and 19 * 660 = 12,540, which is not 13,190).

At that point I decided not to trust Twitter’s results and to call it quits.

A promising direction (and blog post) had fizzled out. I was reminded of trying to use AltaVista to compute co-citation distances between web pages back in 1996. AltaVista was highly variable in its search results, which made it hard to do mathematics.

I’m blogging this as a way to stop thinking about this question and to see if someone else wants to push on it, or email me the answer. Doing the above only took about 10-15 mins. Blogging it took at least a couple of hours :-(

Finally, in case it’s not clear there are lots of assumptions in what I did. Some of them:

• We’re not considering non-English letters (or things like underscores, which are common) in user names.
• The mean length of Twitter user names is probably not 7.
• Twitter search returns user names that don’t contain the searched-for letter (instead, the letter appears in the user’s name, not the username).

I had breakfast this morning at a bar in the Santa Caterina market in Barcelona with Jono Bennett. He’s a writer. We were reflecting on similarities in our struggles to do our own thing. An email about a potential Fluidinfo investor that I’d recently sent to a friend came to mind. I wrote:

I had a really good call with AAA. He told me he’s interested and wants to talk to BBB and CCC. I then got mail the next day from DDD (of the NYT) who told me he’d just had dinner with AAA and BBB and that they’d talked about my stuff. So something may happen there (i.e., I’ll never hear from them again).

The last comment, that I’d probably never hear from them again, was entirely tongue-in-cheek. I wrote it knowing it was a possibility, but not really thinking it would happen.

But it did.

Things like that seem to be part & parcel of the startup world as you attempt to get funded. I have often asked myself how can it be possible for things to be this way? How you can have people so excited, telling you and others you’re going to change the world, be worth billions, and then you never hear from them again? (Yes, of course you have to follow up, and I did. But that’s not the point: If you didn’t follow up you’d never hear from them.)

How can that be? In what sort of world is such a thing possible?

I came up with a highly flawed analogy. Despite its limited accuracy I find it amusing and can’t resist blogging it even if people will label me bitter (I’m not).

First: startup founders are pond scum. Second: potential investors are a troupe of young kids wandering through the park with sticks.

The kids poke into the ponds, stirring up the scum. They’re looking for cool things, signs of life, perhaps even something to take home. They’re genuinely interested. They’re fascinated. The pond scum listen to their excited conversation and think the kids will surely be back tomorrow. But it’s summer, and the world is so very very big.

The pond scum are working on little projects like photosynthesis, enhancements to the Krebs cycle, or the creation of life itself. All the while they’re pondering how to make themselves irresistible, believing that someday the kids with the sticks will be back, that they’ll eventually be scooped up.

As Paul Graham recently wrote, fundraising is brutal. His #1 recommendation is to keep expectations low.

Yep, you’re pond scum.

Get used to it.

Embrace it.

I drove from Oakland up to the O’Reilly Foo camp last Friday. The O’Reilly offices are just outside Sebastopol, CA. I stopped at an ATM and my GPS unit got totally confused. So I took a few turns at random and wound up on Florence Avenue. I drove a couple of hundred meters and started seeing big colorful structures out the front of many houses. They were so good I stopped, got out my camera, and took a whole bunch of pictures.

I talked to a man washing his car in his driveway. He told me that “Patrick” had created all the figures, and installed them on the front lawns. I got the impression that it was all free. Soon after I found the house that was unmistakably Patrick’s and seeing a man loading things into a pickup truck I went up and asked if he was Patrick. It was him and we had a friendly talk (mainly me telling him he was amazing). He gave me a calendar of his work.

Click on the thumbnails below to see bigger versions. There’s even a FC Barcelona structure. As I found out later, lots of people (of course) have seen these sculptures. When I got to Foo, there was one (image above) outside the O’Reilly office. Google for Patrick Amiot or Florence Avenue, Sebastopol and you’ll find much more. And Patrick has his own web site.

I was checking out the new bit.ly URL shortening service from Betaworks.

I started wondering how random the URLs from these URL-shortening services could be. I wrote a tiny script the other day to turn URLs given on the command line into short URLs via is.gd:

import urllib, sys
for arg in sys.argv[1:]:
    print urllib.urlopen(
        'http://is.gd/api.php?longurl=' + arg).read()

I ran it a couple of times to see what URLs it generated. Note that you have to use a new URL each time, as it’s smart enough not to give out a new short URL for one it has seen before. I got the sequence http://is.gd/JzB, http://is.gd/JzC, http://is.gd/JzD, http://is.gd/JzE,…

That’s an invitation to some minor mischief, because you can guess the next URL in the is.gd sequence before it’s actually assigned to redirect somewhere.

We can ask bit.ly for a short URL that redirects to our predicted next is.gd URL. Then we ask is.gd for a short URL that redirects to the URL that bit.ly gives us. If we do this fast enough, is.gd will not yet have assigned the predicted next URL and we’ll get it. So the bit.ly URL will end up redirecting to the is.gd URL and vice versa. In ugly Python (and with a bug/shortcoming in the nextIsgd function):

import urllib, random

def bitly(url):
    return urllib.urlopen(
        'http://bit.ly/api?url=' + url).read()

def isgd(url):
    return urllib.urlopen(
        'http://is.gd/api.php?longurl=' + url).read()

def nextIsgd(url):
    last = url[-1]
    if last == 'z':
        next = 'A'
    else:
        next = chr(ord(last) + 1)
    return url[:-1] + next

def randomURI():
    return 'http://www.a%s.com' % \
           ''.join(map(str, random.sample(xrange(100000), 3)))

isgdURL = isgd(randomURI())
print 'Last is.gd URL:', isgdURL

nextIsgdURL = nextIsgd(isgdURL)
print 'Next is.gd URL will be:', nextIsgdURL

# Ask bit.ly for a URL that redirects to nextIsgdURL
bitlyURL = bitly(nextIsgdURL)
print 'Step 1: bit.ly now redirects %s to %s' % (
    bitlyURL, nextIsgdURL)

# Ask is.gd for a URL that redirects to that bit.ly url
isgdURL2 = isgd(bitlyURL)
print 'Step 2: is.gd now redirects %s to %s' % (
    isgdURL2, bitlyURL)

if nextIsgdURL == isgdURL2:
    print 'Success'
else:
    print 'Epic FAIL'

This worked first time, giving:

Step 1: bit.ly now redirects http://bit.ly/fkuL8 to http://is.gd/JA9
Step 2: is.gd now redirects http://is.gd/JA9 to http://bit.ly/fkuL8

In general it’s not a good idea to use predictable numbers like this, which hardly bears saying as just about every responsible programmer knows that already.

is.gd wont shorten a tinyurl.com link, as tinyurl is on their blacklist. So they obviously know what they’re doing. The bit.ly service is brand new and presumably not on the is.gd radar yet.

And finally, what happens when you visit one of the deadly looping redirect URLs in your browser? You’d hope that after all these years the browser would detect the redirect loop and break it at some point. And that’s what happened with Firefox 3, producing the image above.

If you want to give it a try, http://bit.ly/fkuL8 and http://is.gd/JA9 point to each other. Do I need to add that I’m not responsible if your browser explodes in your face?

Another day, another great sight in the Born.

I was sitting here 15 minutes ago when I became aware of lots of drumming and piping outside. I tossed up whether to go down and film some of it, given that I’ve recently been posting a few things from the Born and it might be getting repetitive. I’m glad I did though, because the giants were out.

I’ve seen these giants dozens and dozens of times over the years in Barcelona, but I’m still not sick of them at all. I find them somehow majestic and solemn, and I love watching them parade down an old street bobbing up and down to the music, doing courtship dances, and spinning around. There’s also lots of variety. Sometimes you’ll see at least a hundred of them out for a special occasion.

Have a look at the video. It’s very impressive live. I hope some of that comes across in these few short clips. You have to wait until about 2 minutes in before the giants start moving and dancing.

Once a year they cook paella in the Born. They put out two long lines of tables and chairs. Anyone who wants a ticket buys one (8 euros in 2008) which gets you a large plate of paella, bread, salad, water and wine.

This year there were seats for 325 people.

It’s changed over the years. There are more tourists and it’s more expensive. Some say the paella is less good, though I don’t agree with that (in fact it’s never been fantastic). The atmosphere is completely relaxed with tons of people from the neighborhood out having a nice meal. I’ve been going every year for 10 years, when not away on travel.

The best part for me is usually watching them make the paella. This year was a bit different. For the first time they had butane gas burners under the paella pan. Usually they just break up a bunch of wood and set a blazing fire right there in the middle of the Born. They also only had one pan, whereas there are usually two.

The photo on the right shows one of the paddles they use to stir the paella. That should give you a good idea of how big these things get. Today they didn’t really need anything that long, but it was nice to have around just in case.

And below is a short movie I made. I chopped it up a little so as to end with a frame of my daughter.

OK, this will be a quick one. I’m trying to post occasional videos taken in my neighborhood.

Here you have a typical Catalan scene: a band playing and people dancing Sardanas. You can see this any weekend in front of the cathedral. But this was in the Born and I happened across it on the way home. It’s 60 steps from my front door (yes, I counted).

I don’t really like this music. Like living in Santa Fe and eating Southwest cuisine, I thought it was great at first but that quickly changed. I don’t enjoy the too-reedy quality of the sound and that it’s almost always identical. It’s also really long. But you may go ahead and enjoy it. Be my guest. I really like it when the first person does the initial piping and beats the tiny drum attached to his forearm near the elbow. The dancing starts a couple of minutes into the video. It’s cute.

The church in the background is Santa Maria del Mar, whose stained glass windows and gargoyles are about 4 meters from my balcony.

[This is a bit rambling / repetitive, sorry. I don’t have time to make it shorter, etc.]

Last year at FOWA I had a discussion with Paul Graham about programming and programmers in which we disagreed over the importance of knowing the fundamentals.

By this I mean the importance of knowing things down to the nuts and bolts level, to really understand what’s going on at the lower levels when you’re writing code. I used to think that sort of thing mattered a lot, but now I think it rarely does.

I well remember learning to program in AWK and being acutely aware of how resource intensive “associative arrays” (as we quaintly called them in those days) were, and knowing full well what was going on behind the scenes. I wrote a full Pascal compiler (no lex, no yacc) in the mid-80’s with Keith Rowe. If you haven’t done that, you really can’t appreciate the amount of computation that goes on when you compile a program to an executable. It’s astonishing. I did lots of assembly language programming, starting from age 15 or so, and spent years squeezing code into embedded environments, where a client might call to ask if you couldn’t come up with a way to reduce your executable code by 2 bytes so it would fit in their device.

But you know what? None of those skills really matter any more. Or they matter only very rarely.

The reason is that best practices have been worked out and incorporated into low-level libraries, and for the most part you don’t need to have any awareness at all of how those levels work. In fact it can be detrimental to you to spend years learning all those details if you could instead be learning how to build great things using the low-level libraries as black-box tools.

That’s the way the world moves in general. Successive generations get the accumulated wisdom of earlier generations packaged up for them. We used log tables, slide rules, and our heads, while our kids use calculators with hundreds of built-in functions. We learned to read analog 12-hour clocks, our kids learn to read digital clocks (so much easier!) and may not be able to read an analog clock until later. And it doesn’t matter. We buy a CD player (remember them?) or an iPod, and when it breaks you don’t even consider getting it “fixed” (remember that?). You just go out and buy another one. That’s because it’s cheaper and much faster and easier to just get a new one that has been put together by a machine than it is to have an actual human try to open the thing and figure out how to repair it. You can’t even (easily) open an iPod. And so the people who know how to do these things dwindle in number until there are none left. Like watch makers or the specialist knife sharpeners we have in Barcelona who ride around on motorcycles with their distinctive whistles, calling to people to bring down their blunt knives. And it doesn’t matter, at least from a technical point of view. Their brilliance and knowledge and hard-won experience has been encapsulated and put into machines and higher-level tools, or simply baked into society in smaller, more accurate and easier to digest forms. In computers it goes down into libraries and compilers and hardware. There’s simply no need for anyone to know how, learn how, or to bother, to do those sorts of things any more.

Note that I’m not saying it’s not nice to have your watch repaired by someone with a jeweler’s eyepiece or your knife or scissors sharpened in the street. I’m just noting the general progression by which knowledge inevitably becomes encapsulated.

In my discussion with Paul Graham, he argued that it was still important for tech founders to be great programmers at a low level. I argued that that’s not right. Sure, people like that are good to have around, but I don’t think you need to be that way and as I said I think it can even be detrimental because all that knowledge comes at a price (other knowledge, other experience).

I work with a young guy called Esteve (Hi Esteve!). He’s great at many levels, including the lower ones. He’s also a product of a new generation of programmers. They’re people who grew up only knowing object-oriented programming, only really writing in very high-level languages (not you Esteve! I mean that just in general), who think in those terms, and who instead of spending many years working with nuts and bolts spent the years working with newer high-level tools.

I think people like Esteve have a triple advantage over us dinosaurs. 1) They tend to use more powerful tools; 2) Because they use better tools, they are more comfortable and think more naturally in the terms of the higher-level abstractions their tools present them; and 3) they also have more experience putting those tools and methods to good use.

The experience gap widens at double speed, just as when a single voter changes side; the gap between the two parties increases by two votes. Even when the dinosaur modernizes itself and learns a few new tricks, you’re still way behind because the 25 year-old you’re working with (again, excluding Esteve) has never had to work at the nuts and bolts level. They think with the new paradigms and can put more general and more powerful tools directly into action. They don’t have to think about protocols or timeouts or dynamically resizing buffers or partial reads or memory management or data structures or error propogation. They simply think “Computer, fetch me the contents of that web page!” And most of the time it all just works. When it doesn’t, you can call in a gray-haired repair person or, more likely, just throw the busted tool away and buy another (or just get it free, in the case of Open Source software).

That’s real progress, and to insist that we should make the young suffer through all the stuff we had to learn in order to build all the libraries and compilers etc., that are now available to us all is just wrong. It’s wrong because it goes against the flow of history, because it’s counter-productive, and because it smacks of “I had to suffer through this stuff, walk barefoot to school in the snow, and therefore you must too.”

Some of the above will probably sound a bit abstract, but to me it’s not. I think it’s important to realize and accept. The fact that your kid can’t tie their shoelaces because they have velcro and have never owned a shoe with a lace is probably a good thing. You don’t know how to hunt your own food or start a fire, and it just doesn’t matter. The same goes for programming. The collective brilliance of generations of programmers is now built in to languages like Java, Python and Ruby, and into operating systems, graphics libraries, etc. etc., and it really doesn’t matter a damn if young people who are using those tools don’t have a clue what’s going on at the lower levels (as I said above, that’s probably a good thing). One day very few people will. The knowledge wont be lost. It’s just encapsulated into more modern environments and tools.

I’m writing all this down because I’ve been thinking about it on and off since FOWA, but also because of what I’m working on right now. I’m trying to modify 12K lines of synchronous Python code to use Twisted (an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers). The work is a bit awkward and three times I’ve not known how best to proceed in terms of design. Each time, Esteve has taken a look at the problem and quickly suggested a fairly clean way to tackle it. Desperate to cook up a way to think that he might not be that much smarter than I am, I’m forced into a corner in which I conclude that he has spent more time working with new tools (patterns, OO, a nice language like Python). So he looks at the world in a different way and naturally says “oh, you just do that”. Then I go do the routine work of making his ideas work – which is great by me, I get to learn in the best way, by doing. How nice to hire people who are better than you are.

That’s it. Encapsulation is inevitable. So you either have to embrace it or become a hand-wringing dinosaur moaning about the kids of today and how they no longer know the fundamentals. It’s not as though any of us could survive if we suddenly had to do everything from first principles (hunt, rub sticks together to make fire, etc). So relax. Enjoy it. The young are much better than we are because they grow up with better tools and they spend more time using them. It’s not enough to learn them when you’re older, even if you can do that really fast. You’ll never catch up on the experience front.

But it sure is fun to try.

In a belated attempt to educate myself by reading some of the things that many people study in high school, I’m reading The Histories of Herodotus. It’s highly entertaining and easy to read. I read The History of the Peloponnesian War by Thucydides a few years ago and enjoyed that even more. Herodotus is the more colorful, but the speeches and drama in Thucydides are fantastic.

There were lots of oracles in classical Greece, and elsewhere.Of the Greek oracles, the Delphic Oracle was, and still is, the best known. People (kings, dictators, emperors, wannabees) would send questions like “Should I invade Persia?” to the oracle and receive typically ambiguous or cryptic responses. We have a large number of famous oracular replies. Herodotus recounts how Croesus decided to test the various oracles by sending them all the same question, asking what he was doing on a certain day. The oracle at Delphi won hands down. Croesus then immediately put more pressing matters to the Delphic oracle, famously misinterpreted the pronouncements, and was duly wiped out by the Persians.

Imagine yourself in the position of the Delphic oracle. You’ve got all sorts of rulers and aspiring rulers constantly sending you their thoughts and questions, asking what you think. You’re in a unique position, simultaneously privy to the most secret potential plans of many powerful rulers. You really know what’s going on. You know what’s likely to succeed or to fail, and why. You get to give the thumbs up or thumbs down. By virtue of your position and the information flowing through your temple, you can direct traffic; you can shape and create history. You might even be tempted to profit from your knowledge. Your successful accurate pronouncements invariably reap you rich tribute.

OK, you can see where this is leading…

Sequoia Capital, and other well-known venture firms, have a somewhat similar position. They have thousands of leaders and wannabee leaders bringing them their detailed secret plans, proposing to mount armies, found cities, build empires, to attack the modern-day Persians, etc. By virtue of their unusual position they probably have a pretty good idea of what might work, and why. Using this knowledge, but without necessarily revealing sources, they can cryptically but assuredly state “oh, that’ll never work” or they can encourage ideas that are new and which they can see will somehow fit and succeed. If company X has consulted the oracle, disclosing a detailed plan to go left, and company Y plans to attack from the right, well…. why not?

Entrepreneurs beg an audience, get a tiny slice of time to make their pitch, and occasionally receive rare clear endorsements. Much more frequently they are left to scratch their heads over cryptic, ambiguous and unexplained responses (and non-responses). You can bet the Delphic oracle didn’t sign NDAs either.

It’s stretching it too far to seriously claim that Sequoia is the modern-day equivalent of the Delphic oracle. But on the other hand, over 2500 years have elapsed, so you’d expect a few changes.

I’ve spent a lot of time thinking about Twitter this year. Here are a few thoughts at random.

Obviously Twitter have tapped into something quite fundamental, which at a high level we might simply call human sociability. We humans are primates, though there’s a remarkably strong tendency to forget or ignore this. We know a lot about the intensely social lives of our fellow primate species. It shouldn’t come as a surprise that we like to Twitter amongst ourselves too.

Here are a couple of interesting (to me) reasons for the popularity of Twitter.

One is that many people are in some sense atomized by the fact that many of us now work in an isolated way. Technical people who can do their work and communicate over the internet probably see less of their peers than others do. That’s just a general point, it’s not specific to Twitter or to 2008. It would have seemed unfathomably odd to humans 50 years ago to hear that many of us would be doing a large percentage of our work and social communication via machines, interacting with people who we don’t otherwise know, and who we rarely or never meet face to face. The rise of internet-based communication is obviously(?) helping to fill a gap created by this generational change.

The second point is specific to Twitter. Through brilliance or accident, the form of communication on Twitter is really special. Building a social network on nothing-implied asymmetric follower relationships is not something I would have predicted as leading to success. Maybe it worked, or could have all gone wrong, just due to random chance. But I’m inclined to believe that there’s more to it than that. Perhaps we’re all secretly voyeurs, or stickybeaks (nosy-parkers). Perhaps we like to see one half of conversations and be able to follow along if we like. Perhaps there’s a small secret thrill to promiscuously following someone and seeing if they follow you back. I don’t know the answer, but as I said above I do think Twitter have tapped into something interesting and strong here. There’s a property of us, we simple primates, that the Twitter model has managed to latch onto.

I think Twitter should change the dynamics for new users by initially assigning them ten random followers. New users can easily follow others, but if no-one is following them….. why bother? New user uptake would be much higher if they didn’t have the (correct) feeling that they were for some reason expected to want to Twitter in a vacuum. You announce a new program, called e.g., Twitter Guides and ask for people to volunteer to be guides (i.e., followers) of newbees. Lend a hand, make new friends, maybe get some followers yourself, etc. Lots of people would click to be a Guide. I bet this would change Twitter’s adoption dynamics. If you study things like random graph theory and dynamic systems, you know that making small changes to (especially initial) probabilities can have a dramatic effect on overall structure. If Twitter is eventually to reach a mass audience (whatever that means), it should be an uncontestable assertion that anything which significantly reduces the difficulty for new users to get into using it is very important.

Twitter should probably fix their reliability issues sometime soon.

I say “probably” because reliability and scaling are obviously not the most important things. Twitter has great value. It must have, or it would have lost its users long ago.

There’s a positive side to Twitter’s unreliability. People are amazed that the site goes down so often. Twitter gets snarled up in ways that give rise to a wide variety of symptoms. The result seems to be more attention, to make the service somehow more charming. It’s like a bad movie that you remember long afterwards because it wasn’t good. We don’t take Twitter for granted and move on the next service to pop up – we’re all busy standing around making snide remarks, playing armchair engineer, knowing that we too might face some of these issues, and talking, talking, talking. Twitter is a fascinating sight. Great harm is done by its unreliability, but the fact that their success so completely flies in the face of conventional wisdom is fascinating – and the fact that we find it so interesting and compelling a spectacle is fantastic for Twitter. They can fix the scaling issues, I hope. They should prove temporary. But the human side of Twitter, its character as a site, the site we stuck with and rooted for when times were so tough, the amazing little site that dropped to the canvas umpteen times but always got back to its feet, etc…. All that is permanent. If Twitter make it, they’re going to be more than just a web service. The public outages are like a rock musician or movie star doing something outrageous or threatening suicide – capturing attention. We’re drawn to the spectacle and the drama. We can’t help ourselves: it is our selves. We love it, we hate it, it brings us together to gnash our teeth when it’s down. But do we leave? Change the channel? No way.

Twitter is both the temperamental child rock star we love and, often, the medium by which we discuss it – an enviable position!

I’m reminded of a trick I learned during tens of thousands of miles of hitch-hiking. A great place to try for a lift is on a fairly high-speed curve on the on-ramp to the freeway / motorway / autopista / autoroute etc. Stand somewhere where a speeding car can only just manage a stop and only just manage to pull in away from the following traffic. Conventional wisdom tells you that you’ll never get a ride. But the opposite is true – you’ll get a ride extremely quickly. Invariably, the first thing the driver says when you get in is “Why on earth where you standing there? You’re very lucky I managed to stop. No-one would have ever picked you up standing there!” I’ve done this dozens of times. Twitter—being incredibly, unbelievably, frustratingly, unreliable and running contrary to all received wisdom—is a powerful spectacle. Human psyche is a funny thing. That’s a part of why it’s probably impossible to foretell success when mass adoption is required.

If I were running Twitter, apart from working to get the service to be more reliable, I’d be telling the engineering team to log everything. There’s a ton of value in the data flowing into Twitter.

Just as Google took internet search to a new level by link analysis, there’s another level of value in Twitter that I don’t think has really begun to be tapped yet.

PageRank, at least as I understand its early operation, ran a kind of iterative relaxation algorithm assigning and passing on credit via linked pages. A similar thing is clearly possible with Twitter, and some people have commented on this or tried to build little things that assign some form of score to users. But I think there’s a lot more that can be done. Because the Twitter API isn’t that powerful (mainly because you’re largely limited to querying as a single authorized user) and certainly because it’s rate-limited to just 70 API calls an hour, this sort of analysis will need to be done by Twitter themselves. I’m sure they’re well aware of that. Rate limiting probably helps them stay up, but it also means that the truly interesting and valuable stuff can’t be done by outsiders. I have no beef with that – I just wish Twitter would hurry up and do some of it.

Some examples in no order:

• The followers to following ratio of a Twitter user is obviously a high-level measure of that user’s “importance” (in some Twitter sense of importance). But there’s more to it than that. Who are the followers? Who do they follow, who follows them? Etc. This leads immediately back to Google PageRank.
• If a user gets followed by many people and doesn’t follow those people back, what does it say about the people involved? If X follows Y and Y then goes to look at a few pages of X’s history but does not then follow X, what do we know?
• If X has 5K followers and re-tweets a twit of Y, how many of X’s followers go check out and perhaps follow Y? What kind of people are these? (How do you advertise to them, versus others?)
• Along the lines of co-citation analysis, Twitter could build up a map showing you who you might follow. I.e., you can get pairwise distances between users X and Y by considering how many people they follow in common and how many they follow not-in-common. That would lead to a people you should be following that you’re not kind of suggestion.
• Even without co-citation analysis (or similar), Twitter should be able to tell me about people that many of the people I follow are following but whom I am not following. I’d find that very useful.
• Twitter could tell me why someone chooses to follow me. What were they looking at (if anything) before they decided to follow me? I.e., were they browsing the following list of someone else? Did they see my user name mentioned in a Tweet? Did they come in from an outside link? Would a premium Twitter user pay to have that information?
• Twitter has tons of links. They know the news as it happens. They could easily create a news site like Digg.
• In some sense the long tail of Twitter is where the value is. For instance, it doesn’t mean much if a user following 10K others follows someone. But if someone is following just 10 people, it’s much more significant. There’s more information there (probably). The Twitter mega users are in some way uninteresting – the more people they have following them and the more they follow, the less you really know (or care) about them. Yes, you could probably figure out more if you really wanted to, but if someone has 10K followers all you really know is that they’re probably famous in some way. If they add another 100 followers it’s no big deal. (I say all this a bit lightly and generally – the details might of course be fascinating and revealing – e.g., if you notice Jason Calacanis and Dave Winer have suddenly started @ messaging each other again it’s like IRC coming back from a network split :-))
• Similarly if someone with a very high followers to following ratio follows a Twitter user who has just a couple of followers, it’s a safe bet that those two are somehow friends with a pre-existing relationship.
• I bet you could do a pretty good job of putting Twitter users into boxes just based on their overall behavior, something like the 16 Myers-Briggs categories. Do you follow people back when they follow you? Do you @ answer people who @ address you (and Twitter knows when you’ve seen the original message)? Do you send @ messages to people (and how influential are those people)? Do those people @ you back (and how influential those people are says something about how interesting / provocative you are)? Do you follow tons and tons of people? Do you follow people and then un-follow them if they don’t follow you back? Do you follow random links in other people’s Twitters, and are those links accompanied by descriptive text or tinyurl links? Do you @ message people after you follow their links? Do your Twitter times follow a strict pattern, or are you on at all hours, or suddenly spending days without Twittering? Do you visit and just read much more than you tweet? How much old stuff do you read? Do you tend to talk in public or via DM? Are your tweets public?All that without even considering the content of your Twitters.
• Could Twitter become a search engine? That’s not a 100% serious question, but it’s worth considering. I don’t mean just making the content of all tweet searchable, I mean it with some sort of ranking algorithm, again perhaps akin to PageRank. If you somehow rank results by the importance or closeness of the user whose tweets match the search terms, you might have something interesting.
• Twitter also presumably know who’s talking about whom in the DM backchat. They can’t use that information in obvious way, but it’s of high value.

I could go on for hours, but that’s more than enough for now. I don’t feel like any of the above list is particularly compelling, but I do think the list of nice things they could be doing is extremely long and that Twitter have only just begun (at least publicly) to tap into the value they’re sitting on.

I think Google should buy Twitter. They have what Twitter needs: 1) engineering and scale, 2) link analysis and algorithm brilliance, and 3) they’re in a position to monetize the value illustrated above (via their search engine, that already has ads) without pissing off the Twitter community by e.g., running ads on Twitter. What percentage of Twitter users also use Google? I bet it’s very high.

I should be coding, not blogging. But a friend noticed I hadn’t blogged in a month, so in lieu of emailing people, here are a couple of comments on programming in Python. There are many things that could be said, but I just want to make two points that I think aren’t so obvious.

1. Python looks great

In Python, indentation is used to delimit code blocks. I like that a lot – you would indent your code anyway, right? It reduces clutter. But apart from that, Python is very minimalistic in its syntax. There are rather few punctuation symbols used, and they’re used pretty consistently. As a result, Python code looks great on the page. It’s not painful to edit, and I mean that figuratively and literally. This is worth noting because when you write complex code it’s nice if the language you’re doing it in is very clean. That’s important because code can become hard to understand and unpleasant to work with. If you have pieces of code that you dread touching, that may be in part because the code is really ugly and complex on the page. Perl is a case in point – there’s tons of punctuation symbols, and in some cases the same thing (e.g., curly braces) is used in multiple (about 5!) different ways to mean different things. If the language is pleasant to look at for longer, you are more willing to work on code that might be more forbidding when expressed in other languages. Esthetics is important. Actively enjoying looking at code simply because the language is so clean is a great advantage—for you, and for the language.

This might not seem like a big point, but it’s important to me, it’s something I’ve never encountered before, and it’s a nice property of Python. BTW, people always make fun of Lisp for its parentheses. But Lisp is the cleanest language I know of in terms of simplicity on the page. The parens and using prefix operators in S-expressions removes the need for almost all other punctuation (and makes programmatically generating code an absolute breeze).

2. Python stays wet longer

I don’t like to do too much formal planning of code. I much prefer to sit down and try writing something to see how it fits. That means I’ll often go through several iterations of code design before I reach the point where I’m happy. Sometimes this is an inefficient way to do things, particularly when you’re working on something very complex that you don’t really have your head around when you start. But I still choose to do things this way because it’s fun.

Sometimes I think of it like pottery. You grab a lump of wet clay and slap it down on the wheel. Then you try out various ideas to shape whatever it is you’re trying to create. If it doesn’t work, you re-shape it—perhaps from scratch. This isn’t a very accurate analogy, but I do think it’s valid to say that preferring to work with real code in an attempt to understand how best to shape your ideas is a much more physical process than trying to spec everything out sans code. I find I can’t know if code to implement an idea or solve a problem is going to feel right unless I physically play with it in different forms.

For me, Python stays wet longer. I can re-shape my code really easily in Python. In other languages I’ve often found myself in a position where a re-design of some aspect involves lots of work. In Python the opposite has been true, and that’s a real pleasure. When you realize you should be doing things differently and it’s just a bit of quick editing to re-organize things, you notice. I might gradually be becoming a better programmer, but I mainly feel that in using Python I simply have better quality clay.

I’d never even heard of a bandoneón before last Thursday.

There was a performance of the guitar class at the kids’ school and after they were done the two teachers played for us all. It was great. The woman on the right playing the bandoneón is Argentinian and has been in Barcelona for 3 months. She’s been learning the bandoneón for 5 years and playing guitar “all my life”. As for the bandoneón, it looks good, sounds good, and doesn’t sound easy to learn. Here’s an excerpt from the Wikipedia page linked above:

Unlike the piano accordion, the bandoneón does not have keyboards per se, but has buttons on both sides; and also unlike most accordions, most buttons on the bandoneón produce a different note when played closing than when played opening. This means that each keyboard has actually two layouts – one for the opening notes, and one for the closing notes. Since the right and left hand keyboards are also different, this adds up to four different keyboard layouts that must be learned in order to play the instrument.

Check it out:

[Listening just now to the Tiger Lillies song Bankrobber Blues I decided to post another little fictional story I wrote some years ago. The first in the series was Lucky Streak. As a kid I had a head full of criminal schemes and at times was convinced I’d inevitably grow up to be a cat burglar. Mathematician by day, cat burglar by night. It didn’t pan out that way.]

Manhole

I recently became a photographer. Before high-quality digital cameras became available, I had never had an interest. Sure, I liked photos as much as the next person, but it was all so messy. So imprecise. So… analog. I mean why bother? I’m into digital music in a way I could never have been with tapes or vinyl. Why do anything else? Nostalgia? Tradition? Gimme a break. Analog devices with linear access? Puhlease. I keep an eye on things, I have a feel for this stuff. When technology crosses some line, it’s time to buy in. Before that it’s just too messy.

I recently became a criminal. Historically, crime at a distance was always a challenge. Physical events took place, objects were exchanged, people met or talked on the phone. Messy, way too messy. There was always a chain, a trail: someone had met the guy, someone had given someone else a brown paper bag stuffed with cash and a pistol. With time and money, the feds were eventually going to catch you. You couldn’t find good help, couldn’t recruit and train them, had to trust them, were betrayed or sold out, either to the cops or a rival. Whatever. It made no sense, unless you were the mob and you paid everyone off. Or bumped them off. Messy.

The internet changed all that. No government wants to allow anonymous action, but the net was too big. For nearly twenty years I have watched and been amazed that they didn’t do something to stop it. Now it’s too late, far too late. Anonymous action is possible. It’s not straightforward, one must be careful, but it can be done. Tools like mixmaster (anonymous remailer), spamgourmet (similar), the anonymizer (anonymous browsing), ssh tunnels, cheap and disposable machines for rent on the network running simple forwarding servers, paypal for anonymous transfers of cash, public auction sites for posting messages to those who know what to look for, steganography, instant messaging, strong crypto, wide open windows boxes (with broadband connections and fixed IP addresses) just waiting to be commandeered, etc. The tools are all out there and in most cases they’re freely available. With some knowledge, one can carefully build an action network that no-one’s ever going to trace. The whole point of some of these tools is to preserve anonymity. No logs are kept. Subpoenas are useless when there simply are no records.

Just to warm up, I robbed a bank. Not a white-collar electronic robbery. I’m talking blue-collar physical bank robbery. It’s hardly a well-paying profession, but it’s a good lark. In fact, it pays rather badly. It’s also increasingly common and easier, as tellers and other bank staff are instructed to do exactly as told, not to make a fuss, etc. So why rob a bank? Well, why take a dumbass picture of a tree or your pet dog with your new digital camera? Why? Because you have the technology and you want to try it out. Because you can. Because you’ve gone digital and you can take a no-cost snap of whatever you damned well please.

I, physically, robbed a bank. That’s messy, true. But at some point I wanted skin in the game, wanted the whole thing to be real, to prove that it all worked. Plus, I can’t solve everything at once: I didn’t want someone else actually robbing the bank. That way, I’d either never see the money or I’d leave a trail that could be followed. So I planned carefully and I did it myself.

I recruited a dozen people online. That was actually the easy part. There’s any number of bored and broke college kids out there on the net. I made two roles: watchers and actors. I had actors do various silly extroverted tasks and each time had a watcher turn up to verify that the actor showed up. Everyone got different instructions and everyone was told enough and reported enough back to me so that a) actors knew they were being watched and b) I knew the test tasks were in fact getting done. I paid on task completion and verification. An actor would be instructed to do something oddball and very specific, like walk into a 7-eleven at a certain time in the middle of summer, buy six dozen hotdogs and a case of Bud Lite and comment to the cashier that it sure did look like it might snow. I’d schedule a watcher, and tell them to look out for the person who bought six dozen hotdogs and said how it looked like snowing. The watcher would be told to pay special attention to the other purchases and to the shoes of the purchaser. Afterwards, when I heard back from both, I’d ask the watcher what the other purchase was and tell the actor what sort of shoes they were wearing. So the actor knew that I was in the loop and I could verify that the watcher had been there to verify. Everyone had to do their part or I’d know. In this way, I built up the confidence of my players: they knew that the game was for real and that, for whatever reason, if they just did some seemingly random and simple acts from time to time, they’d be paid. I let the actors and watchers take turns at both roles. If they ever asked what the deal was I just told them I was a university psych professor experimenting on social norms.

I admit this took some time. Two months. I had a few dropouts: people who found it too weird, people who tried to cheat by not turning up while claiming they did, people who just dropped out of contact.

When I was ready, I gave all my little helpers an identical task, each at a different bank. Each was to walk into their appointed bank, get in line for a teller, hand the teller a note that said they had a gun and to hand over all the cash from the drawer. Having done that, the actor was to turn and walk straight out of the bank. No watchers, just twelve people playing actors. Everyone dressed the same. I figured several would drop out at this stage, but on the other hand I’d had them doing some pretty silly stuff, pushing the edge, and I knew at least half of them would be up for it. I also know a thing or two about banks. When the note gets to the teller, the first thing they’ll do is trigger the silent alarm. The security guard in the bank will still be clueless, but the bank manager will see it and so will the cops, who are directly connected.

The banks are dotted around midtown Manhattan. The area’s thick with banks and crazy with traffic. Between 11:00 and 11:05 on July 2, the day before the holiday weekend break, the direct-connect police alarm systems of the 5th and 6th precincts go nuts. Cops cars are scrambled and routed to what turns out to be nine simultaneous false-alarm bank robberies. And one real robbery.

I walk into my bank at ten minutes after eleven. I’m dressed in a red shirt, just like my actors who are busy triggering the alarms in a dozen other midtown banks. After a few minutes I reach the teller and hand her a note: “I have a gun. Keep your mouth shut. Pass me all your cash.” After a pause, she does this without a word. That’s the training, the banks learned that one a long time ago. A hero teller or guard taking a bullet winds up costing more than dozens of unsophisticated heists. Plus the banks are fully insured. So teller training is explicit: trigger the silent alarm and do anything the nasty bank robber tells you to.

I sweep the cash into my bag, there couldn’t have been more than $10,000 probably closer to $5,000. I hand her two more notes: “The client at my window has a gun. Keep your mouth shut. Pass all the cash from your drawer to your right.” The other is identical, for the other direction. I indicate that she should pass them the notes, which she does. Handfuls of cash are passed from one to another between five tellers, under the glass to me, where I drop them into my bag. Other customers look on, and I guess they figure something odd is happening, but of course no-one says anything.

Start to finish, I’m there putting money into my bag for less than two minutes. Up until the point the first cash hit my bag, I had a pretty good story: some random person on the internet put me up to it, got me to do all this weird shit, watch others do weird random stuff, paid me through paypal. With the money in my bag though, the game has changed. I know the silent alarm was probably tripped in the first thirty seconds. With your typical mid-morning midtown traffic chaos and the cops already dispatched to at least half a dozen banks apparently being robbed simultaneously, I figure they may not get here for twenty minutes. I’m not counting on the delay any more than I have to, but I figure my margin for error is pretty healthy.

I walk out the front door. Fifteen steps from the bank entrance is a convenient manhole. And look… the cover is already off. The hole is surrounded by four orange cones connected by official-looking Men At Work tape. Of course this is my doing: started and finished a few minutes before entering the bank. From my bag I take a hard hat, some old gloves, and an orange reflective sleeveless vest. Standard issue street crew garb, bought for a few bucks here and there at thrift stores. I pick up the steel tool used for removing manhole covers and drop it into the hole. The rest of my props I leave for the cops. I climb down into the hole and pull the cover across on top of me. Although the covers are heavy, putting one in place is actually a surprisingly easy job when done from below.

The sewers of Manhattan are pretty ugly, despite large-scale efforts to sanitize and modernize them in the 90s. But there’s no need to stay down for long in order to simply disappear. I’ve walked this route before a few times. It’s no big deal. How do I know all this? The plans are on the net: obtained and published online by some urban spelunking group. Along the way I remove the red shirt and put it in my bag. A little over six minutes later I emerge from another manhole in a lane behind a hotel. A couple of suits are standing outside having a smoke. Nothing if not brazen, I bludge a cigarette and a light from one of them. He’s only too happy to help a working man in a hard hat. Show his mates he’s on good terms with blue collar America. With a false mustache, non-prescription glasses, a vest, and the hard hat, there’s not going to be much to go on later, even if these guys do wind up talking to the cops. Besides, as is well known, witnesses are wonderfully unreliable, so the more the merrier as far as I’m concerned. You’ve got to know how these things work and use them to your advantage. I tell the suits I’m on a break from the sewer, got to go to my bank, and off I slouch. Around the first corner I remove the working gear and it goes into my bag too. By my reckoning, the cops wont have even gone down the manhole outside the bank. At most they’ll have a call in for someone to remove the cover. They’ll also know that I’m probably no longer underground.

I walk down into the subway and head home to count my cash ($47,000 and change) and to catch the headline news: Daring daylight robbery! Criminal mastermind! Nine (only nine) banks used to create a distraction while one was well and truly robbed. The police chief is interviewed, indignant: we’ll hunt these men down and put them where they belong. A sophisticated operation like this isn’t done in a vacuum.

Well, good luck chief. For me this is just the beginning, just a proof of concept really. No-one should have to rob banks for a living. That kind of crime doesn’t pay well enough, even if you never get caught. Of course, I have other plans. As the world becomes more digital, less messy, things will only get easier for those who have the balls.

I’m a big fan of Google maps.

But sometimes they get things very very wrong. In January I posted this example of them getting the location of the San Francisco international airport way wrong.

The screenshot linked above is supposed to show the location of the hotel Princesa Sofia in Barcelona. They have the address right, the zip code looks about right, but the location is about 30 miles off.

Caveat turista.