Because it’s emacs, it wasn’t hard to write a function to display a buffer mode histogram. Here’s mine:

235 buffers open, in 23 distinct modes

91               python +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
47          fundamental +++++++++++++++++++++++++++++++++++++++++++++++
24                  js2 ++++++++++++++++++++++++
21                dired +++++++++++++++++++++
16                 html ++++++++++++++++
 7                 text +++++++
 4                 help ++++
 4           emacs-lisp ++++
 3                   sh +++
 3       makefile-gmake +++
 2          compilation ++
 2                  css ++
 1          Buffer-menu +
 1                 mail +
 1                 grep +
 1      completion-list +
 1                   vm +
 1                  org +
 1               comint +
 1              apropos +
 1                 Info +
 1           vm-summary +
 1      vm-presentation +

Tempting as it is, I’m not going to go on about the heady delights of having a fully programmable editor. You either already know, or you can just drool in slack-jawed wonder.

Unfortunately I’m a terrible emacs lisp programmer. I can barely remember a thing each time I use it. But the interpreter is of course just emacs itself and the elisp documentation is in emacs, so it’s a really fun environment to develop in. And because emacs lisp has a ton of support for doing things to itself, code that acts on emacs and your own editing session or buffers is often very succinct. See for example the save-excursion and with-output-to-temp-buffer functions below.

Various things about the formatting could be improved. E.g., not use fixed-width fields for the count and the mode names, and make the + signs indicate more than one buffer mode when there are many.

Image: internetworldstats.com

For example, consider Moore’s Law, which describes how "the number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years." If something doubles every two years, at what rate does it increase per month, on average? If you know the rule of 72, you’ll instantly know that the monthly growth rate is about 3%. You get the answer by dividing 72 by 24 (the number of months).

Computer scientists are usually very familiar with powers of two. It’s often convenient to take advantage of the fact that 2^10 is about 1,000. That means that when something increases by a factor of 1,000, it has doubled about 10 times. By extension, and with a little more error, an increase of a million corresponds to 20 doublings, and a billion is 30 doublings (log base two of a billion is actually 29.897, so the error isn’t too wild). You can use this to ballpark the number of doublings in a process really easily, and go directly from that to a growth rate using the rule of 72.

For example, the bottom of this page tells us that there were about 16,000 internet domains on July 1st 1992, and 1.3M of them on July 1st 1997. Let’s think in thousands: that’s a jump from 16 to just over 1,000 in 5 years. To get from 1 to 16 is four doublings, so from 16 to 1,000 is six doublings (because 1,000 is ten doublings from 1). So the number of domains doubled 6 times in 5 years, or 6 times in 60 months, or once every 10 months (on average). If you want something to double in 10 months, the rule of 72 tells us we need a growth rate of 7.2% per month. To check: 16,000 * (1.072 ^ 60) = 1,037,067. That’s a damned good estimate (remember that we were shooting for 1M, not 1.3M) for five seconds of mental arithmetic! Note that the number of domains was growing much faster than Moore’s law (3% per month).

You can quickly get very good at doing these sorts of calculations. Here’s another easy example. This page shows the number of internet users growing from 16M in December 1995 to 2,072M in March of 2011. That’s just like the above example, but it’s 7 doublings in 15.25 years, or 183 months. That’s pretty close to a doubling every 24 months, which we know from above corresponds to 3% growth per month.

You can use facility with growth rates to have a good sense for interest rates in general. You can use it when building simple (exponential) models of product growth. E.g., suppose you’re launching a product and you reckon you’ll have 300K users in a year’s time. You want to map this out in a spreadsheet using a simple exponential model. What should the growth rate be? 300K is obviously not much more than 256 * 1,024, which is 18 doublings in 365 days, or a doubling roughly every 20 days. The rule of 72 gives 72/20 = ~3.5, so you need to grow 3.5% every day to hit your target. Is that reasonable? If it is, it means that when you hit 300K users, you’ll be signing up about 3.5% of that number, or 10,500 users per day. As you can see, familiarity with powers of two (i.e., estimating number of doublings) and with the rule of 72 can give you ballpark figures really easily. You can even use your new math powers to avoid looking stupid in front of VCs.

The math behind the rule of 72 is easy to extend to triplings (rule of 110), quadrupling (rule of 140), quintupling (rule of 160), etc.

Finally, you can use these rules of thumb to do super geeky party tricks. E.g., what’s the tenth root of two? Put another way, what interest rate do you need for something to double after ten periods? The rule of 72 tells you it’s 72/10 = 7.2%, so the tenth root of two will be about 1.072 (in fact 1.072 ^ 10 = 2.004). What’s the 20th root of 5? The rule of 160 tells you you need 160/20 = 8% growth each period, so 1.08 should be about right (the correct answer is ~1.0838).

As with all rules of thumb, it’s good to have a sense of when it’s most applicable. See the wikipedia page or this page for more detailed information. It’s also of course good to understand that it may not be suitable to model growth as an exponential at all.

  4 = 2^2                  729 = 3^6, 9^3                32768 = 2^15, 8^5
  8 = 2^3                 1000 = 10^3                    38416 = 14^4
  9 = 3^2                 1024 = 2^10, 4^5               46656 = 6^6
 16 = 2^4, 4^2            1296 = 6^4                     50625 = 15^4
 25 = 5^2                 1331 = 11^3                    59049 = 3^10, 9^5
 27 = 3^3                 1728 = 12^3                    65536 = 2^16, 4^8, 16^4
 32 = 2^5                 2048 = 2^11                    78125 = 5^7
 36 = 6^2                 2187 = 3^7                     83521 = 17^4
 49 = 7^2                 2197 = 13^3                   100000 = 10^5
 64 = 2^6, 4^3, 8^2       2401 = 7^4                    104976 = 18^4
 81 = 3^4, 9^2            2744 = 14^3                   117649 = 7^6
100 = 10^2                3125 = 5^5                    130321 = 19^4
121 = 11^2                3375 = 15^3                   131072 = 2^17
125 = 5^3                 4096 = 2^12, 4^6, 8^4, 16^3   160000 = 20^4
128 = 2^7                 4913 = 17^3                   161051 = 11^5
144 = 12^2                5832 = 18^3                   177147 = 3^11
169 = 13^2                6561 = 3^8, 9^4               248832 = 12^5
196 = 14^2                6859 = 19^3                   262144 = 2^18, 4^9, 8^6
216 = 6^3                 7776 = 6^5                    279936 = 6^7
225 = 15^2                8000 = 20^3                   371293 = 13^5
243 = 3^5                 8192 = 2^13                   390625 = 5^8
256 = 2^8, 4^4, 16^2     10000 = 10^4                   524288 = 2^19
289 = 17^2               14641 = 11^4                   531441 = 3^12, 9^6
324 = 18^2               15625 = 5^6                    537824 = 14^5
343 = 7^3                16384 = 2^14, 4^7              759375 = 15^5
361 = 19^2               16807 = 7^5                    823543 = 7^7
400 = 20^2               19683 = 3^9                   1000000 = 10^6
512 = 2^9, 8^3           20736 = 12^4                  1048576 = 2^20, 4^10, 16^5
625 = 5^4                28561 = 13^4

I produced the above with this quick hack:

For example, let’s start with Union Square in New York. Fully zoomed in we can see Coffee Shop, so that’s at distance 0. Those things are about a minute’s walk from the center of Union Square. Zooming out one click, we pick up Bowlmor Lanes on University Place and the Whole Foods Market at the South of Union Square. Those things are at GMZD-1. The outer edge of distance 1 is about a few minute’s walk from Union Square. Continuing outwards, Betaworks is at GMZD-4, Central Park at distance 6, Boston and Niagara Falls at distance 12, Florida, Winnipeg, and St John’s, Newfoundland at 14, San Francisco at 15, Barcelona at 16, and Sydney at 17. (Although you can zoom out 18 levels, GMZD-17 seems to be as many as you practically need to see anything.)

You can also think of GMZD as half the number of clicks you’d need to do on Google maps to go from being fully zoomed in on A to being fully zoomed in on B (with some panning in between). When you look at Google maps you can count the number of notches on the little slider (see image on left) above your current zoom level to see the GMZD from the center of the visible map to its outer edges.

Update: I meant to mention that GMZD is not a formal distance metric. It is non-negative (GMZD(A, B) >= 0) and symmetric (GMZD(A, B) = GMZD(B, A)) for all points A and B, but distinct points can be at distance zero and (as a result) the triangle inequality also does not hold (e.g., Union Square is distance zero from Coffee Shop, and Coffee Shop is distance zero from Union Square Cafe, but the distance from Union Square to Union Square Cafe is one. Not being a metric space is what makes it interesting, though :-)

I had a really good call with AAA. He told me he’s interested and wants to talk to BBB and CCC. I then got mail the next day from DDD (of the NYT) who told me he’d just had dinner with AAA and BBB and that they’d talked about my stuff. So something may happen there (i.e., I’ll never hear from them again).

The last comment, that I’d probably never hear from them again, was entirely tongue-in-cheek. I wrote it knowing it was a possibility, but not really thinking it would happen.

But it did.

Things like that seem to be part & parcel of the startup world as you attempt to get funded. I have often asked myself how can it be possible for things to be this way? How you can have people so excited, telling you and others you’re going to change the world, be worth billions, and then you never hear from them again? (Yes, of course you have to follow up, and I did. But that’s not the point: If you didn’t follow up you’d never hear from them.)

How can that be? In what sort of world is such a thing possible?

I came up with a highly flawed analogy. Despite its limited accuracy I find it amusing and can’t resist blogging it even if people will label me bitter (I’m not).

First: startup founders are pond scum. Second: potential investors are a troupe of young kids wandering through the park with sticks.

The kids poke into the ponds, stirring up the scum. They’re looking for cool things, signs of life, perhaps even something to take home. They’re genuinely interested. They’re fascinated. The pond scum listen to their excited conversation and think the kids will surely be back tomorrow. But it’s summer, and the world is so very very big.

The pond scum are working on little projects like photosynthesis, enhancements to the Krebs cycle, or the creation of life itself. All the while they’re pondering how to make themselves irresistible, believing that someday the kids with the sticks will be back, that they’ll eventually be scooped up.

As Paul Graham recently wrote, fundraising is brutal. His #1 recommendation is to keep expectations low.

Yep, you’re pond scum.

Get used to it.

Embrace it.

I talked to a man washing his car in his driveway. He told me that “Patrick” had created all the figures, and installed them on the front lawns. I got the impression that it was all free. Soon after I found the house that was unmistakably Patrick’s and seeing a man loading things into a pickup truck I went up and asked if he was Patrick. It was him and we had a friendly talk (mainly me telling him he was amazing). He gave me a calendar of his work.

Click on the thumbnails below to see bigger versions. There’s even a FC Barcelona structure. As I found out later, lots of people (of course) have seen these sculptures. When I got to Foo, there was one (image above) outside the O’Reilly office. Google for Patrick Amiot or Florence Avenue, Sebastopol and you’ll find much more. And Patrick has his own web site.

I started wondering how random the URLs from these URL-shortening services could be. I wrote a tiny script the other day to turn URLs given on the command line into short URLs via is.gd:

I ran it a couple of times to see what URLs it generated. Note that you have to use a new URL each time, as it’s smart enough not to give out a new short URL for one it has seen before. I got the sequence http://is.gd/JzB, http://is.gd/JzC, http://is.gd/JzD, http://is.gd/JzE,…

That’s an invitation to some minor mischief, because you can guess the next URL in the is.gd sequence before it’s actually assigned to redirect somewhere.

We can ask bit.ly for a short URL that redirects to our predicted next is.gd URL. Then we ask is.gd for a short URL that redirects to the URL that bit.ly gives us. If we do this fast enough, is.gd will not yet have assigned the predicted next URL and we’ll get it. So the bit.ly URL will end up redirecting to the is.gd URL and vice versa. In ugly Python (and with a bug/shortcoming in the nextIsgd function):

This worked first time, giving:

Step 1: bit.ly now redirects http://bit.ly/fkuL8 to http://is.gd/JA9
Step 2: is.gd now redirects http://is.gd/JA9 to http://bit.ly/fkuL8

In general it’s not a good idea to use predictable numbers like this, which hardly bears saying as just about every responsible programmer knows that already.

is.gd wont shorten a tinyurl.com link, as tinyurl is on their blacklist. So they obviously know what they’re doing. The bit.ly service is brand new and presumably not on the is.gd radar yet.

And finally, what happens when you visit one of the deadly looping redirect URLs in your browser? You’d hope that after all these years the browser would detect the redirect loop and break it at some point. And that’s what happened with Firefox 3, producing the image above.

If you want to give it a try, http://bit.ly/fkuL8 and http://is.gd/JA9 point to each other. Do I need to add that I’m not responsible if your browser explodes in your face?

Manhole

I recently became a photographer. Before high-quality digital cameras became available, I had never had an interest. Sure, I liked photos as much as the next person, but it was all so messy. So imprecise. So… analog. I mean why bother? I’m into digital music in a way I could never have been with tapes or vinyl. Why do anything else? Nostalgia? Tradition? Gimme a break. Analog devices with linear access? Puhlease. I keep an eye on things, I have a feel for this stuff. When technology crosses some line, it’s time to buy in. Before that it’s just too messy.

I recently became a criminal. Historically, crime at a distance was always a challenge. Physical events took place, objects were exchanged, people met or talked on the phone. Messy, way too messy. There was always a chain, a trail: someone had met the guy, someone had given someone else a brown paper bag stuffed with cash and a pistol. With time and money, the feds were eventually going to catch you. You couldn’t find good help, couldn’t recruit and train them, had to trust them, were betrayed or sold out, either to the cops or a rival. Whatever. It made no sense, unless you were the mob and you paid everyone off. Or bumped them off. Messy.

The internet changed all that. No government wants to allow anonymous action, but the net was too big. For nearly twenty years I have watched and been amazed that they didn’t do something to stop it. Now it’s too late, far too late. Anonymous action is possible. It’s not straightforward, one must be careful, but it can be done. Tools like mixmaster (anonymous remailer), spamgourmet (similar), the anonymizer (anonymous browsing), ssh tunnels, cheap and disposable machines for rent on the network running simple forwarding servers, paypal for anonymous transfers of cash, public auction sites for posting messages to those who know what to look for, steganography, instant messaging, strong crypto, wide open windows boxes (with broadband connections and fixed IP addresses) just waiting to be commandeered, etc. The tools are all out there and in most cases they’re freely available. With some knowledge, one can carefully build an action network that no-one’s ever going to trace. The whole point of some of these tools is to preserve anonymity. No logs are kept. Subpoenas are useless when there simply are no records.

Just to warm up, I robbed a bank. Not a white-collar electronic robbery. I’m talking blue-collar physical bank robbery. It’s hardly a well-paying profession, but it’s a good lark. In fact, it pays rather badly. It’s also increasingly common and easier, as tellers and other bank staff are instructed to do exactly as told, not to make a fuss, etc. So why rob a bank? Well, why take a dumbass picture of a tree or your pet dog with your new digital camera? Why? Because you have the technology and you want to try it out. Because you can. Because you’ve gone digital and you can take a no-cost snap of whatever you damned well please.

I, physically, robbed a bank. That’s messy, true. But at some point I wanted skin in the game, wanted the whole thing to be real, to prove that it all worked. Plus, I can’t solve everything at once: I didn’t want someone else actually robbing the bank. That way, I’d either never see the money or I’d leave a trail that could be followed. So I planned carefully and I did it myself.

I recruited a dozen people online. That was actually the easy part. There’s any number of bored and broke college kids out there on the net. I made two roles: watchers and actors. I had actors do various silly extroverted tasks and each time had a watcher turn up to verify that the actor showed up. Everyone got different instructions and everyone was told enough and reported enough back to me so that a) actors knew they were being watched and b) I knew the test tasks were in fact getting done. I paid on task completion and verification. An actor would be instructed to do something oddball and very specific, like walk into a 7-eleven at a certain time in the middle of summer, buy six dozen hotdogs and a case of Bud Lite and comment to the cashier that it sure did look like it might snow. I’d schedule a watcher, and tell them to look out for the person who bought six dozen hotdogs and said how it looked like snowing. The watcher would be told to pay special attention to the other purchases and to the shoes of the purchaser. Afterwards, when I heard back from both, I’d ask the watcher what the other purchase was and tell the actor what sort of shoes they were wearing. So the actor knew that I was in the loop and I could verify that the watcher had been there to verify. Everyone had to do their part or I’d know. In this way, I built up the confidence of my players: they knew that the game was for real and that, for whatever reason, if they just did some seemingly random and simple acts from time to time, they’d be paid. I let the actors and watchers take turns at both roles. If they ever asked what the deal was I just told them I was a university psych professor experimenting on social norms.

I admit this took some time. Two months. I had a few dropouts: people who found it too weird, people who tried to cheat by not turning up while claiming they did, people who just dropped out of contact.

When I was ready, I gave all my little helpers an identical task, each at a different bank. Each was to walk into their appointed bank, get in line for a teller, hand the teller a note that said they had a gun and to hand over all the cash from the drawer. Having done that, the actor was to turn and walk straight out of the bank. No watchers, just twelve people playing actors. Everyone dressed the same. I figured several would drop out at this stage, but on the other hand I’d had them doing some pretty silly stuff, pushing the edge, and I knew at least half of them would be up for it. I also know a thing or two about banks. When the note gets to the teller, the first thing they’ll do is trigger the silent alarm. The security guard in the bank will still be clueless, but the bank manager will see it and so will the cops, who are directly connected.

The banks are dotted around midtown Manhattan. The area’s thick with banks and crazy with traffic. Between 11:00 and 11:05 on July 2, the day before the holiday weekend break, the direct-connect police alarm systems of the 5th and 6th precincts go nuts. Cops cars are scrambled and routed to what turns out to be nine simultaneous false-alarm bank robberies. And one real robbery.

I walk into my bank at ten minutes after eleven. I’m dressed in a red shirt, just like my actors who are busy triggering the alarms in a dozen other midtown banks. After a few minutes I reach the teller and hand her a note: “I have a gun. Keep your mouth shut. Pass me all your cash.” After a pause, she does this without a word. That’s the training, the banks learned that one a long time ago. A hero teller or guard taking a bullet winds up costing more than dozens of unsophisticated heists. Plus the banks are fully insured. So teller training is explicit: trigger the silent alarm and do anything the nasty bank robber tells you to.

I sweep the cash into my bag, there couldn’t have been more than $10,000 probably closer to $5,000. I hand her two more notes: “The client at my window has a gun. Keep your mouth shut. Pass all the cash from your drawer to your right.” The other is identical, for the other direction. I indicate that she should pass them the notes, which she does. Handfuls of cash are passed from one to another between five tellers, under the glass to me, where I drop them into my bag. Other customers look on, and I guess they figure something odd is happening, but of course no-one says anything.

Start to finish, I’m there putting money into my bag for less than two minutes. Up until the point the first cash hit my bag, I had a pretty good story: some random person on the internet put me up to it, got me to do all this weird shit, watch others do weird random stuff, paid me through paypal. With the money in my bag though, the game has changed. I know the silent alarm was probably tripped in the first thirty seconds. With your typical mid-morning midtown traffic chaos and the cops already dispatched to at least half a dozen banks apparently being robbed simultaneously, I figure they may not get here for twenty minutes. I’m not counting on the delay any more than I have to, but I figure my margin for error is pretty healthy.

I walk out the front door. Fifteen steps from the bank entrance is a convenient manhole. And look… the cover is already off. The hole is surrounded by four orange cones connected by official-looking Men At Work tape. Of course this is my doing: started and finished a few minutes before entering the bank. From my bag I take a hard hat, some old gloves, and an orange reflective sleeveless vest. Standard issue street crew garb, bought for a few bucks here and there at thrift stores. I pick up the steel tool used for removing manhole covers and drop it into the hole. The rest of my props I leave for the cops. I climb down into the hole and pull the cover across on top of me. Although the covers are heavy, putting one in place is actually a surprisingly easy job when done from below.

The sewers of Manhattan are pretty ugly, despite large-scale efforts to sanitize and modernize them in the 90s. But there’s no need to stay down for long in order to simply disappear. I’ve walked this route before a few times. It’s no big deal. How do I know all this? The plans are on the net: obtained and published online by some urban spelunking group. Along the way I remove the red shirt and put it in my bag. A little over six minutes later I emerge from another manhole in a lane behind a hotel. A couple of suits are standing outside having a smoke. Nothing if not brazen, I bludge a cigarette and a light from one of them. He’s only too happy to help a working man in a hard hat. Show his mates he’s on good terms with blue collar America. With a false mustache, non-prescription glasses, a vest, and the hard hat, there’s not going to be much to go on later, even if these guys do wind up talking to the cops. Besides, as is well known, witnesses are wonderfully unreliable, so the more the merrier as far as I’m concerned. You’ve got to know how these things work and use them to your advantage. I tell the suits I’m on a break from the sewer, got to go to my bank, and off I slouch. Around the first corner I remove the working gear and it goes into my bag too. By my reckoning, the cops wont have even gone down the manhole outside the bank. At most they’ll have a call in for someone to remove the cover. They’ll also know that I’m probably no longer underground.

I walk down into the subway and head home to count my cash ($47,000 and change) and to catch the headline news: Daring daylight robbery! Criminal mastermind! Nine (only nine) banks used to create a distraction while one was well and truly robbed. The police chief is interviewed, indignant: we’ll hunt these men down and put them where they belong. A sophisticated operation like this isn’t done in a vacuum.

Well, good luck chief. For me this is just the beginning, just a proof of concept really. No-one should have to rob banks for a living. That kind of crime doesn’t pay well enough, even if you never get caught. Of course, I have other plans. As the world becomes more digital, less messy, things will only get easier for those who have the balls.

Do you know these girls?

The menu on the camera is in German.

You can see why I’m desperate to get them their camera back.

I don’t want to go into details now or here, but I do want to say that we yesterday published a paper in Science. The paper’s title is The Global Circulation of Seasonal Influenza A (H3N2) Viruses. It digs into how flu viruses circulate around the world and what happens to them in the off season (summer) in temperate zones. This paper was years in the making. And if you consider the data collected by the worldwide influenza surveillance network, it has been decades in the making. As a result the paper has 28 authors, many of whom work at the international flu collaborating centers.

Here’s the official paper in Science. There’s a ton of other coverage, including stories in Science Daily, New Scientist, the LA Times, the Washington Post, National Geographic, Times Online, Reuters, the Associated Press, the Wall Street Journal, and Scientific American. There are plenty more links (currently) available at Google News if you search for influenza.

I’m listed as the #2 author, but it’s really my close friends the first and last authors, Colin Russell and Derek Smith, who did the heavy lifting on making this paper a reality. It’s so nice to see the thing finally published and getting such wide attention.

Could someone please give Natalie Jeremijenko a MacArthur “genius” Grant?

Thank you.

I’m not in charge of these things, obviously. If I were though, I’d be hoping to see Natalie nominated so I could give her a grant. She’s a poster child for a genius grant (as is my good friend Derek Smith; but that’s another story). I just spent 2 hours chatting with Natalie in her NYU office.

Here’s a long article about her in Salon.

The MacArthur Foundation moves in mysterious ways. And so does the blogosphere. So I send out this tiny tug on the invisible strings to my invisible readers, asking them to tug in turn. Like many of Natalie’s many projects, a little collective tugging might do wonders.

I don’t mean that literally everything we think is completely wrong. Some things are certainly partly right, or even mainly or fully right. But to have a high degree of confidence, or to assume we’re right just because we know so much more about the world than our ancestors did, or simply because we think we’re right, is just inviting ridicule. Considering our record, and our continual attendant misguided arrogance and confidence along the way, you’d be nuts to think that we know much today or that our confidence adds any weight at all. Many thousands of years of history argue strongly against that conclusion.

Thinking that almost everything is probably wrong in some important fundamental way is a useful default. That attitude stands you in good stead for digging into things, for reconsidering them, for asking questions at a low level. In mathematics when you know for sure that something is wrong (or right) it helps enormously in proving it. It’s a psychological thing. In my dissertation I proved a statistical result that I knew must be true from running simulations. It took me a week or two to nail the proof, and I would never have gotten there if I hadn’t known in advance that the equality I was trying to prove analytically was certainly true (pp 201-207 here in case you’re interested).

As an example of something that I think will be overturned, I think we’ll come to regard our decades of designing computational systems according to the Von Neumann Architecture as extremely primitive. Maybe that will involve some form of analog or quantum computation. I think we’ll take more and more from nature, for instance in solving optimization problems.

On a less grandiose note but still important, I think we’ll look back on our current information architecture and also see it as being extremely primitive. Or, as I’ve said before, we’re living in the shadow of information architecture decisions that were made decades ago. I think that’s all hopelessly wrong. In the real world, information processing simply doesn’t look much like a hierarchical file system.

Hence Fluidinfo.

And so ends another semi-cryptic and ultimately unsatisfying post. I do, as always, plan to eventually say more. And I will.

After lunch, Ted took me up to the cup room. Or the room that used to be the cup room. You see, there’s a slight problem. No cup. The room was specially built to hold the America’s Cup. It’s perfect, and even has a little viewing platform like the prow of a boat. It’s a beautiful space. And it’s totally empty.

I’m not much of one for nationalistic pride. But I couldn’t resist a little twinge of pleasure recalling that fateful day the Australian boat won the cup after the US had held it for 132 years. Bob Hawke, the Australian Prime Minister, appeared on TV in a bright Green and Gold kangaroo-covered jacket to declare that “any boss who fires a worker for not turning up today is a bum”. It was quite a scene. Good for yachting, I should think, just like when the England cricket team finally beat the Australians a few years ago.

Standing there in the exact spot that the America’s Cup had so immovably and confidently occupied for 132 years was really something. You could almost feel the sense of confusion and cognitive dissonance emanating from that empty space and flowing out to unbalance the entire club building. Ted took photos with his iPhone while I thought of Ozymandias, joked with the staff, and tried to sound like I was from somewhere else.

Then it was upstairs to the banquet hall and model room. There are many hundreds of model yachts on the walls and in glass cases. There are perfect models of every boat to win the America’s Cup, and yes I checked out Ben Lexcen‘s famous winged keel. The accompanying plaque was careful to point out that the boat’s measurements were allowed by the rules. Unwritten: the spirit of yachting itself was shamelessly violated by the genius upstart designer from down under, but, strictly speaking, the boat was legal.

It’s quite a sight.

By modern web attention span standards, this is really long.

LUCKY STREAK

They say the first million is always the hardest. In my case it was actually pretty easy; it just took me a long time to figure out how to do it. They also say “if you’re so smart, why aren’t you rich?” Although I don’t think anyone ever asked me that, I could never shake the question after I first came across it. It always rankled me somehow. If you’re that damned smart, making a mere million bucks shouldn’t even raise a sweat. Just think of all the rich idiots you’ve met over the years. What do they have that you don’t? Luck?

Anyway, that’s all the past. The computer I’m using right now has a monster 52 inch flat-screen plasma display. I’m listening to some of my favorite music on a $27,000 stereo system. I have wireless everything. There’s a high-speed multi-homed satellite connection to the internet hidden in the turret of my castle, and several high-performance motor cars in my graveled driveway. Did I say castle? Correct. I’m in a 16th century castle on a hill in Spain. It’s mine. So just how smart am I? It’s hard to say with certainty, but I’m one certifiably rich white dude. And I’m still raking it in. Over $3M a month, clear. It’s a one-man operation and I work about one hour a week.

Here’s my story.

The seed was sown when a friend told me about the following possibility. First of all, buy a mailing list of 50 million email addresses. These can be had online for under $100. Then, pick some sporting event (let’s say) and make a prediction at random about the outcome: team X will beat team Y (ignore draws for now, that’s just a detail). Send your prediction, couched in the appropriate language about mystical forces, seeing the future etc, to all 50 million people on your mailing list. After the event takes place, half of the people you mailed will have received a correct prediction from you. Chuck out the 25 million addresses of the people who got the incorrect prediction. Now pick another upcoming event. Make a prediction at random and mail it to your 25 million survivors. Do this ten times. Each time, say a little more about how omniscient you are in your email, about how you can look into the future. Whatever. After ten mailings you’ll be down to 50,000 people who will have all received ten correct predictions from you.

I probably don’t need to mention that at this point you’re going to have about 50,000 new friends. You could probably start a religion. You could probably make some real money.

That’s the setup. The problem my friend posed: what to do with these 50,000 believers? How can you turn them into money, and in such a way that you also stay out of jail? Clearly you could pull a crude one-time scam. For example, offer to send them (for a price, naturally) the device you were using to make your predictions, and then mail them a Magic 8 ball. Solutions like that are obvious, but they’re very unsatisfying. For one, you’re blowing away all you credibility in one go. For two, you’re going to have some fraction of 50,000 people complaining that they’ve been scammed (i.e., you’re going into hiding or you’re perhaps going to jail). And for three, you wont make that much money: maybe a million before you figure your costs, and that’s under the highly unrealistic assumption that you’ll be able to convince all those 50,000 people to simultaneously pay you $20 each. So while the basic idea holds water, it’s not clear where to take it and how to maximize its potential while staying on the right side of the law.

I admitted the idea was good. At first I didn’t see what to make of it, if anything. Then two months later, lying in bed, I realized I knew exactly what to do. I quit my job the next day.

I’ll tell you what I do. But first….. Why would I tell you? Because I’ve learned, though actually I knew before I started it, that it just doesn’t matter if you know. It’s not the first time I’ve spilled the beans either, though it is the most explicit and it’s the first time in writing. It just doesn’t matter. Hardly anyone would believe you if you told them the truth. Not even if you presented them with a copy of this with my signature at the bottom. Some would say “well of course” but the rest, the great majority of my users, would be more inclined to argue passionately with you and even stop speaking to you if you will not admit your error. And nope, I’m not kidding.

First of all, I use a trade secret algorithm which combines a person’s email address with the text of a question and comes up with a yes/no answer. The secret sauce, oh so special, far too secret for mere patent protection, is in fact based on a simple MD5 checksum of the concatenated email address and question text. If the first character of the MD5 checksum is 0 to 7, the answer to the question for the person with that email address is Yes. Otherwise it’s No. The system can generate millions of these predictions in minutes.

To prime the service, I just sent out the predictions to people. My initial list was 35 million (valid, non-bouncing, unique) email addresses, and I didn’t whittle it down at all when people got sent a wrong prediction. Once I’d gone through ten iterations, the plan went into serious action.

The main point is to change the focus of the original idea. In the original, the focus or pretense is that you can predict the future. Some number of people are going to believe you. But you’d better take advantage of that belief as quickly as possible because they’re going to stop believing pretty fast once you go wrong and it begins costing them. If you’re infallible and then you start screwing up regularly, you can kiss it all goodbye pretty quick smart.

The new focus? Luck. Instead of telling people you can predict the future, you tell them that everyone has lucky (and unlucky) streaks and that you’ve come up with a way to detect when people are on a roll. You tell them up front (click here to accept the Terms and Conditions of service) that you explicitly cannot predict the future. That they should not bet on your predictions. That, as everyone knows, lucky streaks always come to an end, etc. You cover your ass here. What I offer is a service that advises people when it looks like they’re on a lucky (or unlucky) streak.

After the initial priming, all 35 million emails got a further email introducing them to lucky-streak.com. The introductory email depended on the number of initial predictions the system got right for the email address in question. The people who got sent ten correct initial predictions received a very different email from the ones who had an average time of it. Those with a very low success rate got an appropriate letter. Naturally, the take up rate for the service was extremely heavily skewed towards those who got many correct predictions, with a clear bump at the end of the distribution for those with zero correct predictions.

I mail out the predictions (this is a per-user configuration option) and make them available online so that users can see their prediction history. See their luck. For those that want it, an email or SMS text alert is sent to their PDA or mobile phone to tell them just when it looks like they’re on a roll or when their lucky streak seems to be over or ending. All this for $1.99 a month.

I have a couple of servers, located in Seeland. It’s an offshore secure site with basically no laws. The server takes credit card payments and transfers are made to accounts at one of several banks. Cayman Islands, Bermuda, Switzerland, Channel Isles, etc. I have about 30 offshore bank accounts in seven principalities. My business is highly welcomed, and absolutely no questions have ever been asked. In my very occasional visits to these banks, I am treated like royalty. I tip.

Users get to define their own concept of lucky streak. On average, people choose about 4.3. That is, if 4 or 5 of my predictions go their way in a row, they consider themselves to be in a lucky phase. If the same number go against them, they consider themselves to be unlucky. When these things happen, I send out an alert. Consequently, something between 1/16th and 1/32nd of my users are feeling lucky at any one time and roughly the same fraction are being a little bit more cautious than normal. But that’s neither here nor there – the service costs the same whether you’re in a lucky phase or not. I don’t pretend to know anything for sure. No guarantees, no extra charges.

So I’m like a horoscope service or a palm reader. Everyone knows the service makes mistakes, but people love it. And I mean they _really_ love it. Even when their friends tell them it’s a fraud and that there must be some trick, even if they explain the trick, the believers refuse to listen. They come back for more. You can sign up for a year’s worth for a mere $15. Hundreds of thousands of people take that option.

Behind all this are a couple of machines with a simple database. Just linux PCs from Dell. Nothing too special. The database holds email addresses, the prediction questions, and the record of predictions for each user. When a user pulls up their history, I simply generate a line of recent predictions (red dot = wrong, green dot = right) and let them click on these to see the underlying question. It’s pretty easy to see when you’re on a roll. Lucky and unlucky streaks are undeniable. Of course, most of the time, most people are not on any kind of streak. But that’s no problem, that’s exactly what people expect. After all, you can’t be lucky all the time.

Legally, I’m pretty well covered. First of all, I’m quite hard to locate. I spent several grand on a $400/hr lawyer in New York, crafting the Terms and Conditions agreement. It’s very explicit. It states that my predictions may have no more than a 50% chance of being correct. It warns that users should use the service for entertainment only, and under no circumstance as the basis for any kind of betting or other decision making. It tells them that the concept of a lucky streak is extremely nebulous and unproven and that even if there is such a thing it may end without any warning and at any moment. Then there are half a dozen paragraphs of disclaimers and acknowledgements that basically amount to a General Release. I’ve received two legal challenges over the last two years and both have quickly been thrown out long before reaching any court as the result of filing for declarative judgments. The Terms and Conditions are pretty ironclad. The average time new users spend between receiving the page and clicking “I Agree”? About two seconds.

Naturally, people do bet on my predictions, and in general, when they receive an alert to tell them they may be on a lucky streak. Not surprisingly, that seems to be the principal reason people use the service. In fact, I typically receive a couple of hundred grateful emails a month offering me some cut of their winnings! For legal reasons, I always politely decline and instead suggest that they make a contribution to the political party of their choice. The government isn’t too likely to shut me down. In fact, I expect such contributions are pretty scarce. In any case, as far as I’ve been able to determine, I fall outside the jurisdiction and legal system of all countries except the one I reside in and residence itself is invariably a mathematical concept. Just by spending perpetual spring and early summers in various locations around the world, following the sun, it’s not at all clear that I’m a resident of any country or where taxes might be due, supposing any are due at all. The internet created some extraordinarily gaping holes that the historical national tax and legal systems are going to have a hard time filling any time soon.

So that’s it. Right now (and I do mean right now) I have exactly 1,348,216 monthly subscribers at $1.99 a month, plus 703,237 subs who paid the $15 in advance for a year of alerts. That works out at a touch over $3.5M per month. I have no staff and almost nothing to do. Sometimes, I even consider going back to my old job. The servers have multiple redundant independent power supplies, hourly tape backup, RAID arrays, etc. They’re locked down with tight firewalls and the security team at Seeland apply patches on the rare occasion any problem is found (only 3 patches have been applied in two years – one to apache and two to ssh). The credit card companies take a small cut. They love me too, and are lobbyists with considerable clout. My monthly hosting, service and traffic charge is about a thousand bucks. It took me four months to get the site together originally (this done in parallel with sending out the initial predictions). The lucky-streak.com domain costs me $15 a year.

My users love me too. They thank me. They believe. They have chat rooms, mailing lists, IRC channels, and even support groups (only in America of course). There’s a hard core that refuse to believe something so accurate could possibly be a scam. I’m giving them something they want. They evangelize. They proselytize. They swear by me. And yes, they occasionally swear at me. Articles about the site have appeared in the international media and in many national and regional papers and magazines. I refuse virtually all interviews when occasional reporters do manage to get hold of me, and I never allow photos. I’m on record in a Rolling Stone interview as saying that of course it’s all random and that people shouldn’t take it seriously. Did that put a dent in my numbers? No way. Quite the opposite: subscriptions jumped sharply following the interview. My devoted users, interviewed by the same magazine, and others, simply refuse to believe that the service could be anything but real. Of course many don’t believe it has much, if any, accuracy in its actual predictions. They all have one thing in common: they all believe in luck. They believe that luck is real, and that lucky and unlucky streaks exist. And who doesn’t believe that? I just offer a service, backed by seemingly solid evidence (my personalized prediction history for the user), to help flag the good and bad times.

I read some book about the importance of brand. There was a discussion of the Harley Davidson company as a great example of brand. It concluded something like this: if you can convince your customers to tattoo your brand name onto their bodies, you know you’ve won the branding game. I may not be there quite yet, but that’s the kind of following the service has. Reason and rationality are simply not a factor. The average person has no head for probability, in fact no concept of it. Their friends can’t convince them to stop buying lottery tickets or to cancel their subscription to lucky-streak.com. In the end though, the friends don’t figure there’s much harm. After all, it’s just a couple of bucks a month.

When I said I could start a religion, I wasn’t kidding. Not too many leave the service, and at least at the moment, for every one that leaves, two new ones join. Unlucky streaks are just as important to people as lucky ones. Lots of people have no intention of betting or doing anything crazy, but they want to know when they should be a little extra careful. It’s a vital aspect of the service that I’m there for you in the good and the bad, always trying to help, always on your side, ready to celebrate or commiserate as the case may be. The predictions can be wrong half the time (and, of course, they are), but the service doesn’t lose people as a result. The subscribers don’t care about the individual predictions. They care about when they’re in a lucky phase. When they’re on a roll. When a prediction is wrong and you send them email them telling them to watch out, that their streak may be over, they’ll reply to say thank you. I make a random prediction and send it out. The prediction is completely wrong. Nevertheless, the recipient sends me a thank-you note. It’s wacky, but that’s the way it works. I figure tattoos are not out of the question.

I know it can’t grow like this forever, but when you think about it, a couple of million people really is a very small fraction of the number of people online. I figure it’s going to settle down at around 5-10 million subscribers. At some point, probably within the next two years, I expect something will go wrong and that I’ll shut the whole thing down and walk away from it. You can’t make $35M a year in a vacuum. There are all sorts of people that are simply not going to let that happen. Still, after two full years of operation, I have over $40M spread across my various bank accounts, so I’m doing OK. It’s probably strictly 100% legal. It’s not as easy to spend, invest, or move the money as I would like, but on the other hand I don’t pay tax. I do my serious shopping by wire transfer, often to other offshore accounts. There are clearly ethical questions, but I simply ignore them. Like they say, every man has his price. They also say that lotteries are just a tax on people who are bad at math. But I’m not that callous. I have a service, and I’ve been quite up-front about what it does, what can be expected, and even how it works. People choose to send me money in exchange for the service. And after all, it’s for a very good cause.

Like all my satisfied clients, I know my run will also someday come to an end.

But for now, I’m really on a roll.

One problem with high-dimensional spaces is that they can be highly non-intuitive. I did a lot of work on fitness landscapes, which are a form of high dimensional space, and ran into lots of cases in which problems were exceedingly difficult because it’s not clear how to navigate efficiently in such a space. If you’re trying to find high points (e.g., good solutions), which way is up? We’re all so used to thinking in 3 dimensions. It’s very easy to do the natural thing and let our simplistic lifelong physical and visual 3D experience influence our thinking about solving problems in high-dimensional spaces.

Another problem with high-dimensional spaces is that we can’t visualize them unless they are very simple. You could argue that an airline pilot in a cockpit monitoring dozens of dials (each dial gives a reading on one dimension) does a pretty good job of navigating a high-dimensional space. I don’t mean the 3D space in which the plane is flying, I mean the virtual high-dimensional space whose points are determined by the readings on all the instruments.

I think that’s true, but the landscape is so smooth that we know how to move around on it pretty well. Not too many planes fall out of the sky.

Things get vastly more difficult when the landscape is not smooth. In fact they get positively weird. Even with trivial examples, like a hypercube, things get weird fast. For example, if you’re at a vertex on a hypercube, exactly one half of the space is reachable in a single step. That’s completely non-intuitive, and we haven’t even put fitness numbers on the nodes. When I say fitness, I mean goodness, or badness, or energy level, or heuristic, or whatever it is you’re dealing with.

We can visually understand and work with many 3D spaces (though 3D mazes can of course be hard). We can hold them in our hands, turn them around, and use our visual system to help us. If you had to find the high-point looking out over a collection of sand dunes, you could move to a good vantage point (using your visual system and understanding of 3D spaces) and then just look. There’s no need to run an optimization algorithm to find high points, avoiding getting trapped in local maxima, etc.

But that’s not the case in a high-dimensional space. We can’t just look at them and solve problems visually. So we write awkward algorithms that often do exponentially increasing amounts of work.

If we can’t visually understand a high-dimensional space, is there some other kind of understanding that we could get?

If so, how could we prove that we understood the space?

I think the answer might be that there are difficult high-dimensional spaces that we could understand, and demonstrate that we understand them.

One way to demonstrate that you understand a 3D space is to solve puzzles in it, like finding high points, or navigating over or through it without crashing.

We can apply the same test to a high-dimensional space: build problems and see if they can be solved on the fly by the system that claims to understand the space.

One way to do that is the following.

Have a team of people who will each sit in front of a monitor showing them a 3D scene. They’ll each have a joystick that they can use to “fly” through the scene that they see. You take your data and give 3 dimensions to each of the people. You do this with some degree of dimensional overlap. Then you let the people try to solve a puzzle in the space, like finding a high point. Their collective navigation gives you a way to move through the high-dimensional space.

You’d have to allocate dimensions to people carefully, and you’d have to do something about incompatible decisions. But if you built something like this (e.g., with 2 people navigating through a 4D space), you’d have a distributed understanding of the high-dimensional space. No one person would have a visual understanding of the whole space, but collectively they would.

In a way it sounds expensive and like overkill. But I think it’s pretty easy to build and there’s enormous value to be had from doing better optimization in high-dimensional spaces.

All we need is a web server hooked up to a bunch of people working on Mechanical Turk. Customers upload their high-dimensional data, specify what they’re looking for, the data is split by dimension, and the humans do their 3D visual thing. If the humans are distributed and don’t know each other they also can’t collude to steal or take advantage of the data – because they each only see a small slice.

There’s a legitimate response that we already build systems like this. Consider the hundreds of people monitoring the space shuttle in a huge room, each in front of a monitor. Or even a pilot and co-pilot in a plane, jointly monitoring instruments (does a co-pilot do that? I don’t even know). Those are teams collectively understanding high-dimensional spaces. But they’re, in the majority of cases, not doing overlapping dimensional monitoring, and the spaces they’re working in are probably relatively smooth. It’s not a conscious effort to collectively monitor or understand a high-dimensional space. But the principle is the same, and you could argue that it’s a proof the idea would work – for sufficiently non-rugged spaces.

Apologies for errors in the above – I just dashed this off ahead of going to play real football in 3D. That’s a hard enough optimization problem for me.

I therefore feel eminently qualified to speculate on what user authentication might look like in a world with no free will.

Note that I don’t care whether free will exists or not, and I certainly don’t want to waste my time thinking or talking about it. But if it doesn’t exist, then the following user authentication algorithm does exist. We couldn’t implement it, but it would certainly exist and it’s fun to consider instead of doing real work.

When a computer needs to verify who you are, it tells you to move the mouse around randomly for as long as you like. Or to just bang on the keyboard. The kind of thing you do when you’re generating randomness for the construction of a PGP/GPG key.

But if there’s no free will then it’s not random.

So the algorithm can just look up what you did in a big table to see who you are. As two users could conceivably do the same thing, it probably needs a little more information, like the time of day and your IP address – neither of which you’d have any control over either.

That’s it. No need for anything fancy, just a lookup table. No-one would ever fail to be recognized, no-one would ever be mistaken for someone else, there’d be no identity theft, etc. Even if you just sat there and did nothing for a while the machine would know exactly who you were. You could always log in by just briefly doing nothing at all, and then continuing. The length of time you did nothing for would betray you.

All totally absurd, of course, and thinking about it quickly becomes highly circular. Just like the rest of the debate.

As you were.

You buy a large collection of bike locks. Then you drive around the city and lock people’s bike locks to the bike racks (or poles or railings or whatever). People can still take their bikes and their locks away as usual, and your lock then falls to the ground (but is still attached to the bike rack). Once every X weeks your people go out and do a sweep – taking away bikes and bike fragments that still have your lock on their lock, recovering your locks that are no longer locked to other locks, and using these to lock the locks of bikes that do not have your lock on them.

You can put a brightly colored ring with a warning message on your locks “This bike will be taken away in X weeks if this bicycle has not been used”.

Walking on the UCSD campus in 1998, I passed a bike rack that looked partly like a scrap metal junkyard. There were bikes in many conditions, from perfect to clearly abandoned rusting frames. Some had no wheels or no seat. Several were just a frame and a chain locked to the bike rack with a big expensive U-lock.

I thought “interesting, I wonder how they deal with these old bikes, possibly abandoned or forgotten bikes, abandoned frames, wheeless bikes, bikes whose owners died, etc. How can they know when it’s ok to cut something free and take it away? How can they know when it’s not ok?”

I saw the same problem at my apartment building, a similar tangle of 6 or 8 old bikes. Later, living in NYC down in the village I’d see hundreds of apparently abandoned eyesores. Blots on the fair face of the city.

So there’s the problem.

That is, devise a method or a policy for dealing with the removal of these bikes, ex-bikes, partial bikes, etc. Below are some requirements for a good solution. If you miss on any one of these, whatever you’re thinking of probably isn’t as good as what I thought of :-) There are many partial solutions.

1. It must be cheap. You must be able to employ regular people to carry out your plan. No high tech, no massive salaries, none of that.
2. It must be effective: no abandoned or unused bike will be missed.
3. There must be no waste.
4. Bike owners must get fair warning their bike is going to be taken away.
5. No-one should be able to cause anyone else’s bike to be taken away.
6. No-one should be able to cause a bike that should have been taken away not to be taken away.
7. No-one should be able to make a bike be taken away without the owner getting a fair chance to know it was due to be taken. E.g., with parking tickets I can simply take the parking ticket off any car I like and chuck it in the trash.
8. There should not be (as far as possible) opportunities to exploit the system by criminals.
9. You must not interfere with any bicycle (no marking them, etc).
10. The program must be something that has a high probability of being regarded as fair and which gets good press (everyone loves it, business comes to you).

Those are all the conditions I can think of right now, but there may be more.

You get to make all the decisions. Pretend you’re the mayor of NYC, coming up with a new policy for cleaning up the streets. And it’s an election year.

If you have a secret and you tell someone, it makes no sense to tell them they can’t tell anyone else.

Let’s say there are 2 kinds of secrets you might be tempted to pass along: a) those that are more important to the receiver than they are to you (e.g., you just found out that X is sleeping with your friend Y’s partner and you’re considering telling Y), and b) those that are less important to the receiver than they are to you.

Clearly it doesn’t make much sense to tell the receiver in class (a) that they can’t tell anyone. They probably have less incentive to be telling people than you do, they’re closer to the source than you are, and perhaps the information is “theirs” more than it is “yours”. Things like that.

But it doesn’t make sense to tell the receiver in class (b) that they can’t tell anyone either. That’s because it’s unreasonable to expect them to keep something secret that you’re not keeping secret when it’s even less important to them than it is to you. Even if you swear them to secrecy, as you may have been sworn to secrecy, you can’t rationally expect them to keep the secret.

Most secrets fall into class (b).

The rational and responsible conclusion is that either you decide that the buck stops with you and you don’t pass it on, OR you decide to pass it on, in the full knowledge that you are actively spreading the secret, and in fact lowering the barrier to it spreading more widely. At the very least, have the intellectual honesty not to preface the secret telling with “you can’t tell anyone about this…”