Archive for the ‘other’ Category

I go shopping for a compass, then my Sonos decides it needs one too

Wednesday, December 10th, 2014

Screen Shot 2014-12-10 at 3.42.33 PMLast night I spent some time online looking to buy a compass. I looked at many of the Suunto models. Also yesterday I installed Little Snitch after noticing that an unknown gamed process wanted to establish a TCP/IP connection.

Anyway… a few minutes ago, 10 or 11 hours after I eventually bought a compass, a message pops up from Little Snitch telling me that the Mac OS X desktop Sonos app was trying to open a connection to ns.suunto.com. See image on right (click for the full-sized version).

WTF?!?!

Can someone explain how that works? Either Little Snitch is mightily confused or…… Or what? Has the Sonos app has been digging around in my Chrome browser history or cache or cookie jar? Is Chrome somehow complicit locally? Or something with cookies (but that would require the Sonos to be accessing and sending cookies stored by Chrome). Or…. what?

And, why ns.suunto.com? There’s an HTTP server there, but its / resource is not very informative:

$ curl -S -v http://ns.suunto.com
* Rebuilt URL to: http://ns.suunto.com/
* Hostname was NOT found in DNS cache
*   Trying 23.63.99.202...
* Connected to ns.suunto.com (23.63.99.202) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.1
> Host: ns.suunto.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
* Server Apache is not blacklisted
< Server: Apache
< Content-Type: text/html; charset=iso-8859-1
< Date: Wed, 10 Dec 2014 16:02:47 GMT
< Content-Length: 16
< Connection: keep-alive
<
* Connection #0 to host ns.suunto.com left intact
File not found."⏎

Unfortunately, Little Snitch doesn't tell me the full URL that the Sonos app was trying to access.

Anyone care to speculate what's going on here?

A simple way to calculate the day of the week for any day of a given year

Sunday, November 11th, 2012

March 29th

Image: Jeremy Church

The other day I read a tweet about how someone was impressed that a friend had been able to tell them the day of the week given an arbitrary date.

There are a bunch of general methods to do this listed on the Wikipedia page for Determination of the day of the week. Typically, there are several steps involved, and you need to memorize some small tables of numbers.

I used to practice that mental calculation (and many others) when I was about 16. Although all the steps are basic arithmetic, it’s not easy to do the calculation in your head in a couple of seconds. Given that most of these questions that you’re likely to face in day-to-day life will be about the current year, it seemed like it might be a poor tradeoff to learn the complicated method to calculate the day of the week for any date if there was a simpler way to do it for a specific year.

The method I came up with after that observation is really simple. It just requires you to memorize a single 12-digit sequence for the current year. The 12 digits correspond to the first Monday for each month.

For example, the sequence for 2012 is 265-274-263-153. Suppose you’ve memorized the sequence and you need to know the day of the week for March 29th. You can trivially calculate that it’s a Thursday. You take the 3rd digit of the sequence (because March is the 3rd month), which is 5. That tells you that the 5th of March was a Monday. Then you just go backward or forward as many weeks and days as you need. The 5th was a Monday, so the 12th, 19th, and 26th were too, which means the 29th was a Thursday.

It’s nice because the amount you need to memorize is small, and you can memorize less digits if you only want to cover a shorter period. The calculation is very simple and always the same in every case, and you never have to think about leap years. At the start of each year you just memorize a single sequence, which is quickly reinforced once you use it a few times.

Here’s Python code to print the sequence for any year.

#!/usr/bin/env python

import datetime, sys

try:
    year = int(sys.argv[1])
except IndexError:
    year = datetime.datetime.today().year

firstDayToFirstMonday = ['1st', '7th', '6th', '5th', '4th', '3rd', '2nd']
months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
          'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
summary = ''

for month in range(12):
    firstOfMonth = datetime.datetime(year, month + 1, 1).weekday()
    firstMonday = firstDayToFirstMonday[firstOfMonth]
    print months[month], firstMonday
    summary += firstMonday[0]

print 'Summary:', '-'.join(summary[x:x+3] for x in range(0, 12, 3))

The output for 2012 looks like

Jan 2nd
Feb 6th
Mar 5th
Apr 2nd
May 7th
Jun 4th
Jul 2nd
Aug 6th
Sep 3rd
Oct 1st
Nov 5th
Dec 3rd
Summary: 265-274-263-153

The memory task is made simpler by the fact that there are only 14 different possible sequences. Or, if you consider just the last 10 digits of the sequences (i.e., starting from March), there are only 7 possible sequences. There are only 14 different sequences, so if you use this method in the long term, you’ll find the effort of remembering a sequence will pay off when it re-appears. E.g., 2013 and 2019 both have sequence 744-163-152-742. There are other nice things you can learn that can also make the memorization and switching between years easier (see the Corresponding months section on the above Wikipedia page).

Here are the sequences through 2032:

2012 265-274-263-153
2013 744-163-152-742
2014 633-752-741-631
2015 522-641-637-527
2016 417-426-415-375
2017 266-315-374-264
2018 155-274-263-153
2019 744-163-152-742
2020 632-641-637-527
2021 411-537-526-416
2022 377-426-415-375
2023 266-315-374-264
2024 154-163-152-742
2025 633-752-741-631
2026 522-641-637-527
2027 411-537-526-416
2028 376-315-374-264
2029 155-274-263-153
2030 744-163-152-742
2031 633-752-741-631
2032 521-537-526-416

SOBGTR OCCC AILD FUNEX?

Friday, August 10th, 2012

Suppose you had to pick a very small set of character strings that you, and only you, could identify without hesitation in a particular way. What would you choose? How small a set could you choose and still be unique? For example, SOBGTR OCCC AILD FUNEX? is a set of strings that I think would uniquely identify me. (My interpretation is below.) I’m pretty sure that almost any subset of 3 of them would suffice. Coming up with a set of two wouldn’t be hard, I don’t think – but it feels risky.

There are 7 billion people on the planet. So if you just pick 3 reasonably obscure acronyms, e.g., things that only 1 person in 2000 would recognize, you’re heading in the right direction (since 2000 cubed is 8 billion). But that’s only if the obscurity of the things you pick is independent. For example, it’s less good to pick 3 computer acronyms from the 1960s than to choose just one of them plus some things from very different areas of your knowledge.

The rules

  1. Each of your strings with its meaning to you must be findable on Google.
  2. To match with you, another person must interpret all your strings the way you do.

Rule 1 prevents you from choosing something like your bank PIN number, that only you could possibly know. Without this rule, everyone could trivially choose a set of one string. The rule makes thinking up a uniquely identifying set for yourself like a game. Given that all your strings and their interpretations are on Google, each of your strings will likely be recognized by someone in the way you recognize it, so your set will probably have at least 2 strings. You need to choose a set of strings whose set of interpretations, taken as a whole, make you unique (Rule 2).

Why is this interesting?

I find this interesting for many reasons. It seems clear that uniquely identifying sets are fairly easy to construct for people and they’re very small. Certainly small enough to fit in a tweet. Although it’s easy to make a set for yourself, it’s hard to make one for someone else – you might even argue that by definition it’s not possible. If someone else makes one, you can’t produce their set of interpretations without spending time on Google, and even then you’d probably have to know the person pretty well.

Is there a new authentication scheme here somewhere? It’s tempting to think yes, but there probably isn’t. This is less secure than asking people for a set of secrets that are not each findable in Google, so anything you come up with is almost certain to be less secure than the same thing based on a set of actual secrets. It’s more of a fun thought exercise (or Twitter game). It’s not hard to imagine some form of authentication. For example, identify which of a set of symbols are special to you (avoiding others chosen randomly from, say, the set of all acronyms), and their correct interpretations for you, and do it rapidly. Or if a clone shows up one day, claiming to be you, and you’ve thoughtfully put a sealed set of unique symbol strings in your safe, you should be able to convince people that you’re the real you :-)

Answer

Here’s my unhesitating interpretation of the set of 4 strings above:

Remember, to be me you have to get them all. It’s not enough to get a couple, or even three of them.

Emacs buffer mode histogram

Thursday, November 10th, 2011

Tonight I noticed that I had over 200 buffers open in emacs. I’ve been programming a lot in Python recently, so many of them are in Python mode. I wondered how many Python files I had open, and I counted them by hand. About 90. I then wondered how many were in Javascript mode, in RST mode, etc. I wondered what a histogram would look like, for me and for others, at times when I’m programming versus working on documentation, etc.

Because it’s emacs, it wasn’t hard to write a function to display a buffer mode histogram. Here’s mine:

235 buffers open, in 23 distinct modes

91               python +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
47          fundamental +++++++++++++++++++++++++++++++++++++++++++++++
24                  js2 ++++++++++++++++++++++++
21                dired +++++++++++++++++++++
16                 html ++++++++++++++++
 7                 text +++++++
 4                 help ++++
 4           emacs-lisp ++++
 3                   sh +++
 3       makefile-gmake +++
 2          compilation ++
 2                  css ++
 1          Buffer-menu +
 1                 mail +
 1                 grep +
 1      completion-list +
 1                   vm +
 1                  org +
 1               comint +
 1              apropos +
 1                 Info +
 1           vm-summary +
 1      vm-presentation +

Tempting as it is, I’m not going to go on about the heady delights of having a fully programmable editor. You either already know, or you can just drool in slack-jawed wonder.

Unfortunately I’m a terrible emacs lisp programmer. I can barely remember a thing each time I use it. But the interpreter is of course just emacs itself and the elisp documentation is in emacs, so it’s a really fun environment to develop in. And because emacs lisp has a ton of support for doing things to itself, code that acts on emacs and your own editing session or buffers is often very succinct. See for example the save-excursion and with-output-to-temp-buffer functions below.

(defun buffer-mode-histogram ()
  "Display a histogram of emacs buffer modes."
  (interactive)
  (let* ((totals ‘())
         (buffers (buffer-list()))
         (total-buffers (length buffers))
         (ht (make-hash-table :testequal)))
    (save-excursion
      (dolist (buffer buffers)
        (set-buffer buffer)
        (let
            ((mode-name (symbol-name major-mode)))
          (puthash mode-name (1+ (gethash mode-name ht 0)) ht))))
    (maphash (lambda (key value)
               (setq totals (cons (list key value) totals)))
             ht)
    (setq totals (sort totals (lambda (x y) (> (cadr x) (cadr y)))))
    (with-output-to-temp-buffer "Buffer mode histogram"
      (princ (format "%d buffers open, in %d distinct modes\n\n"
                      total-buffers (length totals)))
      (dolist (item totals)
        (let
            ((key (car item))
             (count (cadr item)))
          (if (equal (substring key -5) "-mode")
              (setq key (substring key 0 -5)))
          (princ (format "%2d %20s %s\n" count key
                         (make-string count ?+))))))))
 

Various things about the formatting could be improved. E.g., not use fixed-width fields for the count and the mode names, and make the + signs indicate more than one buffer mode when there are many.

fishus

Tuesday, August 16th, 2011

Date: Tue, 26 Sep 95 15:55 MDT
From: mosterin@hydra.unm.edu (Ana Mosterin)
To: dsmith@cs.unm.edu

well, you should love wild cooking too
you have to find the right attitude:

you have to be sensitive enough
to feel the fear and shudder a bit at what you’re doing
and to love your piece of fishus enough
to touch it and smell it
with patience and lust
and then aaaaaaaarrrh! sacrifice it
and chop it skillfully
and be matter-of-fact enough
to to act like you’ve done it before
and professionally dry your hands
with your apron
and and have your hands on your hip
as you listen and smell to
the sound of the frying
breath in through your nose
as you watch the pan with love and think
“no, no more garlic,
just a half-cup of wine”
and relax!
it’s the ferocious poetry
of the wild cooking job
and then eating it will be twice as lovely
you’ll see

hey, derek,
cooking is not mary poppins!

Back of the envelope calculations with The Rule of 72

Monday, June 20th, 2011

Image: internetworldstats.com

The Rule of 72 deserves to be better known among technical people. It’s a widely-known financial rule of thumb used for understanding and calculating interest rates. But others, including computer scientist and start-up founders, are often concerned with growth rates. Knowing and applying the rule of 72 can help in developing numerical literacy (numeracy) around growth.

For example, consider Moore’s Law, which describes how "the number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years." If something doubles every two years, at what rate does it increase per month, on average? If you know the rule of 72, you’ll instantly know that the monthly growth rate is about 3%. You get the answer by dividing 72 by 24 (the number of months).

Computer scientists are usually very familiar with powers of two. It’s often convenient to take advantage of the fact that 2^10 is about 1,000. That means that when something increases by a factor of 1,000, it has doubled about 10 times. By extension, and with a little more error, an increase of a million corresponds to 20 doublings, and a billion is 30 doublings (log base two of a billion is actually 29.897, so the error isn’t too wild). You can use this to ballpark the number of doublings in a process really easily, and go directly from that to a growth rate using the rule of 72.

For example, the bottom of this page tells us that there were about 16,000 internet domains on July 1st 1992, and 1.3M of them on July 1st 1997. Let’s think in thousands: that’s a jump from 16 to just over 1,000 in 5 years. To get from 1 to 16 is four doublings, so from 16 to 1,000 is six doublings (because 1,000 is ten doublings from 1). So the number of domains doubled 6 times in 5 years, or 6 times in 60 months, or once every 10 months (on average). If you want something to double in 10 months, the rule of 72 tells us we need a growth rate of 7.2% per month. To check: 16,000 * (1.072 ^ 60) = 1,037,067. That’s a damned good estimate (remember that we were shooting for 1M, not 1.3M) for five seconds of mental arithmetic! Note that the number of domains was growing much faster than Moore’s law (3% per month).

You can quickly get very good at doing these sorts of calculations. Here’s another easy example. This page shows the number of internet users growing from 16M in December 1995 to 2,072M in March of 2011. That’s just like the above example, but it’s 7 doublings in 15.25 years, or 183 months. That’s pretty close to a doubling every 24 months, which we know from above corresponds to 3% growth per month.

You can use facility with growth rates to have a good sense for interest rates in general. You can use it when building simple (exponential) models of product growth. E.g., suppose you’re launching a product and you reckon you’ll have 300K users in a year’s time. You want to map this out in a spreadsheet using a simple exponential model. What should the growth rate be? 300K is obviously not much more than 256 * 1,024, which is 18 doublings in 365 days, or a doubling roughly every 20 days. The rule of 72 gives 72/20 = ~3.5, so you need to grow 3.5% every day to hit your target. Is that reasonable? If it is, it means that when you hit 300K users, you’ll be signing up about 3.5% of that number, or 10,500 users per day. As you can see, familiarity with powers of two (i.e., estimating number of doublings) and with the rule of 72 can give you ballpark figures really easily. You can even use your new math powers to avoid looking stupid in front of VCs.

The math behind the rule of 72 is easy to extend to triplings (rule of 110), quadrupling (rule of 140), quintupling (rule of 160), etc.

Finally, you can use these rules of thumb to do super geeky party tricks. E.g., what’s the tenth root of two? Put another way, what interest rate do you need for something to double after ten periods? The rule of 72 tells you it’s 72/10 = 7.2%, so the tenth root of two will be about 1.072 (in fact 1.072 ^ 10 = 2.004). What’s the 20th root of 5? The rule of 160 tells you you need 160/20 = 8% growth each period, so 1.08 should be about right (the correct answer is ~1.0838).

As with all rules of thumb, it’s good to have a sense of when it’s most applicable. See the wikipedia page or this page for more detailed information. It’s also of course good to understand that it may not be suitable to model growth as an exponential at all.

The eighty six non-trivial powers ≤ 2^20

Wednesday, March 30th, 2011

Tonight Jamu Kakar mentioned in IRC that a program of his had unexpectedly crashed after processing 1,048,376 items. I think it’s a useful debugging skill to have to be able to recognize numbers like that (it’s very close to 2^20). I’ve often wanted to write a tiny program to print out all the non-trivial powers, and since I have far more important and pressing things to be doing, I immediately went to write the code. At a minimum it seems prudent to recognize all powers up to 1000, and the powers of 2 to much higher. Below you have all 86 non-trivial powers up to 2^20. I don’t know them all, but I wish I did.

  4 = 2^2                  729 = 3^6, 9^3                32768 = 2^15, 8^5
  8 = 2^3                 1000 = 10^3                    38416 = 14^4
  9 = 3^2                 1024 = 2^10, 4^5               46656 = 6^6
 16 = 2^4, 4^2            1296 = 6^4                     50625 = 15^4
 25 = 5^2                 1331 = 11^3                    59049 = 3^10, 9^5
 27 = 3^3                 1728 = 12^3                    65536 = 2^16, 4^8, 16^4
 32 = 2^5                 2048 = 2^11                    78125 = 5^7
 36 = 6^2                 2187 = 3^7                     83521 = 17^4
 49 = 7^2                 2197 = 13^3                   100000 = 10^5
 64 = 2^6, 4^3, 8^2       2401 = 7^4                    104976 = 18^4
 81 = 3^4, 9^2            2744 = 14^3                   117649 = 7^6
100 = 10^2                3125 = 5^5                    130321 = 19^4
121 = 11^2                3375 = 15^3                   131072 = 2^17
125 = 5^3                 4096 = 2^12, 4^6, 8^4, 16^3   160000 = 20^4
128 = 2^7                 4913 = 17^3                   161051 = 11^5
144 = 12^2                5832 = 18^3                   177147 = 3^11
169 = 13^2                6561 = 3^8, 9^4               248832 = 12^5
196 = 14^2                6859 = 19^3                   262144 = 2^18, 4^9, 8^6
216 = 6^3                 7776 = 6^5                    279936 = 6^7
225 = 15^2                8000 = 20^3                   371293 = 13^5
243 = 3^5                 8192 = 2^13                   390625 = 5^8
256 = 2^8, 4^4, 16^2     10000 = 10^4                   524288 = 2^19
289 = 17^2               14641 = 11^4                   531441 = 3^12, 9^6
324 = 18^2               15625 = 5^6                    537824 = 14^5
343 = 7^3                16384 = 2^14, 4^7              759375 = 15^5
361 = 19^2               16807 = 7^5                    823543 = 7^7
400 = 20^2               19683 = 3^9                   1000000 = 10^6
512 = 2^9, 8^3           20736 = 12^4                  1048576 = 2^20, 4^10, 16^5
625 = 5^4                28561 = 13^4

I produced the above with this quick hack:

from collections import defaultdict

powers = defaultdict(list)
lim = 20

for a in range(2, lim + 1):
    for b in range(2, lim + 1):
        n = a ** b
        if n > 2 ** lim:
            break
        powers[n].append((a, b))

for n in sorted(powers.keys()):
    print '%7d = %s' % (n,
                        ', '.join('%d^%d' % (a, b)
                                  for (a, b) in powers[n]))
 

GMZD: Google Maps Zoom-out Distance

Sunday, February 6th, 2011

Here’s a fun and simple measure of distance between any two locations, A and B. First, find A using Google maps and zoom in (centered) as far as you can go, though don’t go into street view as lots of places still don’t have that. Now, if you can already see B on the map, it has a Google Maps Zoom-out Distance (GMZD) of zero because you don’t need to zoom out at all to see B. If you have to zoom out once, then A and B are at distance 1 according to GMZD, etc.

For example, let’s start with Union Square in New York. Fully zoomed in we can see Coffee Shop, so that’s at distance 0. Those things are about a minute’s walk from the center of Union Square. Zooming out one click, we pick up Bowlmor Lanes on University Place and the Whole Foods Market at the South of Union Square. Those things are at GMZD-1. The outer edge of distance 1 is about a few minute’s walk from Union Square. Continuing outwards, Betaworks is at GMZD-4, Central Park at distance 6, Boston and Niagara Falls at distance 12, Florida, Winnipeg, and St John’s, Newfoundland at 14, San Francisco at 15, Barcelona at 16, and Sydney at 17. (Although you can zoom out 18 levels, GMZD-17 seems to be as many as you practically need to see anything.)

You can also think of GMZD as half the number of clicks you’d need to do on Google maps to go from being fully zoomed in on A to being fully zoomed in on B (with some panning in between). When you look at Google maps you can count the number of notches on the little slider (see image on left) above your current zoom level to see the GMZD from the center of the visible map to its outer edges.

Update: I meant to mention that GMZD is not a formal distance metric. It is non-negative (GMZD(A, B) >= 0) and symmetric (GMZD(A, B) = GMZD(B, A)) for all points A and B, but distinct points can be at distance zero and (as a result) the triangle inequality also does not hold (e.g., Union Square is distance zero from Coffee Shop, and Coffee Shop is distance zero from Union Square Cafe, but the distance from Union Square to Union Square Cafe is one. Not being a metric space is what makes it interesting, though :-)

Paella

Sunday, June 14th, 2009

671505-paella-3

Just little and crazy

Monday, January 26th, 2009

Tonight Lucas was crying about having to have dinner instead of being able to use the laptop. I asked him if he even remembered what life was like before he had a laptop to play with: “Yes. Before the laptop we had the Wii. Before that was the Nintendo DS. Before the DS was the Micro. And before that I was just little and crazy.”

Lucas does a crazy solo dance

Sunday, December 28th, 2008

Pond scum

Friday, September 5th, 2008

Pond scumI had breakfast this morning at a bar in the Santa Caterina market in Barcelona with Jono Bennett. He’s a writer. We were reflecting on similarities in our struggles to do our own thing. An email about a potential Fluidinfo investor that I’d recently sent to a friend came to mind. I wrote:

I had a really good call with AAA. He told me he’s interested and wants to talk to BBB and CCC. I then got mail the next day from DDD (of the NYT) who told me he’d just had dinner with AAA and BBB and that they’d talked about my stuff. So something may happen there (i.e., I’ll never hear from them again).

The last comment, that I’d probably never hear from them again, was entirely tongue-in-cheek. I wrote it knowing it was a possibility, but not really thinking it would happen.

But it did.

Things like that seem to be part & parcel of the startup world as you attempt to get funded. I have often asked myself how can it be possible for things to be this way? How you can have people so excited, telling you and others you’re going to change the world, be worth billions, and then you never hear from them again? (Yes, of course you have to follow up, and I did. But that’s not the point: If you didn’t follow up you’d never hear from them.)

How can that be? In what sort of world is such a thing possible?

I came up with a highly flawed analogy. Despite its limited accuracy I find it amusing and can’t resist blogging it even if people will label me bitter (I’m not).

Kids with sticksFirst: startup founders are pond scum. Second: potential investors are a troupe of young kids wandering through the park with sticks.

The kids poke into the ponds, stirring up the scum. They’re looking for cool things, signs of life, perhaps even something to take home. They’re genuinely interested. They’re fascinated. The pond scum listen to their excited conversation and think the kids will surely be back tomorrow. But it’s summer, and the world is so very very big.

The pond scum are working on little projects like photosynthesis, enhancements to the Krebs cycle, or the creation of life itself. All the while they’re pondering how to make themselves irresistible, believing that someday the kids with the sticks will be back, that they’ll eventually be scooped up.

As Paul Graham recently wrote, fundraising is brutal. His #1 recommendation is to keep expectations low.

Kid with stickYep, you’re pond scum.

Get used to it.

Embrace it.

GPS serendipity: Florence Avenue, Sebastopol

Monday, July 14th, 2008

img_0601.jpgI drove from Oakland up to the O’Reilly Foo camp last Friday. The O’Reilly offices are just outside Sebastopol, CA. I stopped at an ATM and my GPS unit got totally confused. So I took a few turns at random and wound up on Florence Avenue. I drove a couple of hundred meters and started seeing big colorful structures out the front of many houses. They were so good I stopped, got out my camera, and took a whole bunch of pictures.

I talked to a man washing his car in his driveway. He told me that “Patrick” had created all the figures, and installed them on the front lawns. I got the impression that it was all free. Soon after I found the house that was unmistakably Patrick’s and seeing a man loading things into a pickup truck I went up and asked if he was Patrick. It was him and we had a friendly talk (mainly me telling him he was amazing). He gave me a calendar of his work.

Click on the thumbnails below to see bigger versions. There’s even a FC Barcelona structure. As I found out later, lots of people (of course) have seen these sculptures. When I got to Foo, there was one (image above) outside the O’Reilly office. Google for Patrick Amiot or Florence Avenue, Sebastopol and you’ll find much more. And Patrick has his own web site.

img_0556.jpgimg_0558.jpgimg_0560.jpgimg_0561.jpgimg_0567.jpgimg_0568.jpgimg_0569.jpgimg_0570.jpgimg_0572.jpgimg_0573.jpgimg_0579.jpgimg_0581.jpgimg_0582.jpgimg_0585.jpgimg_0586.jpgimg_0589.jpgimg_0592.jpgimg_0595.jpgimg_0599.jpgimg_0575.jpgimg_0577.jpgimg_0564.jpgimg_0566.jpg

Minor mischief: create redirect loops from predictable short URLs

Tuesday, July 1st, 2008

redirect loopI was checking out the new bit.ly URL shortening service from Betaworks.

I started wondering how random the URLs from these URL-shortening services could be. I wrote a tiny script the other day to turn URLs given on the command line into short URLs via is.gd:

import urllib, sys
for arg in sys.argv[1:]:
    print urllib.urlopen(
        ‘http://is.gd/api.php?longurl=’ + arg).read()

I ran it a couple of times to see what URLs it generated. Note that you have to use a new URL each time, as it’s smart enough not to give out a new short URL for one it has seen before. I got the sequence http://is.gd/JzB, http://is.gd/JzC, http://is.gd/JzD, http://is.gd/JzE,…

That’s an invitation to some minor mischief, because you can guess the next URL in the is.gd sequence before it’s actually assigned to redirect somewhere.

We can ask bit.ly for a short URL that redirects to our predicted next is.gd URL. Then we ask is.gd for a short URL that redirects to the URL that bit.ly gives us. If we do this fast enough, is.gd will not yet have assigned the predicted next URL and we’ll get it. So the bit.ly URL will end up redirecting to the is.gd URL and vice versa. In ugly Python (and with a bug/shortcoming in the nextIsgd function):

import urllib, random

def bitly(url):
    return urllib.urlopen(
        ‘http://bit.ly/api?url=’ + url).read()

def isgd(url):
    return urllib.urlopen(
        ‘http://is.gd/api.php?longurl=’ + url).read()

def nextIsgd(url):
    last = url[-1]
    if last == ‘z’:
        next = ‘A’
    else:
        next = chr(ord(last) + 1)
    return url[:-1] + next

def randomURI():
    return ‘http://www.a%s.com’ % \
           .join(map(str, random.sample(xrange(100000), 3)))

isgdURL = isgd(randomURI())
print ‘Last is.gd URL:’, isgdURL

nextIsgdURL = nextIsgd(isgdURL)
print ‘Next is.gd URL will be:’, nextIsgdURL

# Ask bit.ly for a URL that redirects to nextIsgdURL
bitlyURL = bitly(nextIsgdURL)
print ‘Step 1: bit.ly now redirects %s to %s’ % (
    bitlyURL, nextIsgdURL)

# Ask is.gd for a URL that redirects to that bit.ly url
isgdURL2 = isgd(bitlyURL)
print ‘Step 2: is.gd now redirects %s to %s’ % (
    isgdURL2, bitlyURL)

if nextIsgdURL == isgdURL2:
    print ‘Success’
else:
    print ‘Epic FAIL’

This worked first time, giving:

Step 1: bit.ly now redirects http://bit.ly/fkuL8 to http://is.gd/JA9
Step 2: is.gd now redirects http://is.gd/JA9 to http://bit.ly/fkuL8

In general it’s not a good idea to use predictable numbers like this, which hardly bears saying as just about every responsible programmer knows that already.

is.gd wont shorten a tinyurl.com link, as tinyurl is on their blacklist. So they obviously know what they’re doing. The bit.ly service is brand new and presumably not on the is.gd radar yet.

And finally, what happens when you visit one of the deadly looping redirect URLs in your browser? You’d hope that after all these years the browser would detect the redirect loop and break it at some point. And that’s what happened with Firefox 3, producing the image above.

If you want to give it a try, http://bit.ly/fkuL8 and http://is.gd/JA9 point to each other. Do I need to add that I’m not responsible if your browser explodes in your face?

Manhole

Friday, May 9th, 2008

[Listening just now to the Tiger Lillies song Bankrobber Blues I decided to post another little fictional story I wrote some years ago. The first in the series was Lucky Streak. As a kid I had a head full of criminal schemes and at times was convinced I’d inevitably grow up to be a cat burglar. Mathematician by day, cat burglar by night. It didn’t pan out that way.]

Manhole

I recently became a photographer. Before high-quality digital cameras became available, I had never had an interest. Sure, I liked photos as much as the next person, but it was all so messy. So imprecise. So… analog. I mean why bother? I’m into digital music in a way I could never have been with tapes or vinyl. Why do anything else? Nostalgia? Tradition? Gimme a break. Analog devices with linear access? Puhlease. I keep an eye on things, I have a feel for this stuff. When technology crosses some line, it’s time to buy in. Before that it’s just too messy.

I recently became a criminal. Historically, crime at a distance was always a challenge. Physical events took place, objects were exchanged, people met or talked on the phone. Messy, way too messy. There was always a chain, a trail: someone had met the guy, someone had given someone else a brown paper bag stuffed with cash and a pistol. With time and money, the feds were eventually going to catch you. You couldn’t find good help, couldn’t recruit and train them, had to trust them, were betrayed or sold out, either to the cops or a rival. Whatever. It made no sense, unless you were the mob and you paid everyone off. Or bumped them off. Messy.

The internet changed all that. No government wants to allow anonymous action, but the net was too big. For nearly twenty years I have watched and been amazed that they didn’t do something to stop it. Now it’s too late, far too late. Anonymous action is possible. It’s not straightforward, one must be careful, but it can be done. Tools like mixmaster (anonymous remailer), spamgourmet (similar), the anonymizer (anonymous browsing), ssh tunnels, cheap and disposable machines for rent on the network running simple forwarding servers, paypal for anonymous transfers of cash, public auction sites for posting messages to those who know what to look for, steganography, instant messaging, strong crypto, wide open windows boxes (with broadband connections and fixed IP addresses) just waiting to be commandeered, etc. The tools are all out there and in most cases they’re freely available. With some knowledge, one can carefully build an action network that no-one’s ever going to trace. The whole point of some of these tools is to preserve anonymity. No logs are kept. Subpoenas are useless when there simply are no records.

Just to warm up, I robbed a bank. Not a white-collar electronic robbery. I’m talking blue-collar physical bank robbery. It’s hardly a well-paying profession, but it’s a good lark. In fact, it pays rather badly. It’s also increasingly common and easier, as tellers and other bank staff are instructed to do exactly as told, not to make a fuss, etc. So why rob a bank? Well, why take a dumbass picture of a tree or your pet dog with your new digital camera? Why? Because you have the technology and you want to try it out. Because you can. Because you’ve gone digital and you can take a no-cost snap of whatever you damned well please.

I, physically, robbed a bank. That’s messy, true. But at some point I wanted skin in the game, wanted the whole thing to be real, to prove that it all worked. Plus, I can’t solve everything at once: I didn’t want someone else actually robbing the bank. That way, I’d either never see the money or I’d leave a trail that could be followed. So I planned carefully and I did it myself.

I recruited a dozen people online. That was actually the easy part. There’s any number of bored and broke college kids out there on the net. I made two roles: watchers and actors. I had actors do various silly extroverted tasks and each time had a watcher turn up to verify that the actor showed up. Everyone got different instructions and everyone was told enough and reported enough back to me so that a) actors knew they were being watched and b) I knew the test tasks were in fact getting done. I paid on task completion and verification. An actor would be instructed to do something oddball and very specific, like walk into a 7-eleven at a certain time in the middle of summer, buy six dozen hotdogs and a case of Bud Lite and comment to the cashier that it sure did look like it might snow. I’d schedule a watcher, and tell them to look out for the person who bought six dozen hotdogs and said how it looked like snowing. The watcher would be told to pay special attention to the other purchases and to the shoes of the purchaser. Afterwards, when I heard back from both, I’d ask the watcher what the other purchase was and tell the actor what sort of shoes they were wearing. So the actor knew that I was in the loop and I could verify that the watcher had been there to verify. Everyone had to do their part or I’d know. In this way, I built up the confidence of my players: they knew that the game was for real and that, for whatever reason, if they just did some seemingly random and simple acts from time to time, they’d be paid. I let the actors and watchers take turns at both roles. If they ever asked what the deal was I just told them I was a university psych professor experimenting on social norms.

I admit this took some time. Two months. I had a few dropouts: people who found it too weird, people who tried to cheat by not turning up while claiming they did, people who just dropped out of contact.

When I was ready, I gave all my little helpers an identical task, each at a different bank. Each was to walk into their appointed bank, get in line for a teller, hand the teller a note that said they had a gun and to hand over all the cash from the drawer. Having done that, the actor was to turn and walk straight out of the bank. No watchers, just twelve people playing actors. Everyone dressed the same. I figured several would drop out at this stage, but on the other hand I’d had them doing some pretty silly stuff, pushing the edge, and I knew at least half of them would be up for it. I also know a thing or two about banks. When the note gets to the teller, the first thing they’ll do is trigger the silent alarm. The security guard in the bank will still be clueless, but the bank manager will see it and so will the cops, who are directly connected.

The banks are dotted around midtown Manhattan. The area’s thick with banks and crazy with traffic. Between 11:00 and 11:05 on July 2, the day before the holiday weekend break, the direct-connect police alarm systems of the 5th and 6th precincts go nuts. Cops cars are scrambled and routed to what turns out to be nine simultaneous false-alarm bank robberies. And one real robbery.

I walk into my bank at ten minutes after eleven. I’m dressed in a red shirt, just like my actors who are busy triggering the alarms in a dozen other midtown banks. After a few minutes I reach the teller and hand her a note: “I have a gun. Keep your mouth shut. Pass me all your cash.” After a pause, she does this without a word. That’s the training, the banks learned that one a long time ago. A hero teller or guard taking a bullet winds up costing more than dozens of unsophisticated heists. Plus the banks are fully insured. So teller training is explicit: trigger the silent alarm and do anything the nasty bank robber tells you to.

I sweep the cash into my bag, there couldn’t have been more than $10,000 probably closer to $5,000. I hand her two more notes: “The client at my window has a gun. Keep your mouth shut. Pass all the cash from your drawer to your right.” The other is identical, for the other direction. I indicate that she should pass them the notes, which she does. Handfuls of cash are passed from one to another between five tellers, under the glass to me, where I drop them into my bag. Other customers look on, and I guess they figure something odd is happening, but of course no-one says anything.

Start to finish, I’m there putting money into my bag for less than two minutes. Up until the point the first cash hit my bag, I had a pretty good story: some random person on the internet put me up to it, got me to do all this weird shit, watch others do weird random stuff, paid me through paypal. With the money in my bag though, the game has changed. I know the silent alarm was probably tripped in the first thirty seconds. With your typical mid-morning midtown traffic chaos and the cops already dispatched to at least half a dozen banks apparently being robbed simultaneously, I figure they may not get here for twenty minutes. I’m not counting on the delay any more than I have to, but I figure my margin for error is pretty healthy.

I walk out the front door. Fifteen steps from the bank entrance is a convenient manhole. And look… the cover is already off. The hole is surrounded by four orange cones connected by official-looking Men At Work tape. Of course this is my doing: started and finished a few minutes before entering the bank. From my bag I take a hard hat, some old gloves, and an orange reflective sleeveless vest. Standard issue street crew garb, bought for a few bucks here and there at thrift stores. I pick up the steel tool used for removing manhole covers and drop it into the hole. The rest of my props I leave for the cops. I climb down into the hole and pull the cover across on top of me. Although the covers are heavy, putting one in place is actually a surprisingly easy job when done from below.

The sewers of Manhattan are pretty ugly, despite large-scale efforts to sanitize and modernize them in the 90s. But there’s no need to stay down for long in order to simply disappear. I’ve walked this route before a few times. It’s no big deal. How do I know all this? The plans are on the net: obtained and published online by some urban spelunking group. Along the way I remove the red shirt and put it in my bag. A little over six minutes later I emerge from another manhole in a lane behind a hotel. A couple of suits are standing outside having a smoke. Nothing if not brazen, I bludge a cigarette and a light from one of them. He’s only too happy to help a working man in a hard hat. Show his mates he’s on good terms with blue collar America. With a false mustache, non-prescription glasses, a vest, and the hard hat, there’s not going to be much to go on later, even if these guys do wind up talking to the cops. Besides, as is well known, witnesses are wonderfully unreliable, so the more the merrier as far as I’m concerned. You’ve got to know how these things work and use them to your advantage. I tell the suits I’m on a break from the sewer, got to go to my bank, and off I slouch. Around the first corner I remove the working gear and it goes into my bag too. By my reckoning, the cops wont have even gone down the manhole outside the bank. At most they’ll have a call in for someone to remove the cover. They’ll also know that I’m probably no longer underground.

I walk down into the subway and head home to count my cash ($47,000 and change) and to catch the headline news: Daring daylight robbery! Criminal mastermind! Nine (only nine) banks used to create a distraction while one was well and truly robbed. The police chief is interviewed, indignant: we’ll hunt these men down and put them where they belong. A sophisticated operation like this isn’t done in a vacuum.

Well, good luck chief. For me this is just the beginning, just a proof of concept really. No-one should have to rob banks for a living. That kind of crime doesn’t pay well enough, even if you never get caught. Of course, I have other plans. As the world becomes more digital, less messy, things will only get easier for those who have the balls.

Digital camera found in Barcelona. Do you know these girls?

Saturday, April 19th, 2008

tounges smallWe found a digital camera down in Barceloneta this afternoon. Here are a couple of the images on it.

Do you know these girls?

The menu on the camera is in German.

You can see why I’m desperate to get them their camera back.

three girls small


Paper on the global spread of influenza published in Science

Friday, April 18th, 2008

flu spreadI spent Sept. 2004 to Sept. 2007 as a postdoc in the Zoology Department at the University of Cambridge. We did research into influenza virus using a technique we called Antigenic Cartography.

I don’t want to go into details now or here, but I do want to say that we yesterday published a paper in Science. The paper’s title is The Global Circulation of Seasonal Influenza A (H3N2) Viruses. It digs into how flu viruses circulate around the world and what happens to them in the off season (summer) in temperate zones. This paper was years in the making. And if you consider the data collected by the worldwide influenza surveillance network, it has been decades in the making. As a result the paper has 28 authors, many of whom work at the international flu collaborating centers.

Here’s the official paper in Science. There’s a ton of other coverage, including stories in Science Daily, New Scientist, the LA Times, the Washington Post, National Geographic, Times Online, Reuters, the Associated Press, the Wall Street Journal, and Scientific American. There are plenty more links (currently) available at Google News if you search for influenza.

I’m listed as the #2 author, but it’s really my close friends the first and last authors, Colin Russell and Derek Smith, who did the heavy lifting on making this paper a reality. It’s so nice to see the thing finally published and getting such wide attention.

Could someone please give Natalie Jeremijenko a MacArthur grant?

Monday, April 14th, 2008

natbot

Could someone please give Natalie Jeremijenko a MacArthur “genius” Grant?

Thank you.

I’m not in charge of these things, obviously. If I were though, I’d be hoping to see Natalie nominated so I could give her a grant. She’s a poster child for a genius grant (as is my good friend Derek Smith; but that’s another story). I just spent 2 hours chatting with Natalie in her NYU office.

Here’s a long article about her in Salon.

The MacArthur Foundation moves in mysterious ways. And so does the blogosphere. So I send out this tiny tug on the invisible strings to my invisible readers, asking them to tug in turn. Like many of Natalie’s many projects, a little collective tugging might do wonders.

Everything you think you know is wrong

Friday, April 11th, 2008

wrongI’m often surprised at how confident people are about their knowledge of the world. Looking at the history of thought and of science, you quickly see that it’s strewn with discredited and totally incorrect theories about almost everything. So I don’t understand why it’s not more commonplace to look at history and to arrive immediately at the most likely conclusion: that we too have almost everything wrong.

I don’t mean that literally everything we think is completely wrong. Some things are certainly partly right, or even mainly or fully right. But to have a high degree of confidence, or to assume we’re right just because we know so much more about the world than our ancestors did, or simply because we think we’re right, is just inviting ridicule. Considering our record, and our continual attendant misguided arrogance and confidence along the way, you’d be nuts to think that we know much today or that our confidence adds any weight at all. Many thousands of years of history argue strongly against that conclusion.

Thinking that almost everything is probably wrong in some important fundamental way is a useful default. That attitude stands you in good stead for digging into things, for reconsidering them, for asking questions at a low level. In mathematics when you know for sure that something is wrong (or right) it helps enormously in proving it. It’s a psychological thing. In my dissertation I proved a statistical result that I knew must be true from running simulations. It took me a week or two to nail the proof, and I would never have gotten there if I hadn’t known in advance that the equality I was trying to prove analytically was certainly true (pp 201-207 here in case you’re interested).

As an example of something that I think will be overturned, I think we’ll come to regard our decades of designing computational systems according to the Von Neumann Architecture as extremely primitive. Maybe that will involve some form of analog or quantum computation. I think we’ll take more and more from nature, for instance in solving optimization problems.

On a less grandiose note but still important, I think we’ll look back on our current information architecture and also see it as being extremely primitive. Or, as I’ve said before, we’re living in the shadow of information architecture decisions that were made decades ago. I think that’s all hopelessly wrong. In the real world, information processing simply doesn’t look much like a hierarchical file system.

Hence Fluidinfo.

And so ends another semi-cryptic and ultimately unsatisfying post. I do, as always, plan to eventually say more. And I will.

A curiously empty space in the heart of Manhattan

Wednesday, April 2nd, 2008

empty cup roomI was taken to lunch at the New York Yacht Club today by Ted Carroll of Noson Lawen Partners. By some miracle I happened to be dressed well enough to just scrape in – sans jacket and tie. It’s not the sort of place too many casual NY visitors get to see. Suffice to say, they’re a little on the exclusive side.

After lunch, Ted took me up to the cup room. Or the room that used to be the cup room. You see, there’s a slight problem. No cup. The room was specially built to hold the America’s Cup. It’s perfect, and even has a little viewing platform like the prow of a boat. It’s a beautiful space. And it’s totally empty.

I’m not much of one for nationalistic pride. But I couldn’t resist a little twinge of pleasure recalling that fateful day the Australian boat won the cup after the US had held it for 132 years. Bob Hawke, the Australian Prime Minister, appeared on TV in a bright Green and Gold kangaroo-covered jacket to declare that “any boss who fires a worker for not turning up today is a bum”. It was quite a scene. Good for yachting, I should think, just like when the England cricket team finally beat the Australians a few years ago.

Standing there in the exact spot that the America’s Cup had so immovably and confidently occupied for 132 years was really something. You could almost feel the sense of confusion and cognitive dissonance emanating from that empty space and flowing out to unbalance the entire club building. Ted took photos with his iPhone while I thought of Ozymandias, joked with the staff, and tried to sound like I was from somewhere else.

model roomThen it was upstairs to the banquet hall and model room. There are many hundreds of model yachts on the walls and in glass cases. There are perfect models of every boat to win the America’s Cup, and yes I checked out Ben Lexcen‘s famous winged keel. The accompanying plaque was careful to point out that the boat’s measurements were allowed by the rules. Unwritten: the spirit of yachting itself was shamelessly violated by the genius upstart designer from down under, but, strictly speaking, the boat was legal.

It’s quite a sight.