Posted Saturday, September 12th, 2009 at 2:25 pm under FluidDB, python, tech, twisted.

Facebook release Tornado and it’s not based on Twisted?

Image: Jay Smith

Image: Jay Smith

To their great credit, Facebook have just open-sourced more of their core software. This time it’s Tornado, an asynchronous web server written in Python.

Surely that can only mean one thing: Tornado is based on Twisted. Right?

Incredibly, no. Words fail me on this one. I’ve spent some hours today trying to put my thoughts into order so I could put together a reasonably coherent blog post on the subject. But I’ve failed. So here are some unstructured thoughts.

First of all, I’m not meaning to bash Facebook on this. At Fluidinfo we use their Thrift code. We’ll almost certainly use Scribe for logging at some point, and we’re seriously considering using Cassandra. Esteve Fernandez has put a ton of work into txAMQP to glue together Thrift, Twisted, and AMQP, and in the process became a Thrift committer.

Second, you can understand—or make an educated guess at—what happened: the Facebook programmers, like programmers everywhere, were strongly tempted to just write their own code instead of dealing with someone else’s. It’s not just about learning curves and fixing deficiencies, there are also issues of speed of changes and of control. At Fluidinfo we suffered through six months of maintaining our own set of related patches to Thrift before the Twisted support Esteve wrote was finally merged to trunk. That was painful and the temptation to scratch our own itch, fork, and forget about the official Thrift project was high.

Plus, Twisted suffers from the fact that the documentation is not as good as it could be. I commented at length on this over three years ago. Please read the follow-up posts in that thread for an illustration (one of many) of the maturity of the people running Twisted. Also note that the documentation has improved greatly since then. Nevertheless, Twisted is a huge project, it has tons of parts, and it’s difficult to document and to wrap your head around no matter what.

So you can understand why Facebook might have decided not to use Twisted. In their words:

We ended up writing our own web server and framework after looking at existing servers and tools like Twisted because none matched both our performance requirements and our ease-of-use requirements.

I’m betting it’s that last part that’s the key to the decision not to use Twisted.

But seriously…… WTF?

Twisted is an amazing piece of work, written by some truly brilliant coders, with huge experience doing exactly what Facebook set out to reinvent.

This is where I’m at a loss for words. I think: “what an historic missed opportunity” and “reinventing the wheel, badly” and “no, no, no, this cannot be” and “this is just so short-sighted” and “a crying shame” and many things besides.

Sure, Twisted is difficult to grok. But that’s no excuse. It’s a seriously cool and powerful framework, it’s vastly more sophisticated and useful and extensible than what Facebook have cobbled together. Facebook could have worked to improve twisted.web (which everyone agrees has some shortcomings) which could have benefitted greatly from even a small fraction of the resources Facebook must have put into Tornado. The end result would have been much better. Or Facebook could have just ignored twisted.web and built directly on top of the Twisted core. That would have been great too.

Or Facebook could have had a team of people who knew how to do it better, and produced something better than Twisted. I guess that’s the real frustration here – they’ve put a ton of effort into building something much more limited in scope and vision, and even the piece that they did build looks like a total hack built to scratch their short term needs.

What’s the biggest change in software engineering over the last decade? Arguably it’s the rise of test-driven development. I’m not the only one who thinks so. Yet here we are in late 2009 and Facebook have released a major piece of code with no test suite. Amazing. OK, you could argue this is a minor thing, that it’s not core to Tornado. That argument has some weight, but it’s hard to think that this thing is not a hack.

If you decide to use an asynchronous web framework, do you expect to have to manually set your sockets to be non-blocking? Do you feel like catching EWOULDBLOCK and EAGAIN yourself? Those sorts of things, and their non-portability (even within the world of Linux) are precisely the kinds of things that lead people away from writing C and towards doing rapid development using something robust and portable that looks after the details. They’re precisely the things Twisted takes care of for you, and which (at least in Twisted) work across platforms, including Windows.

It looks like Tornado are using a global reactor, which the Twisted folks have learned the hard way is not the best solution.

Those are just some of the complaints I’ve heard and seen in the Tornado code. I confess I’ve looked only superficially at their code – but more than enough to feel such a sense of lost opportunity. They built a small subsection of Twisted, they’ve done it with much less experience and elegance and hiding of detail than the Twisted code, and the thing doesn’t even come with a test suite. Who knows if it actually works, or when, or where, etc.?

And…. Twisted is so much more. HTTP is just one of many protocols Twisted speaks, including (from their home page): “TCP, UDP, SSL/TLS, multicast, Unix sockets, a large number of protocols including HTTP, NNTP, IMAP, SSH, IRC, FTP, and others”.

Want to build a sophisticated, powerful, and flexible asynchronous internet service in Python? Use Twisted.

A beautiful thing about Twisted is that it expands your mind. Its abstractions (particularly the clean separation and generality of transports, protocols, factories, services, and Deferreds—see here and here and here) makes you a better programmer. As I wrote to some friends in April 2006: “Reading the Twisted docs makes me feel like my brain is growing new muscles.”

Twisted’s deferreds are extraordinarily elegant and powerful, I’ve blogged and posted to the Twisted mailing list about them on multiple occasions. Learning to think in the Twisted way has been a programming joy to me over the last 3 years, so you can perhaps imagine my dismay that a company with the resources of Facebook couldn’t be bothered to get behind it and had to go reinvent the wheel, and do it so poorly. What a pity.

In my case, I threw away an entire year of C code in order to use Twisted in FluidDB. That was a decision I didn’t take lightly. I’d written my own libraries to do lots of low level network communications and RPC – including auto-generating server and client side glue libraries to serialize and unserialize RPC calls and args (a bit like Thrift), plus a server and tons of other stuff. I chucked it because it was too brittle. It was too much of a hack. It wasn’t portable enough. It was too get the details right. It wasn’t extensible.

In other words….. it was too much like Tornado! So I threw it all away in favor of Twisted. As I happily tell people, FluidDB is written in Python so it can use Twisted. It was a question of an amazingly great asynchronous networking framework determining the choice of programming language. And this was done in spite of the fact that I thought the Twisted docs sucked badly. The people behind Twisted were clearly brilliant and the community was great. There was an opportunity to make a bet on something and to contribute. I wish Facebook had made the same decision. It’s everyone’s loss that they did not. What a great pity.

  • Pingback: Cirujano Plastico Monterrey()

  • lvh

    And that's a crying shame. But it would've been awesome if we heard from these folks back when they decided to do or not do something, so we could figure out what they wanted but wasn't there.

  • lvh

    And that's a crying shame. But it would've been awesome if we heard from these folks back when they decided to do or not do something, so we could figure out what they wanted but wasn't there.

  • casper

    I am really glad facebook did not use twisted and pushed for their own solution. It brings fresh wind. And that is what twisted really needs.

  • Thomas Small

    The take away that Twisted should take from this is that it's easier to write an functioning io loop than it is to wade through Twisted's documentation and code.

  • Hi Lacker,

    I don't think your description is quite accurate :). Twisted isn't too complicated for what you want, twisted is too *simple* for what you want. Which is to say that Twisted doesn't have a Comet backend, so doing what you want is a lot of work.

    But you could just use Orbited <>, without writing any code of your own. As it happens it's already Twisted-based, so you can use Twisted code with it if you want to. (Worth noting that Orbited *also* started off not using Twisted, but then they realized it was actually easier to use Twisted than to deal with pyevent directly :-)).

  • HI Lacker

    Yes, the competition is great. But see my comments above re the needs of the individual programmer (I agree, Twisted is too complex too hard to get into) and the needs of a company that takes this stuff so seriously that they're prepared to write the whole thing from scratch. Those are very different cases. In the first I'm totally with you. In the second (I, and Fluidinfo our company) thought it *much* better to go with Twisted, even if it meant throwing out a year of code. We're perfectly capable of writing it ourselves (and I did to some extent – the extent where I had a functional product), but we're just not that good. It made more sense to throw in our lot with Twisted, to help make that better, and to be happy to have people like Glyph and JP dealing with tons of nitty gritty details. They've been doing it for a very long time. There's something about community too which rankles me.

    Thanks for stopping by!

  • Hi Gregg. Yes, agreed. I do the same – I used Django for serving the automated docs of FluidDB (at*/*/*) and we even used Rails for because it was simple.

    But that's you and I as individuals doing smallish projects. If you're a well funded company looking to build a robust asynchronous web server and you're deliberately taking the decision to build your own reactor, your own non-blocking I/O, your own web server, then you obviously have many more resources available, you're probably much more concerned about scaling and getting things really right, and you obviously have a pretty high tolerance for pain & the effort involved in producing such a thing. In that case, the effort spent to learn Twisted pales in comparison to the effort required to build your own from scratch. It boggles my mind that they didn't show up in the Twisted community, ask a few questions, etc. You can understand from some points of view, of course, but I find it totally weird, *especially* given that the end result is something that's not better than Twisted. The fact that it's easier to use almost certainly wasn't one of the design goals – I doubt they really cared about that; this wasn't built with community in mind, or they'd have gone about it differently (like talking to the community for starters). It was built as a hack to scratch their own needs by programmers who knew they could get the job done to the extent they needed to. That's all a guess, of course. I don't condemn them for that, I'm sure we've all done the same kind of thing (I know I have). I just think it's a great pity, and (from the non-Friendfeed perspective) is very short-sighted and is based on a false economy. I guess the non-Friendfeed perspective wasn't worth worrying about at the time.

  • Hi Manuel

    Yes, I guess I'm wrong. I guess you'd just subclass HTTPServer and use the listen class of your parent. Your “just like …Twisted does” is more to the point – Twisted already does all this stuff (and much more) and much more portably. Reinventing the wheel badly is my major frustration, and that all that effort should be expended when just a fraction of it could have been used to make twisted.web better, meaning a better outcome for all. Anyway, thanks for commenting.

  • Pingback: Tornado Web Server « My Python (& Vista!) stuff()

  • Pingback: Tornado « Blog MichaƂa()

  • Pingback: When Great Features Aren’t Enough: Twisted, Tornado, the Zero-Step, and Activation Energy « write-only by Gregg Lind()

  • I've never been happy with Twisted. It's far too complicated for what I usually want, which is just a python backend for a comet frontend. And in my experience other libraries that use Twisted tend to be buggy. So I'm glad there's a competitor.

  • As a person, I like having my mind expanded. As a dev, sometimes I just want to get things done. I wrote a longish essay about why a bit of chrome, and a good “hello, world” matters in this over at…. Twisted is truly great, but learning to use is should be gentler. For all all its faults, Tornado looks to start off gentle. Maybe this will spur on Twisted.web :)

  • Err… FWIW, I don't think you have to set the socket options yourself when you use Tornado… Tornado does that internally, just like I assume that Twisted does.

  • The cool fact is that it seems not that complex replace the core piece of code in order to leverage the maturity of Twisted:

  • I didn't even know that Tornado had come out of Friendfeed…