Posted Sunday, October 14th, 2012 at 11:52 pm under deferreds, python, tech, twisted.

Simpler Twisted deferred code via decorated callbacks

This morning I was thinking about Twisted deferreds and how people find them difficult to grasp, but how they’re conceptually simple once you get it. I guess most of us tell people a deferred is something to hold a result that hasn’t arrived yet. Sometimes, though, deferreds do have a result in them immediately (e.g., using succeed or fail to get an already-fired deferred).

I wondered if it might work to tell people to think of a deferred as really being the result. If that were literally true, then instead of writing:

d = getPage(...)
d.addErrback(errcheck, args)
d.addCallback(cleanup, args)
d.addCallback(reformat, args)
return d

We might write something like:

result1 = getPage(...)
result2 = errcheck(result1, args)
result3 = cleanup(result2, args)
return reformat(result3, args)

And if you could write that, you could obviously instead write:

return reformat(cleanup(errcheck(getPage(...), args), args), args)

If we could write Twisted code that way, I think using deferreds would be simpler for people unfamiliar with them. We could show them Twisted code and not even have to mention deferreds (see below).

In the style we’re all used to, the programmer manually adds callbacks and errbacks. That’s basically boilerplate. It gets worse when you then need to also use DeferredList, etc. It’s a little confusing to read deferred code at first, because you need to know that the deferred result/failure is automatically passed as the first arg to callbacks/errbacks. It seems to take a year or more for people to finally realize how the callback & errback chains actually interact :-) Also, I wonder how comfortable programmers are with code ordered innermost function first, as in the normal d.addCallback(inner).addCallback(outer) Twisted style, versus outer(inner()), as in the line above.

Anyway… I realized we CAN let people use the succinct style above, by putting boilerplate into decorators. I wrote two decorators, called (surprise!) callback and errback. You can do this:

def errcheck(failure, arg):

def cleanup(page, arg):

def reformat(page, arg):

reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3)

The deferred callback and errback chains are hooked up automatically. You still get a regular deferred back as the return value.

And… the “deferred” aspect of the code (or at least the need to talk about or explain it) has conveniently vanished.

You can also do things like

func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3())))

This gets the result of deferreds 2 & 3 and (if neither fails) passes the result of calling func2 on both results through to func1, which is called along with the result of deferred 1. You don’t need to use DeferredLists, as the decorator makes them for you. The errcheck function wont be called at all unless there’s an error.

That’s nice compared to the verbose equivalent:

d1 = DeferredList([getDeferred2(), getDeferred3()])
d2 = DeferredList([getDeferred1(), d1])

Or the more compact but awkward:

DeferredList([getDeferred(), DeferredList([getDeferred(), getDeferred()]).addCallback(

There’s lots more that could be said about this, but that’s enough for now. The code (surely not bulletproof) and some tests are on Github. I’ll add a README sometime soon. This is still pretty much proof of concept, and some it could be done slightly differently. I’m happy to discuss in more detail if people are interested.

  • You’re very welcome! Good luck. BTW, I wrote a book about Javascript deferreds in case you’re interested. The ideas in it are very relevant to Twisted deferreds, but the deferred APIs and behavior are different. It should be useful to understand deferreds, but there are many places you can read about them on the web too. And, there’s a Twisted mailing list that you should join – full of generous and helpful people (many of whom helped me to learn).

  • NCS_One

    Thanks you very much for your anwsers.
    I’m going to follow your advice and start with raw deferreds.
    Thanks again.

  • You should definitely take the time to learn raw deferreds properly. They’re actually very simple but can be hard to think about – everyone has problems with them initially. inlineCallbacks is very convenient in some cases, e.g., when your code has non-trivial conditional branching structure, and in writing tests. It can hugely simplify the reading and writing of your code, but I still think there’s great value in working exclusively at the lower (raw deferred) level initially.

  • NCS_One

    I’m new to twister, soo what do you recommend for new users?
    Raw deferreds or inline callbacks?

  • Hi. No, I don’t think I’d recommend it. I thought of the idea and wanted to see if it was possible. It took some fiddling and head-scratching to get it right. I’m not sure if anyone ever used it. For me it was more of an exercise / challenge. I think you’d learn more by starting with real deferreds before trying to use an abstraction like this whose purpose in a way is to hide what’s really going on. Terry.

  • NCS_One


    Do you still recommend this approach?


  • Hi Michael! Yes. I got a mail on Saturday telling me that discussion was going on. On Sunday I was up at 5:30am with this idea :-) I still haven’t read any of the posts over there, but I saw there were multiple threads and what looked like hundreds of postings. Have you been involved?

  • mcherm

    I’m curious whether the timing of this insight is in any way related to Guido’s recent call for input from people experienced in using deferred and other asynchronous APIs.

  • Hi Clinton. Yes, thanks. I’ve used inlineCallbacks thousands of times (including in the test suite for this code). I can understand your comment, I think. The two have a similar aim of making async code look simpler, but they do it in very different ways. E.g., my decorators don’t use yield. I have one or two posts on inlineCallbacks in the other Twisted posts on this blog – see

    Thanks for commenting!

  • Clinton Roy

    Have a look at the inlinecallbacks decorator, basically, instead of having callbacks for everything, any time you’ve got an an async call to make:

    res = yield async_call()

    For the common case of having one async call at a time its great.