Posted Saturday, August 14th, 2010 at 2:25 am under deferreds, python, twisted.

Twisted’s Deferred class chainDeferred method is too simplistic?

I think it’s worth spending some time thinking about whether chainDeferred in twisted.internet.defer.Deferred is too simplistic. I’ve thought for a while that it could be more helpful in preventing people from doing unintended things and/or cause fewer surprises (e.g., see point #1 in this post).

Those are smaller things that people can obviously live with. But a more important one concerns deferred cancellation. See below for why.

Here are five examples of chainDeferred behavior that I think could be better. In an attempt to fix these, I made a few changes to a copy of (immodestly called, sorry) which you can grab, with runnable examples, here. The code is meant as a suggested approach. I doubt that it’s bulletproof.

Here are the examples.

# Normal deferreds: this raises defer.AlreadyCalledError because
# the callback of d1 causes the callback of d2 to be called, but d2 has
# already been cancelled (and hence called).

# With there is no error because d1.callback will not call
# d2 as it has already been cancelled.

def printCancel(fail):
    print ‘cancelled’

def canceller(d):
    print ‘cancelling’

d1 = defer.Deferred()
d2 = defer.Deferred(canceller)

# Normally: raises defer.AlreadyCalledError because calling d1.callback
# will call d2, which has already been called.

# With Raises AssertionError: "Can’t callback an already
# chained deferred" because calling callback on a deferred that’s
# already been chained is asking for trouble (as above).

d1 = defer.Deferred()
d2 = defer.Deferred()

# Normally: although d2 has been chained to d1, when d1 is cancelled,
# d2’s cancel method is never called. Even calling d2.cancel ourselves
# after the call to d1.cancel has no effect, as d2 has already been
# called.

# With tdefer: both cancel1 and cancel2 are called when d1.cancel is
# called. The additional final call to d2.cancel correctly has no
# effect as d2 has been called (via d1.cancel).

def cancel1(d):
    print ‘cancel one’

def cancel2(d):
    print ‘cancel two’

def reportCancel(fail, which):
    print ‘cancelled’, which

d1 = defer.Deferred(cancel1)
d1.addErrback(reportCancel, ‘one’)
d2 = defer.Deferred(cancel2)
d2.addErrback(reportCancel, ‘two’)

# Normally: prints "called: None", instead of the probably expected
# "called: hey"

# prints "called: hey"

def called(result):
    print ‘called:’, result

d1 = defer.Deferred()
d2 = defer.Deferred()

# Normally: prints
#   called 2: hey
#   called 3: None

# prints
#   called 2: hey
#   called 3: hey

def report2(result):
    print ‘called 2:’, result

def report3(result):
    print ‘called 3:’, result

d1 = defer.Deferred()
d2 = defer.Deferred().addCallback(report2)
d3 = defer.Deferred().addCallback(report3)

I wont go into detail here as this post is already long enough. Those are 3 classes of behavior arising from chainDeferred being very simplistic. Comments welcome, of course. Once again, the runnable code is here.

  • Personally, I've never understood why chainDeferred exists at all, and I'm not sure there's any real reason to use it. In fact, since a recent fix for deferred chaining (which, confusingly, has nothing to do with the chainDeferred method) landed in trunk, there is a very good reason *not* to use chainDeferred: it doesn't have this fix. In any event, every time I've thought that I might want to use chainDeferred, it turned out not to do what I wanted.