A Python metaclass for Twisted allowing __init__ to return a Deferred
OK, I admit, this is geeky.
But we’ve all run into the situation in which you’re using Python and Twisted, and you’re writing a new class and you want to call something from the
__init method that returns a Deferred. This is a problem. The
__init method is not allowed to return a value, let alone a Deferred. While you could just call the Deferred-returning function from inside your
__init, there’s no guarantee of when that Deferred will fire. Seeing as you’re in your
__init method, it’s a good bet that you need that function to have done its thing before you let anyone get their hands on an instance of your class.
For example, consider a class that provides access to a database table. You want the
__init__ method to create the table in the db if it doesn’t already exist. But if you’re using Twisted’s
twisted.enterprise.adbapi class, the
runInteraction method returns a Deferred. You can call it to create the tables, but you don’t want the instance of your class back in the hands of whoever’s creating it until the table is created. Otherwise they might call a method on the instance that expects the table to be there.
A cumbersome solution would be to add a callback to the Deferred you get back from
runInteraction and have that callback add an attribute to
self to indicate that it is safe to proceed. Then all your class methods that access the db table would have to check to see if the attribute was on
self, and take some alternate action if not. That’s going to get ugly very fast plus, your caller has to deal with you potentially not being ready.
I ran into this problem a couple of days ago and after scratching my head for a while I came up with an idea for how to solve this pretty cleanly via a Python metaclass. Here’s the metaclass code:
def __new__(mcl, classname, bases, classdict):
hidden = ‘__hidden__’
instantiate = ‘__instantiate__’
for name in hidden, instantiate:
if name in classdict:
‘Class %s contains an illegally-named %s method’ %
origInit = classdict[‘__init__’]
origInit = lambda self: None
def newInit(self, *args, **kw):
hiddenDict = dict(args=args, kw=kw, __init__=origInit)
setattr(self, hidden, hiddenDict)
return (self, result)
hiddenDict = getattr(self, hidden)
d = defer.maybeDeferred(hiddenDict[‘__init__’], self,
classdict[‘__init__’] = newInit
classdict[instantiate] = _instantiate
return super(TxDeferredInitMeta, mcl).__new__(
mcl, classname, bases, classdict)
I’m not going to explain what it does here. If it’s not clear and you want to know, send me mail or post a comment. But I’ll show you how you use it in practice. It’s kind of weird, but it makes sense once you get used to it.
First, we make a class whose metaclass is TxDeferredInitMeta and whose
__init__ method returns a deferred:
__metaclass__ = TxDeferredInitMeta
d = aFuncReturningADeferred()
__init__ return anything other than
None is illegal in normal Python classes. But this is not a normal Python class, as you will now see.
Given our class, we use it like this:
# instance is an instance of MyClass
# result is from the callback chain of aFuncReturningADeferred
d = MyClass()
That may look pretty funky, but if you’re used to Twisted it wont seem too bizarre. What’s happening is that when you ask to make an instance of MyClass, you get back an instance of a regular Python class. It has a method called
__instantiate__ that returns a Deferred. You add a callback to that Deferred and that callback is eventually passed two things. The first is an instance of MyClass, as you requested. The second is the result that came down the callback chain from the Deferred that was returned by the
__init__ method you wrote in MyClass.
The net result is that you have the value of the Deferred and you have your instance of MyClass. It’s safe to go ahead and use the instance because you know the Deferred has been called. It will probably seem a bit odd to get your instance later as a result of a Deferred firing, but that’s perfectly in keeping with the Twisted way.
That’s it for now. You can grab the code and a trial test suite to put it through its paces at http://foss.fluidinfo.com/txDeferredInitMeta.zip. The code could be cleaned up somewhat, and made more general. There is a caveat to using it – your class can’t have
__instantiate__ methods. That could be improved. But I’m not going to bother for now, unless someone cares.