Fluidinfo

If you’re a programmer and you’ve played around with Tickery, it should be clear that Tickery is functionally very simple when looked at from a traditional database perspective. Tickery looks like an application in its own right. It tries to offer something so simple that anyone (at least any Twitter user) can understand and use the Simple “Enter two Twitter usernames” functionality. But we actually designed and built Tickery mainly as a demo of what’s possible with Fluidinfo (description, API). So here are some first details on how Tickery uses Fluidinfo, and how you can use it too.

Fluidinfo objects

The most important thing to understand about Fluidinfo initially is that it maintains a collection of tagged objects that are not owned. Tags have permissions and a tag on an object can optionally have a value associated with it. Here’s a conceptual view of an object that has two tags on it that were added by Tickery.

The long identifier is the object’s unique id in Fluidinfo. The left column shows tag names, such as twitter.com/users/screen_name, and the right column shows the value (if any) of the tag on the object. Because objects in Fluidinfo are not owned, anyone (which is to say any application) can put additional tags onto this object. I’m going to ignore permissions in what follows – that’s a subject for a separate posting.

Any application can find this object in Fluidinfo, using a simple query, like twitter.com/users/id = 42983 or twitter.com/users/screen_name = "terrycojones".

Now let’s suppose Tickery adds @esteve to Fluidinfo, and wants to indicate that Esteve currently follows terrycojones. Tickery creates a new tag, twitter.com/friends/esteve in Fluidinfo, and adds it to the above object. The object then looks like this:

Similarly, Tickery adds a twitter.com/friends/esteve to the objects representing all the Twitter users Esteve follows. At this point it is easy to retrieve all those users via the Fluidinfo query has twitter.com/friends/esteve (i.e., get me all the objects that have a twitter.com/friends/esteve tag, irrespective of the tag’s value, if any).

Suppose Tickery now adds another Twitter user, @fergusstothart who currently follows me. It adds another tag to the object, resulting in

and also puts a twitter.com/friends/fergusstothart tag onto the objects for the other users that Fergus follows. It finds these objects via a Fluidinfo query (using the Twitter id of Fergus, obtained from the Twitter API). If Tickery needs to tag an object for a user that it hasn’t created yet, it simply creates a new object for that user, and tags it.

Given the above, we’ve seen enough to know how Tickery does most of its work. For example, getting things like the set of people Esteve and Fergus follow in common is just an and query has twitter.com/friends/esteve and has twitter.com/friends/fergusstothart, or the set of people Esteve follows but Fergus does not has twitter.com/friends/esteve except has twitter.com/friends/fergusstothart, etc.

Where we come in

That’s all well and good, but it’s all about Tickery. What if other programmers, who perhaps don’t care or even know about Tickery want to add data and search on it too? In a normal database or with a normal application, you’d probably expect to have to ask permission. Then, supposing it was granted, you could only do the kinds of things that had been anticipated and provided for.

But in Fluidinfo it’s completely different. Any application can come along and put whatever it likes onto the above object, or any other object (that it can find). As a trivial example, Esteve and I have also added tags to the Fluidinfo objects to indicate which of the people we follow we have also met in person. Esteve has an esteve/met tag, and because we’ve met (in fact we built Fluidinfo together), he has put that tag onto the above object:

Think about what just happened. An unknown 3rd party (well, let’s pretend Esteve was unknown) just came along, sometime after the Tickery data already existed in Fluidinfo, and added something completely new and unanticipated to an existing object, without asking for permission, and without in any way disturbing the original content. Esteve, or anyone else who can read his tag, can now do interesting searches, like has twitter.com/friends/esteve except has esteve/met, which shows the people he follows but has not yet met. Further, his searches seamlessly combine the existing Tickery data with his own data, and could also include other tags that other applications add.

That kind of unanticipated use of information, flexibility in representation and search, and change of control is what’s at the heart of Fluidinfo.

Twitter lists

If you think about it, Esteve adding esteve/met tags to the objects for Twitter users is exactly like making a list in Twitter using their new lists function. But it’s more useful for two main reasons.

Firstly, you can query across lists, e.g., has terrycojones/met and has esteve/met will find people we have both met in person, and (has twitter.com/friends/esteve and has terrycojones/met) except has esteve/met will find people Esteve follows that I have met in person but whom he has not. As you can see, querying on lists makes them much more useful.

Secondly, you can use the Fluidinfo permissions system to control who can see or read your tags. So it’s not only possible to have a completely private list, a public one, but you can also have a list that’s visible just to some friends, or one that you let certain other people add to (by giving them write permission on the tag involved).

Permissions in Fluidinfo are very simple and very flexible, and because they apply at the level of the tag (not the object), you can control who can do what to individual pieces (tags) of an object. That’s a subject for another post, as I mentioned. You might like to have a read of the Fluidinfo permissions docs and/or check out Nicholas Radcliffe‘s post Permissions Worth Getting Excited About, plus see the comment by Nicholas Tollervey, who writes Fluidinfo’s "killer feature" is actually its permissions system and the implications thereof. It is so important that I’ll save that topic for its own blog post later.

More Tickery tags

Tickery also saves a few more Twitter user details onto objects in Fluidinfo. The object above has some additional tags:

You can put these into queries too, of course. The Tickery Advanced tab lets you type them in, e.g., I can see which of the people that Jack follows are very popular has twitter.com/friends/jack and twitter.com/users/followers_count > 100000.

Running on ahead of Tickery

Finally, here’s a subtle but very important point. What if you write an application that uses Fluidinfo to store data, and you want it to interact fully with Tickery, but you want to store information about a user that Tickery doesn’t know about yet?

This is crucially important because it’s about information control, and if control is completely in the hands of Tickery, other developers will be less likely to want to add information. Exactly this scenario plays out in many domains: e.g., suppose Amazon released something that let you indicate which books you own, but that you own things that are not in the Amazon database. How can you run on ahead of Amazon to insert your data before they create an object for the book, if ever? How can you do it in a way so that when they finally do create the book your data and theirs are seamlessly joined without anyone having to lift a finger or even be aware of the other party?

This is one area in which the special Fluidinfo “about” tag (full name fluiddb/about) makes all the difference. You can read about it here, and be sure to check out Nicholas Radcliffe‘s blog which is titled, not coincidentally, About Tag.

Tickery uses the Fluidinfo about tag to hold a Twitter user id, like this:

There’s a ton that could be written about this. Very briefly, the about tag is immutable and can only be set on an object when it’s created (in fact the about tag shown above was put there when the object was made). So, if you want to add data to Twitter user that Tickery hasn’t gotten to, just look up the user’s Twitter id (say XXXX) with the Twitter API, create the object in Fluidinfo with about tag twitter.com:uid:XXXX, and put your tags onto that object. If Fluidinfo doesn’t have an object with that about tag, it will make one for you. When/if Tickery gets around to adding its information for that user, it will put it in the same place. Magic.

Convenience API

As a convenience, though note that it’s optional and it’s use is up to you, Tickery provides a small API that you can use to have it put its twitter.com/users/screen_name and twitter.com/users/id tags onto objects for you and give you the Fluidinfo object id its using for a user.

E.g., if you do an HTTP GET on http://tickery.net/api/screennames/terrycojones you’ll see the object id from our examples above. Or if you happened to know my Twitter user id (42983) via the Twitter API, you could do a GET on http://tickery.net/api/uids/42983 and receive the same thing.

Truly social data

This API is just for convenience. Tickery uses the about tag in order to be able to share Fluidinfo objects with other apps – including apps that want to add information about a user that Tickery has not gotten to yet. Just like Fluidinfo, Tickery wants to encourage what we call Truly Social Data. Tickery doesn’t place itself in the center, doesn’t make its data more important than anyone else’s, and doesn’t act as a gatekeeper.

In fact, it gets even better: a normal user can turn around and stop Tickery from reading the data that Tickery stored on the user’s behalf. That’s as it should be. Users should have control over their data, and a choice of application shouldn’t result in lock in.

Getting access

Fluidinfo is still very new, and we’re in a private alpha phase. If you’d like to use the API, there are two steps: 1) reserve a username, and 2) send us email mentioning the name you reserved and a line or two about what you’d like to do. We apologize for this early restriction – please rest assured that we’re planning to open Fluidinfo up to everyone before too long. That’s the whole point.

More soon. Thanks very much for reading!

Various articles have recently discussed adding metadata to Twitter tweets – see the posts by Nova Spivack, Robert Scoble, and Dave Winer (who also suggests we need a programming language built into a Twitter client).

These are the sorts of things that Fluidinfo was designed to support, and you can do them today. If you want a password to start playing with the Fluidinfo API, send email to api at fluidinfo dot com and we’ll set you up.

In the meantime, here are some examples. I’m doing this at the iPython command line, using the Fluid Object Mapper (FOM) library, written by Ali Afshar. FOM provides a natural way to work with Fluidinfo objects, namespaces, tags, etc. But you could use any client-side software you like. The Fluidinfo API is just HTTP.

First, let’s get a connection to Fluidinfo:

from fom.session import Fluid

fdb = Fluid()
fdb.db.client.login('terrycojones', 'PASSWORD')
fdb.bind()

That last line is a bit of internal FOM magic that makes interactive use simpler in what follows. Ignore it for now.

To put metadata onto a tweet, we’ll first ask Fluidinfo for the object that’s about a particular tweet. Let’s take the one in the image above by @novaspivack. That tweet has a URL of http://twitter.com/novaspivack/status/4999653280. We ask Fluidinfo to give us the object “about” that URL:

from fom.mapping import Object

o = Object()
o.create('http://twitter.com/novaspivack/status/4999653280')
o.uid
>>> u'ab7fa032-06df-45be-9bb2-859c18c4d342'

The argument in the o.create call is the value of the Fluidinfo about tag. If an object with that about tag already exists, Fluidinfo gives it to us. Otherwise, a new object with that about tag is created. As you can see, the object also has an identifier (o.uid). In case you’re not familiar with Python, the “u” printed in front of the id indicates that the value is a unicode string.

This is a first point of interest. We’ve just created a Fluidinfo object corresponding to an arbitrary tweet. We didn’t ask for permission, we just did it. It’s a bit like a wiki: you can ask a wiki for its page on anything, and if no such page exists, the wiki just makes you a new one. Fluidinfo does the same thing with its objects and about tag. If you want to think about Fluidinfo in all its generality, you should now consider that the about tag above could have been for any tweet, including tweets that don’t exist (or don’t yet exist), for any URL, in fact for any string. We also could have followed Nova’s suggestion and used an about value like "twitter.com/id=4999653280". But we’re getting ahead of ourselves.

Fluidinfo has a simple query language, so let’s quickly confirm that we can find this object with a search:

fdb.objects.get('fluiddb/about = "http://twitter.com/novaspivack/status/4999653280"')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

The 200 is an HTTP OK status telling us the call succeeded, and you can see one object matched the search and that its id is as expected.

So how about some metadata? Let’s say I want to add a rating to the object. Here’s a bit of one-time setup. First I get my top-level namespace (which corresponds to my Fluidinfo user name). Then I create a new tag called rating in that namespace:

from fom.mapping import Namespace

ns = Namespace('terrycojones')
ns.create_tag("rating", "A tag for Terry's ratings.", False)

The False argument is telling Fluidinfo that I don’t want the tag to be indexed. Ignore that for now.

The magic of FOM lets us directly examine the tag using Python attributes. So you can get the tag and see its description like so:

rating = ns.tag('rating')
rating.description
>>> u"A tag for Terry's ratings."

At this point we have a new tag, or an abstract tag if you prefer, but we haven’t actually tagged any objects with it. So let’s tag the object we created above for Nova’s tweet:

o.set('terrycojones/rating', 6)

That was pretty easy! The Fluidinfo object that’s about Nova’s tweet now has some metadata on it, a ‘terrycojones/rating’ tag, with a value of 6. Let’s make sure we can get that value back:

o.get('terrycojones/rating')
>>> (6, None)

We get a 2-tuple whose second value is None when the tag’s value is a primitive Python type (in this case an integer).

Let’s do a couple of quick searches for objects with terrycojones/rating tags:

fdb.objects.get('terrycojones/rating = 6')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

fdb.objects.get('terrycojones/rating > 4')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

fdb.objects.get('has terrycojones/rating')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

In each case just that one object is returned, as expected. Note that the last query just tests for the presence of the tag, irrespective of the tag’s value (if any).

So there you have it: arbitrary metadata on tweets, and with a query language to help find things.

But let’s press on and see how things get more interesting.

First of all, you may have noticed that I didn’t have to deal with permissions at all in the above. I was able to create the Fluidinfo object about Nova’s tweet and to tag it without asking permission. In Fluidinfo that’s always the case.

But there is a permissions system. Let’s log in as a different user and try a few things to see how it works. First, I’ll log in as njr another user whose password I happen to know:

fdb.db.client.login('njr', 'PASSWORD')

The njr user is actually Nicholas Radcliffe who has written several great introductory articles about Fluidinfo over at About Tag.

Let’s try (as Nick) getting the terrycojones/rating tag from the object for Nova’s tweet:

o.get('terrycojones/rating')
>>> (6, None)

That still works, so we can infer that the terrycojones/rating tag is readable by the njr user. Let’s log in as terrycojones again and have a look at the permissions:

fdb.db.client.login('terrycojones', 'PASSWORD')
fdb.permissions.tag_values['terrycojones/rating'].get('read')
>>> (200, {u'exceptions': [], u'policy': u'open'}

We’ve asked Fluidinfo for read permissions on tag values for the tag terrycojones/rating. The result is a general policy (open), with exceptions (currently empty). Now I’ll put the njr user into the exceptions list, and confirm the result:

fdb.permissions.tag_values['terrycojones/rating'].put('read', 'open', ['njr'])
>>> (204, None)
fdb.permissions.tag_values['terrycojones/rating'].get('read')
>>> (200, {u'exceptions': [u'njr'], u'policy': u'open'}

The 204 status above is just the HTTP way of telling us that the call succeeded and that the reply has no content (as expected).

Now let’s reconnect as njr and try getting the terrycojones/rating tag again:

fdb.db.client.login('njr', 'PASSWORD')
o.get('terrycojones/rating')
>>>

You can see we got nothing back. If FOM handled non-OK HTTP responses a little more carefully, you’d see that this request actually got a 401 (Permission Denied) status. Fluidinfo is now refusing to let njr read the tag.

Nick already has a rating tag, called njr/rating, so let’s go get it, make sure there’s not one already on our object, and then tag our object with it:

ns = Namespace('njr')
rating = ns.tag('rating')
o.get('njr/rating')
o.set('njr/rating', 4)
o.get('njr/rating')
>>> (4, None)

Now things are getting interesting. We have tags from different users on the same object. That’s part of the point of Fluidinfo and its where the value comes from: putting information together allows you to do nice things, like query on it. After re-connecting as terrycojones, I can now do queries like this:

fdb.objects.get('terrycojones/rating > 5 and njr/rating > 3')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

fdb.objects.get('terrycojones/rating > 5 and njr/rating < 3')
>>> (200, {u'ids': []})

fdb.objects.get('has terrycojones/rating and njr/rating >= 4')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

fdb.objects.get('has terrycojones/rating and has njr/rating')
>>> (200, {u'ids': [u'ab7fa032-06df-45be-9bb2-859c18c4d342']})

fdb.objects.get('has terrycojones/rating except has njr/rating')
>> (200, {u'ids': []})

There’s a lot more I could do too, like giving Nick permission to add terrycojones/rating tag to objects. By the way, Nick has written some nice articles about the Fluidinfo permissions model. See Permissions Worth Getting Excited About and The Permissions Sketch.

For a final look at metadata, let’s put something totally different onto our object:

ns = Namespace('terrycojones')
page = ns.create_tag("page", "Terry's page tag.", False)
o.set('terrycojones/page', 'Hello there!', 'text/html')

I’ve just made a new tag called terrycojones/page and tagged our object with it. What’s different here is that the value is a string, and I’m passing a MIME type with it. If I retrieve the value of the tag on the object, you’ll see the MIME type comes back too:

o.get('terrycojones/page')
>>> ('Hello there!',
 'text/html')

and as you might hope, if you go get that tag from that object using a browser, the MIME type is returned in the HTTP Content-type header, so you end up with a real web page, with a predictable URL. Try clicking: http://fluiddb.fluidinfo.com/objects/ab7fa032-06df-45be-9bb2-859c18c4d342/terrycojones/page. We can do the same for any MIME type at all – including ones you invent for your own convenience.

So there you go. That’s metadata on tweets. With a permissions model, with a query language, with user identity, with the freedom to add anything you want, and with typed data. We don’t need a new programming language for doing this sort of thing. What we need is a better data architecture.

Fluidinfo was designed with exactly this kind of use in mind. And it’s not specific to Twitter or tweets, or anything in fact. So you can put metadata onto anything you like, search on it, continue to own/control your own data, combine it as you like, and get data in and out using a simple HTTP API.

This is all live. It’s up and running, you can do this today. I should also add that Fluidinfo is still an early alpha, and is not yet particularly fast. For more information on Fluidinfo, start with the high-level description and if you’re a programmer, read the API docs.

Next time I’ll show you how we’re putting metadata onto Twitter users, and how you can too, of course! I might also start to talk about Tickery, our upcoming Twitter query application.

If you like all this, please pass on this article. We’d love to get the word out about Fluidinfo. It’s a little difficult from Barcelona.