Archive for the ‘Programming’ Category

The structure of O’Reilly book and author data in Fluidinfo

Monday, March 21st, 2011

This short post explains how the O’Reilly catalog is represented in Fluidinfo.

Put simply, we annotate two types of object: those representing products (usually books) and those representing authors. We annotate them using namespaces and tags within the oreilly.com top level namespace so you can be sure that this is bona fide O’Reilly information.

Within the oreilly.com namespace we store a bunch of “top level” tags that describe a product in the O’Reilly catalogue (title, summary, URL and so on). The oreilly.com namespace has two child namespaces: “authors” and “media“. (If you want a visual representation of this structure head on over to the Fluidinfo explorer and explore, starting from the tree menu on the left hand side.)

The authors namespace contains tags that define information about an author (name, biography, homepage and so on) and also contains a child namespace called “expertise“. The expertise namespace contains a set of tags that map to the list of areas of expertise that O’Reilly uses to categorise their authors. So, for example, an object representing the O’Reilly author “Chris DiBona” looks like this:

Notice how Chris’s object has tags under the oreilly.com/authors namespace including several under the oreilly.com/authors/expertise namespace. Importantly, the object also has tags that were not provided by the O’Reilly data. Terry has added a tag terrycojones/met to indicate (rather obviously) that he’s met Chris and the fluiddb/about tag is used to indicate that the object is about the author called Chris diBona.

What about the objects that represent books..? What do they look like..? Well let’s consider a current favourite of mine: “XMPP: The Definitive Guide”. Here’s how Nick Radcliffe’s excellent abouttag utility displays the object representing this book:

Whoa! Lots more tags! Many of them are from the oreilly.com domain (although notice how there are 15 missing). Once again it’s possible to see who/what else has been tagging the object. I’ve added a review and rating (ntoll/review and ntoll/rating) and various other people have annotated useful information that wasn’t at first in the dataset provided by O’Reilly.

How are authors and books linked..?

Every author object has an oreilly/authors/works tag that contains a list of the 13 digit O’Reilly ID / ISBN for each work they were involved in. Every book object has a corresponding oreilly.com/id and oreilly.com/isbn tag.

Alternatively, every book object has an oreilly.com/authors-urls tag that contains a list of it’s author’s homepages on the O’Reilly website and every author object has an associated oreilly.com/url containing the same information.

Finally, for the sake of completeness here’s a list of all the book and author tags along with a description of what each one represents:

Book tags

  • publication-day: The day of the month upon which the item was published.
  • publication-month – The number of the month within which the item was published.
  • duration – The duration of this item in minutes.
  • subtitle – The subtitle associated with the item.
  • id – The unique ID used by O’Reilly to identify the item, usually the 13-digit ISBN number (as a string).
  • page-count-is-estimate – A flag to indicate that any associated page count value is only an estimate.
  • cover-medium – The URL for a medium size image of the cover at the oreilly.com domain.
  • toc – The table of contents as text/html.
  • homepage – A URL to the item’s homepage on the O’Reilly website.
  • description – A long description of the item as text/html.
  • cover-small – The URL for a small size image of the cover at the oreilly.com domain.
  • author-urns – A list of unique reference numbers used by O’Reilly to reference the authors of the item.
  • cover-large – The URL for a large size image of the cover at the oreilly.com domain.
  • isbn – The 13-digit ISBN number (as a string).
  • safari-url – A URL to the item’s page on O’Reilly’s Safari service.
  • author-urls – A list of URLs pointing to the author’s homepages on the O’Reilly website.
  • pages – The number of pages this item has.
  • publisher – The name of the publisher of the item.
  • price-us – The advertised US price in cents.
  • title – The title of the item.
  • author-names – A list of author names.
  • summary – A short summary of the item as text/html.
  • publication-date – The publication date as YYYY-MM-DD.
  • price-uk – The advertised UK price in pence.
  • media – A list of the type[s] of media in which the item is available. Can be one or more of: ‘up-to-date’, ‘rough cut’, ‘dvd’, ‘ebook’, ‘kit’, ‘video’, ‘print’, ‘early release ebook’, ‘safari books online’ or ‘merchandise'”

Author tags

  • name – The author’s full name.
  • url – A URL to the author’s homepage on the O’Reilly website.
  • photo – A path to an image file containing a photo of the author hosted at the oreilly.com domain.
  • twitter – The author’s Twitter username.
  • works – A list of the ids of items that the author has created.
  • expertise – A list of the expertise tags associated with the author.
  • biography – The author’s biography as text/html.

Examples of Fluidinfo O’Reilly API queries

Monday, March 21st, 2011

This post is all about querying the O’Reilly book and author information recently imported into Fluidinfo. If you want the skinny on Fluidinfo’s query language in glorious in-depth techno-geek-speak then check out the documentation. If you’d rather see some real world examples, read on…

In Fluidinfo, objects represent things (and all objects have a unique id). Information is added to objects using tags. Tags can have values, and tag names are organized into namespaces that give them context. Permissions control who can see and use namespaces and tags.

Objects do not belong to anyone and don’t have permissions associated with them. They’re openly writable. Anyone can tag anything to any object. Many objects have a special globally unique “about” tag value that indicates what they are about. Interaction with Fluidinfo is via a REST API.

That’s Fluidinfo in a nutshell.

In another article published today I describe the Fluidinfo tags and namespaces used to annotate objects with O’Reilly data. The tags are attached to objects for O’Reilly books and authors. Both kinds of objects have about tags. So a trivial first kind of query is to go directly to an object that’s about a book. For example, to get information about the object representing the book “Open Government” visit the URL http://fluiddb.fluidinfo.com/about/book:open government (daniel lathrop; laurel ruma).

You’ll get back a JSON response containing a list of all the tags (that you have permission to read) attached to that object and the object’s globally unique id. Similarly, you can go directly to the object for an O’Reilly author http://fluiddb.fluidinfo.com/about/author:tim oreilly.

In case you’re wondering about the format of these book and author about tags, we used the abouttag library written by Nicholas Radcliffe to generate them. They’re designed to be readable, easy to generate programmatically, and unlikely to result in collisions. You don’t have to remember them though, as there are many other ways to get at objects, via querying, as we’re about to see.

Queries on tags and their values

Below are some examples of using Fluidinfo’s query language.

Presence

Return all the objects that have an O’Reilly title:

has oreilly.com/title

You can see the results at the following URL: http://fluiddb.fluidinfo.com/objects?query=has oreilly.com/title. Once again, the result is in JSON. It simply contains a list of the ids of matching objects (representing things that O’Reilly have tagged with a title).

That’s the equivalent of the following SQL statement:

SELECT id FROM oreilly.com WHERE title IS NOT NULL;

Caveat: There are no tables in Fluidinfo so it’s impossible to make a direct translation to SQL. This example and those that follow simply illustrate a conceptual equivalence to make it easier for those of you familiar with SQL to get your heads around the Fluidinfo query language.

Comparison

Return all the O’Reilly objects whose price is less than $40 (the price is stored in cents).

oreilly.com/price-us < 4000

Here it is as a URL: http://fluiddb.fluidinfo.com/objects?query=oreilly.com/price-us < 4000

In SQL it would be:

SELECT id FROM oreilly.com WHERE price-us < 4000;

Text Matching

Return all the O’Reilly objects that have “Python” in the title.

oreilly.com/title matches "Python"

The resulting URL: https://fluiddb.fluidinfo.com/objects?query=oreilly.com/title matches “Python”

In SQL:

SELECT id FROM oreilly.com WHERE title LIKE '%Python%';

Set Contents

Return all the O’Reilly objects representing authors who were involved in writing the work with ISBN “9781565923607” (which is the unique ID O’Reilly use in their catalog). The value of oreilly.com/authors/works tags is always a set of unique ISBN numbers like this: ["9781565923607", "9781565563728", "9781627397284"].

oreilly.com/authors/works contains "9781565923607"

The resulting URL: http://fluiddb.fluidinfo.com/objects?query=oreilly.com/authors/works contains “9781565923607”

In SQL:

SELECT id FROM oreilly.com/authors WHERE '9781565923607' IN (SELECT works FROM oreilly.com/authors);

(Actually, the similar “IN” operation in SQL isn’t a very good example since it results in verbose monstrosities like the above.)

Exclusion

Return all the O’Reilly books that were published in 2001 except those published in April.

oreilly.com/publication-year=2010 except oreilly.com/publication-month=4

The resulting URL: https://fluiddb.fluidinfo.com/objects?query=oreilly.com/publication-year=2010 except oreilly.com/publication-month=4

In SQL:

SELECT id FROM oreilly.com WHERE year=2010 AND month<>4;

Logic

It’s possible to use the and and or logical operations. For example, return all the O’Reilly books whose title matches “Python” and were published before 2005:

oreilly.com/title matches "Python" and oreilly.com/publication-year < 2005

The resulting URL: http://fluiddb.fluidinfo.com/objects?query=oreilly.com/title matches “Python” and oreilly.com/publication-year < 2005

In SQL:

SELECT id FROM oreilly.com WHERE title LIKE '%Python%' AND year < 2005

Grouping

Return all the objects representing O’Reilly books mentioning “Python” in their title that were published in either 2008 or 2010.

oreilly.com/title matches "Python" and (oreilly.com/publication-year=2008 or oreilly.com/publication-year=2010)

The resulting URL: http://fluiddb.fluidinfo.com/objects?query=oreilly.com/title matches “Python” and (oreilly.com/publication-year=2008 or oreilly.com/publication-year=2010)

In SQL:

SELECT id FROM oreilly.com WHERE title LIKE '%Python%' AND (year = 2008 OR year = 2010);

Querying across different data sets

Fluidinfo can query seamlessly across tags from different sources that are stored on the same object. E.g., return the titles of all O’Reilly books that Terry Jones owns.

has oreilly.com/title and has terrycojones/owns

The resulting URL: http://fluiddb.fluidinfo.com/objects?query=has oreilly.com/title and has terrycojones/owns

In SQL:

Well, it’s actually not clear how you’d do this in SQL. Presumably there’d need to be some kind of table join, supposing that were possible!

Getting back tags on objects matching a query

It’s also possible to indicate which tag values to return for each matching object. This is done by using the Fluidinfo /values HTTP endpoint and specifying the tag values to return as arguments in the URL path. For example, if I wanted the title, author names and publication year of all the O’Reilly books with the word “Python” in the title published before 2006 then I’d use the following query:

oreilly.com/title matches "Python" and oreilly.com/publication-year < 2006

and append the wanted tags to the URL after the query (in any order):

&tag=oreilly.com/title&tag=oreilly.com/author-names&tag=oreilly.com/publication-year

The resulting URL: http://fluiddb.fluidinfo.com/objects?query=oreilly.com/title matches “Python” and oreilly.com/publication-year < 2006&tag=oreilly.com/title&tag=oreilly.com/author-names&tag=oreilly.com/publication-year

This is similar to the following SQL:

SELECT title, authors, year FROM oreilly.com WHERE title LIKE '%Python%' AND year < 2006;

Fluidinfo returns a JSON object like this:

{u'results': {u'id': {u'1a91e021-7bce-4693-bfa5-0dc437fe1817':
    {u'oreilly.com/author-names': {u'value': [u'Anna Ravenscroft', u'David Ascher', u'Alex Martelli']},
     u'oreilly.com/publication-year': {u'value': 2005},
     u'oreilly.com/title': {u'value': u'Python Cookbook, Second Edition'}},
u'1d25baae-b977-4ff4-bb77-01c52bd1d339':
    {u'oreilly.com/author-names': {u'value': [u'Fredrik Lundh']},
     u'oreilly.com/publication-year': {u'value': 2001},
     u'oreilly.com/title': {u'value': u'Python Standard Library'}},
u'3360f05f-9bf4-4da5-abc0-0e3742809b98':
    {u'oreilly.com/author-names': {u'value': [u'Fred L. Drake Jr', u'Christopher A. Jones']},
     u'oreilly.com/publication-year': {u'value': 2001},
     u'oreilly.com/title': {u'value': u'Python &amp; XML'}},
u'9845b184-ef1b-46fb-8e7c-011da053dcb6':
    {u'oreilly.com/author-names': {u'value': [u'Andy Robinson', u'Mark Hammond']},
     u'oreilly.com/publication-year': {u'value': 2000},
     u'oreilly.com/title': {u'value': u'Python Programming On Win32'}}}}}

It’s also possible to update and delete tag values from matching objects. This process is explained in detail in the Fluidinfo documentation and this blog post.

Finally, rather than interacting with Fluidinfo directly using the raw HTTP API it’s a good idea to use one of the client libraries listed here. For example, using the fluidinfo.py library the last example query can be executed as follows:

>>> import fluidinfo
>>> import pprint
>>> headers, result = fluidinfo.call('GET', '/values', tags=['oreilly.com/title', 'oreilly.com/author-names', 'oreilly.com/publication-year'], query='oreilly.com/title matches "Python" and oreilly.com/publication-year < 2006')
>>> pprint.pprint(headers)
{'cache-control': 'no-cache',
 'connection': 'keep-alive',
 'content-length': '937',
 'content-location': 'https://fluiddb.fluidinfo.com/values?query=oreilly.com%2Ftitle+matches+%22Python%22+and+oreilly.com%2Fpublication-year+%3C+2006&tag=oreilly.com%2Ftitle&tag=oreilly.com%2Fauthor-names&tag=oreilly.com%2Fpublication-year',
 'content-type': 'application/json',
 'date': 'Thu, 10 Mar 2011 15:17:58 GMT',
 'server': 'nginx/0.7.65',
 'status': '200'}
>>> pprint.pprint(result)
{u'results': {u'id': {u'1a91e021-7bce-4693-bfa5-0dc437fe1817': {u'oreilly.com/author-names': {u'value': [u'Anna Ravenscroft',
... etc ...
 

Learn more

Hopefully, this has explained enough to get you started. If you don’t have a Fluidinfo account, you can sign up here. If you have any questions, please don’t hesitate to get involved with the Fluidinfo community, contact us directly or join us on IRC. We’ll be more than happy to help!

O’Reilly Fluidinfo Chrome extension

Monday, March 21st, 2011

To help people get going with the API competition announced today on the O’Reilly Radar site, Emanuel Carnevale has written a cool extension for Google’s Chrome browser. The extension shows some of the non-O’Reilly tags on the book objects and also lets you indicate which O’Reilly books you own. It does this by putting tags onto the objects representing O’Reilly books in Fluidinfo.

To install the extension onto your Chrome browser click on the following link (from within Chrome): https://fluiddb.fluidinfo.com/about/oreilly.com/fluidinfo/chrome-extension.crx. Your browser will guide you through what to do. It’s pretty obvious stuff. Once it’s installed you’ll see a new icon in the top right hand corner of the browser window between the address bar and the little spanner icon:

Click the icon and sign in with your Fluidinfo credentials. If you don’t yet have an account on Fluidinfo you can sign up here.

How do you use it..?

Simple. Go visit the O’Reilly catalog and click on one of the books you own. For example, I happen to be the proud owner of Natural Language Processing with Python. If you visit the page for the book you’ll notice a new small Fluidinfo icon in the book details:


Click the icon and you’ll see a pop-up like this:

You can click on the appropriate statement at bottom to indicate ownership or not, as the case may be.

The writable API gives us all a voice

The extension uses an “owns” tag in your top-level Fluidinfo namespace to indicate book ownership on the objects in Fluidinfo. For example, my tag is called “ntoll/owns”. The extension attaches this tag to the object representing the O’Reilly book whose page you are visiting.

Because the extension tags the exact same Fluidinfo objects that have the O’Reilly information, I can start to do some really cool searches. For example, I happen to know Terry has a particularly large O’Reilly “zoo” as do I (in fact, doesn’t every developer..?). We can see what books we both own about Python with the following query:

oreilly.com/title matches "Python" and has terrycojones/owns and has ntoll/owns

The following code snippet for running this query uses the fluidinfo.py client library from within the Python shell. Alternatively, you can see the result directly if you visit this URL.

>>> import fluidinfo
>>> import pprint
>>> headers, result = fluidinfo.call('GET', '/values', tags=['oreilly.com/title',], query='oreilly.com/title matches "Python" and has terrycojones/owns and has ntoll/owns')
>>> pprint.pprint(result)
{u'results': {u'id': {
                      u'01371c03-9097-4267-a137-ae88a23790ef': {u'oreilly.com/title': {u'value': u'Python Pocket Reference, Fourth Edition'}},
                      u'4e9c42b6-68cb-43f5-9b75-60af9c0bd5a7': {u'oreilly.com/title': {u'value': u'Programming Python, Fourth Edition'}},
                      u'cd0838db-96ae-42ae-98c9-248a1507e2bb': {u'oreilly.com/title': {u'value': u'Python in a Nutshell, Second Edition'}}}}}

This illustrates how anyone can add tags to the objects being used by O’Reilly, and can then search based on their own additions and those of others. That’s why we say that Fluidinfo provides writable APIs. Cool :-)

Run with it!

There’s obviously a lot more that could be done with this extension. We kept it simple mainly because we wanted to give an example of how such an extension could be written. We hope it can provide a basis for your own efforts, especially if you’re entering the O’Reilly API competition. Emanuel has released the source code for the extension so you can grab it from Github and take it from there!

Indicating (shared) interest in things without disclosing what they are

Saturday, March 5th, 2011

Imagine you want wanted to tell the world you were interested in something, for example an email address or a phone number, without telling the world what that thing was. That may not sound so interesting, but if several people were doing the same thing, it would be a mechanism for discovery of private things you had in common, without telling anyone else what those things were.

Russell Manley and I just thought of a simple way to do this using Fluidinfo. Here’s how we did it for the email addresses we know.

For each email address, compute its MD5 sum. Then, put a rustlem/knows or terrycojones/knows tag onto the object whose fluiddb/about value is the MD5 sum. The MD5 algorithm is essentially one-way, so even if someone finds a Fluidinfo object with either of our tags on it (which is trivial) they cannot recover the original email address.

This is pretty nice. We’re independently indicating things of interest, but neither of us is publicly saying what those things are. Because we’re putting our information onto the same objects in Fluidinfo, we can then easily discover things we have in common with each other (and with others), without the world knowing what. We can do the same thing for phone numbers, or anything else.

Getting the data into Fluidinfo was trivial. Here’s code I used to put a terrycojones/knows tag (with value True) onto the appropriate objects:

import sys, hashlib
from fom.session import Fluid

fdb = Fluid()
fdb.login('terrycojones', 'PASSWORD')

for thing in sys.stdin.readlines():
    about = hashlib.md5(thing[:-1]).hexdigest()
    fdb.about[about]['terrycojones/knows'].put(True)
 

You pass a list of email addresses to this script on standard input.

Russell and I each had about a thousand email addresses in our address books. A first question is how many addresses we know in common. You can get the answer to this with the simple Fluidinfo query has terrycojones/knows and has rustlem/knows. It turns out there are 53 common addresses. But the results don’t tell us which addresses those are, which is also interesting.

We also wrote a small script to print any tags ending in /knows for a set of email addresses given on the command line.

import sys, hashlib
from fom.session import Fluid
from fom.errors import Fluid404Error
fdb = Fluid()

for thing in sys.argv[1:]:
    about = hashlib.md5(thing).hexdigest()
    print thing, about
    try:
        for tag in fdb.about[about].get().value['tagPaths']:
            if tag.endswith('/knows'):
                print '\t', tag
    except Fluid404Error:
        print '\tunknown'
 

So given an email address, we can run the above and see who else knows (or claims to) that email address.

We find all this quite thought provoking. Without going into details of the social side of this, it’s worth pointing out that Fluidinfo makes this kind of information sharing very easy because it has a guaranteed writable object for everything, including all MD5 sums. Because the fluiddb/about tag is unique and isn’t owned by anyone, any user can add their knows tag to the object for any MD5 sum. The ability for users and applications to work independently and yet to share information by just following a fluiddb/about convention is one of the coolest things about Fluidinfo.

Finally, note that this system does not guarantee privacy. If someone already knows an email address or phone number (etc) they can compute its MD5 sum and examine the Fluidinfo tags on the corresponding object. Doing so they might see a rustlem/knows tag and would then be free to draw their own conclusion.

You can play too. All you need is a Fluidinfo account and the above code. Please let us know how you get on. For example, you can freely tweet any MD5 sums we have in common. We’re going to use the hashtag #incommon, like this.

How to make an API in Fluidinfo

Wednesday, February 23rd, 2011

It’s very simple really:

1. Register a domain/user on Fluidinfo

Start here. If you’re registering a domain name then we will require proof of ownership (the instructions explaining how to do this are very simple).

2. Create your namespaces and tags

Be careful, you’re choosing how your data will be structured in Fluidinfo. Some tips we’ve found useful:

  • Flat is good.
  • Use namespaces to differentiate between the different sorts of things you’ll be tagging (e.g. between books and authors).
  • Copy conventions (how do others organise their data?).
  • KISS! Keep it simple (stupid!).

3. Import your data into Fluidinfo

To help you, we have a Python based script/library called FLIMP. Nevertheless, there are lots of freely available libraries that you may want to adapt yourself.

4. Announce your new API

Programmers will interact with your data via the general Fluidinfo API, which is simple and well documented. All you need to do is tell the world that your data is available, and what namespaces and tags you’re using to store it in Fluidinfo.

That’s it.

Please feel free to get in touch at any time if you have any questions or would like to explore the possibility of Fluidinfo Inc. helping you to add your data to Fluidinfo.

How I made a writable API for Union Square Ventures in an hour

Tuesday, February 15th, 2011

Image: Eric Archivell

I was mailing Fred Wilson and Albert Wenger of Union Square Ventures late last year, talking about Fred’s article Giving every person a voice. Fred said

I hadn’t really thought that we are all about shrinking the minimal viable publishing object, but that may well be true in hindsight.

I wanted to illustrate Fluidinfo as doing both: providing a minimal viable way to publish data (with an API), and also giving everyone a voice. So I decided to build Union Square Ventures a minimal API, and to then add my voice. In an hour.

A minimal viable API for USV

USV currently has 30 investments. If you want to get a list of the 30 company URLs, how would you do it? A non-programmer would have no choice but to go to the USV portfolio page, and click on each company in turn, then right-click on the link to each company’s home page and copy the link address, and then add that URL to your list. That process is boring and error prone.

If you’re a programmer though, you’d find this ridiculously manual. You’d much rather do that in one command, for example if you’re collecting information on VC company portfolios, perhaps for research or to get funded. Or if you were building an application, perhaps to do what Jason Calacanis is doing as part of the collecting who’s funding whom on Twitter and Facebook. You want your application to be able to fetch the list of USV company URLs in one simple call.

So I made a unionsquareventures.com user in Fluidinfo (sign up here), did the repetitive but one-time work of getting their portfolio companies’ URLs out of their HTML (so you wouldn’t have to), and added it to Fluidinfo. I put a unionsquareventures.com/portfolio tag onto the Fluidinfo object about each of those URLs. In other words, because Fluidinfo has an object for everything (including all URLs), I asked it to tag that object.

That was just 7 lines of code using the elegant and simple Python FOM library for Fluidinfo written by Ali Afshar:

import sys
from fom.session import Fluid

fdb = Fluid()
fdb.login('unionsquareventures.com', 'password')
urls = [i[:-1] for i in sys.stdin.readlines()] # Read portfolio URLs from stdin

for url in urls:
    fdb.about[url]['unionsquareventures.com/portfolio'].put(True)
 

As a result, using the jsongrep script I wrote to get neater output from JSON, I can now use curl and the Fluidinfo /values method to get the list of USV portfolio companies in the blink of an eye:

curl ‘http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio&tag=fluiddb/about’ |
jsongrep.py results . . fluiddb/about value | sort
u’http://amee.cc’
u’http://getglue.com’
u’http://stackoverflow.com’
u’http://tumblr.com’
u’http://www.10gen.com’
u’http://www.boxee.tv’
u’http://www.buglabs.net’
u’http://www.clickable.com’
u’http://www.cv.im’
u’http://www.disqus.com’
u’http://www.edmodo.com’
u’http://www.etsy.com’
u’http://www.flurry.com’
u’http://www.foursquare.com’
u’http://www.hashable.com’
u’http://www.heyzap.com’
u’http://www.indeed.com’
u’http://www.meetup.com’
u’http://www.oddcast.com’
u’http://www.outside.in’
u’http://www.returnpath.net’
u’http://www.shapeways.com’
u’http://www.simulmedia.com’
u’http://www.soundcloud.com’
u’http://www.targetspot.com’
u’http://www.twilio.com’
u’http://www.twitter.com’
u’http://www.workmarket.com’
u’http://www.zemanta.com’
u’http://zynga.com’

There you have it, a sorted list of all Union Square Ventures portfolio companies’ URLs, from the command line. I can do it, you can do it, and any application can do it.

The jsongrep.py program can also be used to pull out selective pieces of the output. For example, which of the companies have “ee” in their URL?

curl ‘http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio&tag=fluiddb/about’ |
jsongrep.py results . . fluiddb/about value ‘.*ee’ | sort
u’http://www.meetup.com’
u’http://amee.cc’
u’http://www.indeed.com’
u’http://www.boxee.tv’

So maybe, in order to be funded by USV, it helps to have “ee” in your URL? :-)

What about USV companies that don’t have “.com” URLs?

curl ‘http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio&tag=fluiddb/about’ |
jsongrep.py results . . fluiddb/about value ‘.*(?<!\.com)$’
u’http://www.outside.in’
u’http://amee.cc’
u’http://www.cv.im’
u’http://www.buglabs.net’
u’http://www.returnpath.net’
u’http://www.boxee.tv’

OK, these things are geeky, but that’s part of the point of an API: to enable applications to do things. We’ve made the portfolio available programmatically, and you can immediately see how to do fun things with it that you couldn’t easily do before. In fact, it’s quite a bit more interesting than that. As a result of doing this work, I can tell you that there was a company listed a couple of months ago on the portfolio page that is no longer there. And there’s a company that’s been invested in that’s not yet listed. That’s a different subject, but it does illustrate the power of doing things programmatically.

This is a minimal viable API for USV because there’s only one piece of information being made available (so far). But an API it is, and it’s already useful.

It’s also writable.

Giving everyone a voice

In a sense we’ve just seen that everyone has a voice. USV put a tag onto the Fluidinfo objects that correspond to the URLs of their portfolio companies and they didn’t have to ask permission to do so.

But what about me? I’m a person too. I’ve met the founders of some of those companies, so I’m going to put a terrycojones/met-a-founder-of tag onto the same objects. Fluidinfo lets me do that because its objects don’t have owners, its permission system is instead based at the level of the tags on the objects.

So I wrote another 7 line program, like the one above, and added those tags. I also added another USV tag, called unionsquareventures.com/company-name. Let’s pull back just the names of the companies whose founders I’ve met:

curl ‘http://fluiddb.fluidinfo.com/values?query=has%20unionsquareventures.com/portfolio%20and%20has%20terrycojones/met-a-founder-of&tag=unionsquareventures.com’/company-name |
jsongrep.py results . . . value | sort
u’Bug Labs’
u’Foursquare’
u’GetGlue’
u’Meetup’
u’Shapeways’
u’Stack Overflow’
u’Tumblr’
u’Twitter’
u’Zemanta’

Isn’t that cool? I do indeed have a voice!

You have one too. If you sign up for a Fluidinfo account you can add your own tags and values to anything in Fludinfo. And you can use Fluidinfo, just as I’ve illustrated above, to make your own writable API. See also: our post from yesterday, What is a writable API?

Mining the BoingBoing API

Wednesday, February 9th, 2011

With all the BoingBoing data from the past ten years now in Fluidinfo the next question is “what can we do with it..?”. That’s what I’ll be answering in this technical how-to, so expect lots of code / examples!

I’ve organised the article into four parts:

  1. Basic Fluidinfo concepts
  2. How BoingBoing data is organised
  3. Minecraft (example data mining interactions with the API)
  4. Super-duper cool stuff (this is the best bit!)

Basic Fluidinfo Concepts

Understanding Fluidinfo involves four simple concepts:

  1. Objects represent things.
  2. Tags define objects’ attributes.
  3. Namespaces organise tags.
  4. Permissions apply to namespaces and tags.

How does this all fit together..? Objects are simply tagged with data. Put another way, tags associate a value with an object.

The other important concept to make clear is that nobody owns objects, there are no permissions associated with objects and objects last for ever. Although every object has a unique ID they are also usually identified by a globally unique and immutable “about” tag value. It’s used as you’d expect: to indicate what the object is supposed to be about. Finally, anyone can add data to any object (more on this later).

(er… that’s really all it is.)

Of course, since Fluidinfo is a data-store it is possible to do searches, link objects and store all sorts of different types of data (from primitive types like numbers, booleans and text to more opaque values such as images, video, sound and other binary data).

Oh yeah, interaction with the data is via a simple yet powerful REST API. There are plenty of client libraries in many different languages which allow you to work without worrying about the dirty implementation details.

How the BoingBoing data is organised in Fluidinfo

Each of the 64,000 BoingBoing articles is represented by a corresponding Fluidinfo object whose about tag value is the URL of the original post on boingboing.net. In the original XML dump, each post looked something like this:

   <row>
        <permalink>http://boingboing.net/2000/01/21/street-tech-reviews-.html</permalink>
        <created_on>2000-01-21 14:07:38</created_on>
        <basename>street_tech_reviews_</basename>
        <author>Mark Frauenfelder</author>
        <title>Street Tech Reviews and news</title>
        <body><A HREF="http://www.streettech.com/">Street Tech</A> Reviews and news for gadget-lovers and propeller heads of all stripes.</body>
        <body_more>NULL</body_more>
        <comment_count>0</comment_count>
        <categories>NULL</categories>
    </row>

I’ve done the simplest thing possible: created a top-level boingboing.net namespace in Fluidinfo under which all tags used to annotate BoingBoing data are defined. I’ve added tags to this namespace that map to the original XML elements: permalink, created_on, basename, author, title, body, body_more, comment_count and categories. The Fluidinfo objects representing BoingBoing posts have data associated with them using these tags. For example, the object representing the post described in the XML example above has a boingboing.net/title tag with the associated value: “Street Tech Reviews and news”.

Since I was also cleaning the raw XML I decided to extract / re-structure some of the data. This resulted in some additional tags: year, month, day, timestamp, links and domains. The function of the date related tags should be clear. The links and domains tags are interesting because I scraped all the anchor tags in the body and body_more fields and processed the href values. Obviously the links tag references a list of all the URLs referenced in an article and the domains tag references a related list containing just the domain names.

I did one final enhancement to the data dump. I extracted all the authors and categories and turned them into tags. When I imported the data I used these tags in the “delicious” way of tagging: simply by having such a tag (with no associated value) an object is associated with an author or category.

Here’s what an object representing a BoingBoing article looks like:

An object

Another interesting view on the data is to explore the BoingBoing tags and namespaces in the Fluidinfo Explorer (see the screen-shot on the right). In the Explorer, if you right-click on a tag and select “Open Object” you’ll see the object that represents the tag in the main area of the application. This object is itself tagged with useful information – such as a description (containing copyright information). Yeah, I know, it sounds odd but this makes meta-tagging possible.

In addition to creating Fluidinfo objects for all the BoingBoing articles I also created an object for every domain referenced by BoingBoing throughout the last ten years.

The about tag value for these domain objects is the domain name itself. For example, there is an object about the “bbc.co.uk” domain.

Each of these domain objects has been tagged with a list of all the BoingBoing articles that mention them. This is, I think, rather cool. To continue the example, the bbc.co.uk domain was referenced in 177 BoingBoing articles.

Minecraft (example data mining interactions with the API)

So here comes the cool how-to stuff…

Should you need to, use the existing documentation to read about the Fluidinfo API in super-painfully-precise-techno-vision. However, I’m going to present a quick guided tour in the form of a Python session using the fluiddb.py module (remember my advice to use one of the client libs). The advantage of using fluiddb.py is that it’s a very thin layer on top of the HTTP API so you get a feel for how various things work. The other advantage is that reading Python is like reading pseudo-code and is thus a great teaching tool.

In the following example I simply import the fluiddb module and ask it for information about my user (ntoll). The basic pattern for calling Fluidinfo is: fluiddb.call(“HTTP-VERB“, “PATH IN API“, OTHER OPTIONAL ARGS)

>>> import fluiddb # loads the module into the session
>>> headers, body = fluiddb.call('GET', '/users/ntoll') # 'GET' is the HTTP verb & '/users/ntoll' is the API path
>>> headers # contains the HTTP headers returned from Fluidinfo
{'cache-control': 'no-cache',
 'connection': 'keep-alive',
 'content-length': '76',
 'content-location': 'https://fluiddb.fluidinfo.com/users/ntoll',
 'content-type': 'application/json',
 'date': 'Tue, 08 Feb 2011 19:42:10 GMT',
 'server': 'nginx/0.7.65',
 'status': '200'}
>>> body # contains the actual result, in this case basic information about the user ntoll (me)
{u'id': u'a694f2d0-428e-4aaf-85d1-58e903f56b30',
 u'name': u'Nicholas Tollervey'}

Notice how the “content-location” in the headers tells you what the full URL of the API call is (this is interesting since fluiddb.py creates this automagically for you). The body (result) is a Python dict object that basically mirrors the JSON dict object Fluidinfo served up.

The following example grabs information about a specific object. Notice that I pass in the path to the Fluidinfo resource I’m GETting as a list. This ensures that the BoingBoing URL gets correctly percent encoded.

>>> headers, body = fluiddb.call('GET', ['about', 'http://boingboing.net/2000/01/21/street-tech-reviews-.html']) # get basic information about the object about "http://boingboing.net/2000/01/21/street-tech-reviews-.html"
>>> headers
{'cache-control': 'no-cache',
 'connection': 'keep-alive',
 'content-length': '455',
 'content-location': 'https://fluiddb.fluidinfo.com/about/http%3A%2F%2Fboingboing.net%2F2000%2F01%2F21%2Fstreet-tech-reviews-.html',
 'content-type': 'application/json',
 'date': 'Tue, 08 Feb 2011 19:45:27 GMT',
 'server': 'nginx/0.7.65',
 'status': '200'}
>>> body
{u'id': u'469257cf-2c33-4628-a97e-47166bae24fa',
 u'tagPaths': [u'boingboing.net/timestamp',
               u'fluiddb/about',
               u'boingboing.net/day',
               u'boingboing.net/month',
               u'boingboing.net/year',
               u'boingboing.net/authors/markfrauenfelder',
               u'boingboing.net/comment_count',
               u'boingboing.net/author',
               u'boingboing.net/basename',
               u'boingboing.net/body',
               u'boingboing.net/domains',
               u'boingboing.net/created_on',
               u'boingboing.net/permalink',
               u'boingboing.net/title',
               u'boingboing.net/links']}
>>>

Hopefully, the result speaks for itself: it contains the unique ID of the Fluidinfo object that is about the BoingBoing URL, and a list of the tags on that object. Getting the value of a specific tag is simple:

>>> headers, body = fluiddb.call('GET', '/objects/469257cf-2c33-4628-a97e-47166bae24fa/boingboing.net/title')
>>> body
u'Street Tech Reviews and news'

I simply appended the path to the tag onto the object’s unique ID (this also works with the about tag too as used in the prior example).

Returning tag values for a set of results that match a query is also easy. The equivalent of the following SQL-esque query:

SELECT title, categories, created_on FROM boingboing.net WHERE authors="markfrauenfelder" AND year=2010;

… is:

>>> headers, body = fluiddb.call('GET', '/values', tags=['boingboing.net/title', 'boingboing.net/created_on', 'boingboing.net/categories'], query="has boingboing.net/authors/markfrauenfelder and boingboing.net/year=2010")

A call is made to the “/values” endpoint with a list of tags whose values we want returned and a query to generate the result set. The query is written in Fluidinfo’s super-simple query language. The headers of the response look like this:

>>> headers
{'cache-control': 'no-cache',
 'connection': 'keep-alive',
 'content-length': '287328',
 'content-location': 'https://fluiddb.fluidinfo.com/values?query=has+boingboing.net%2Fauthors%2Fmarkfrauenfelder+and+boingboing.net%2Fyear%3D2010&tag=boingboing.net%2Ftitle&tag=boingboing.net%2Fcreated_on&tag=boingboing.net%2Fcategories',
 'content-type': 'application/json',
 'date': 'Wed, 09 Feb 2011 10:55:50 GMT',
 'server': 'nginx/0.7.65',
 'status': '200'}

The actual results are a JSON object (of which the following is only a fragment):

{
  "results": {
    "id": {
      "f2976562-eba6-47e4-94a1-b36ffe9a2ab1": {
        "boingboing.net/created_on": {
          "value": "2010-10-14 13:14:14"
        },
        "boingboing.net/categories": {
          "value": [
            "science",
            "technology",
            "art and design",
            "design"
          ]
        },
        "boingboing.net/title": {
          "value": "TED releases iPad app today"
        }
      },
      "627ebf2e-e38d-41da-a709-16294b4ab6f2": {
        "boingboing.net/created_on": {
          "value": "2010-02-19 11:29:36"
        },
        "boingboing.net/categories": {
          "value": [
            "culture"
          ]
        },
        "boingboing.net/title": {
          "value": "Miniboss T-shirt in the Boing Boing Bazaar"
        }
      } // etc... for lots of results
    }
  }
}

Happily, fluiddb.py has converted it into the Python equivalent so we can find out some useful information and look at individual results.

>>> len(body['results']['id']) # how many results do we have..?
1214
>>> body['results']['id'].keys()[0] # what's the id of the first result..?
u'f2976562-eba6-47e4-94a1-b36ffe9a2ab1'
>>> body['results']['id']['f2976562-eba6-47e4-94a1-b36ffe9a2ab1'] # show the record for the first result...
{u'boingboing.net/categories': {u'value': [u'science',
                                           u'technology',
                                           u'art and design',
                                           u'design']},
 u'boingboing.net/created_on': {u'value': u'2010-10-14 13:14:14'},
 u'boingboing.net/title': {u'value': u'TED releases iPad app today'}}

Great! So you have all the tools you need to search and explore all the BoingBoing articles from the last ten years. That’s what a conventional data API provides.

However, Fluidinfo can do additional super-duper cool stuff..!

Super-duper cool stuff!

Fluidinfo is an openly writeable database where objects have value because they are annotated with data from different sources. That’s why anyone can tag any data to any object. Since you control who can use, read and control your namespaces and tags, you still maintain control of data and importantly create a mechanism for trust.

You can trust values annotated with tags from the boingboing.net namespace because only BoingBoing is allowed to create and edit anything under this namespace. Since BoingBoing has annotated objects with information about articles then it’s safe to assume the objects are about a BoingBoing articles.

Here’s the super-duper stuff: you can contribute data to these objects too.

How..?

I’m glad you asked… :-)

First of all you’ll need an account on Fluidinfo. Once you’ve signed up you’ll be the proud owner of a top-level namespace with the same name as your username. Before you can add data to objects you’ll need to create some tags to achieve this:

>>> fluiddb.login('ntoll', 'top-secret-password') # change as appropriate
>>> newTag = {'name': 'tuba', 'description': 'Related to Tubas in some way so it must be awesome!', 'indexed': False})
>>> headers, body = fluiddb.call('POST', '/tags/ntoll', newTag) # create new tag in /ntoll namespace
>>> headers
{'cache-control': 'no-cache',
 'connection': 'keep-alive',
 'content-length': '104',
 'content-type': 'application/json',
 'date': 'Wed, 09 Feb 2011 13:08:52 GMT',
 'location': 'https://sandbox.fluidinfo.com/tags/ntoll/tuba',
 'server': 'nginx/0.7.65',
 'status': '201'}
>>> headers, body = fluiddb.call('GET', '/tags/ntoll/tuba', returnDescription=True)
>>> body
{u'description': u'Related to Tubas in some way so it must be awesome!',
 u'id': u'b03f6937-cebf-481d-a0eb-5fd355a8a602',
 u'indexed': False}

The new tag is given a name (“tuba”), description and an indication if it should be indexed. The “201” status that Fluidinfo returned confirms that the new tag was successfully created under the “ntoll” namespace.

In case you hadn’t guessed I like tubas! I’d like others to find other tuba related objects in Fluidinfo so I’ve decided I’ll attach this newly created tag to anything tuba-related, including BoingBoing posts. As it happens Fluidinfo helps me get a bunch of these posts with a search like this:

>>> headers, body = fluiddb.call('GET', '/values', tags=['boingboing.net/title', 'fluiddb/about',], query = 'fluiddb/about matches "tuba"')
>>> body
{
  "results": {
    "id": {
      "e6c108f4-bd10-4cd3-b7d5-ad549b988c28": {
        "fluiddb/about": {
          "value": "http://boingboing.net/2006/06/22/flaming-tuba-guy-dav.html"
        },
        "boingboing.net/title": {
          "value": "Flaming Tuba guy David Silverman on NBC Tonight Show 6/23"
        }
      },
      "0c006f04-0663-48d6-9f11-4e082e75eb51": {
        "fluiddb/about": {
          "value": "http://boingboing.net/2010/11/22/tuba-skinny-old-time.html"
        },
        "boingboing.net/title": {
          "value": "Tuba Skinny: Old timey blues and jazz street act from New Orleans"
        }
      }
    }
  }
}

I’ve simply queried for matches for the word “tuba” in the fluiddb/about tag. Now that I’ve got a couple of results I can tag them like so:

>>> for tubaItem in body['results']['id']:
...     header, body = fluiddb.call('PUT', '/objects/%s/ntoll/tuba' % tubaItem, "Umpah-tastical, man!")
...     print header['status']
'204'
'204'

Yay! I’ve added some information to a couple of objects about BoingBoing articles! Let’s just confirm this by asking Fluidinfo for all the objects tagged with ntoll/tuba:

>>> headers, body = fluiddb.call('GET', '/values', tags=['fluiddb/about', 'boingboing.net/title', 'ntoll/tuba', ], query="has ntoll/tuba")
>>> body
{
  "results": {
    "id": {
      "e6c108f4-bd10-4cd3-b7d5-ad549b988c28": {
        "ntoll/tuba": {
          "value": "Umpah-tastical, man!"
        },
        "fluiddb/about": {
          "value": "http://boingboing.net/2006/06/22/flaming-tuba-guy-dav.html"
        },
        "boingboing.net/title": {
          "value": "Flaming Tuba guy David Silverman on NBC Tonight Show 6/23"
        }
      },
      "0c006f04-0663-48d6-9f11-4e082e75eb51": {
        "ntoll/tuba": {
          "value": "Umpah-tastical, man!"
        },
        "fluiddb/about": {
          "value": "http://boingboing.net/2010/11/22/tuba-skinny-old-time.html"
        },
        "boingboing.net/title": {
          "value": "Tuba Skinny: Old timey blues and jazz street act from New Orleans"
        }
      },
      "024bf1b6-348d-4839-8700-cbb30d86fb97": {
        "ntoll/tuba": {
          "value-type": "image/jpg",
          "size": 467947
        },
        "fluiddb/about": {
          "value": "CrossCountryTuba"
        }
      },  
      "a694f2d0-428e-4aaf-85d1-58e903f56b30": {
        "ntoll/tuba": {
          "value": "I play tuba!"
        },
        "fluiddb/about": {
          "value": "Object for the user named ntoll"
        }
      }
    }
  }
}

Oops, I forgot I’d already tagged a couple of non-BoingBoing objects with the ntoll/tuba tag: one whose about tag value is “CrossCountryTuba” and the other being the object that represents me in Fluidinfo.

Notice how the value for the ntoll/tuba tag on the object about “CrossCountryTuba” contains only metadata: the type of data stored by that tag on that particular object (image/jpg) and the size of the data (467947 bytes). Looks like it’s an image of some sort. Let’s get it and see:

>>> headers, body = fluiddb.call('GET', '/objects/024bf1b6-348d-4839-8700-cbb30d86fb97/ntoll/tuba')
>>> image = open('tuba.jpg', 'w')
>>> image.write(body)
>>> image.close()

And what does tuba.jpg contain..?

CrossCountryTuba

Cool! Fluidinfo stores any type of data so long as you supply the appropriate MIME type when you upload the data.

How did I get the data into Fluidinfo..?

>>> tuba = open('Desktop/tuba.jpg', 'r') # open the original image
>>> header, body = fluiddb.call('PUT', '/objects/024bf1b6-348d-4839-8700-cbb30d86fb97/ntoll/tuba', tuba.read(), mime='image/jpg') # notice how I specify the MIME type
>>> tuba.close()
>>> headers['status'] # check we got a 200 OK response
'200'
>>> header, body = fluiddb.call('PUT', '/objects/024bf1b6-348d-4839-8700-cbb30d86fb97/ntoll/attribution', 'Tuba photo source: http://www.flickr.com/photos/dust/3813581130 licensed under a CC-BY 2.0 license') # need to add attribution as per the license

Simple..!

Now we’ve covered a lot of ground, so let’s just consider where we’ve got to.

  • We have a consistent, simple and powerful API to play with.
  • We can retrieve values using a simple query language referencing data contributed from many different users.
  • We can contribute data ourselves in such a way that the data remains under our control.
  • We can put all our data in the right place. If I want to contribute something about a BoingBoing article I just tag it to the object representing the right BoingBoing article.
  • We can contribute all sorts of data be it searchable primitive values like numbers, text and booleans or opaque data such as images, audio or anything else for which you can specify a MIME type.

You’re armed with enough basic knowledge to both mine BoingBoing data and contribute to it too. In fact, if you look carefully you’ll find all sorts of interesting objects in Fluidinfo. Remember, to find out more about the API check out our technical documentation.

Dive in, have fun and we’re more than happy to answer questions.

Image credits: BoingBoing’s logo and font butchered with permission (thanks @mustardhamsters!), diagram generated by abouttag written by Nick Radcliffe and the “Cross Country Tuba image” is © 2009 Amanda M Hatfield under a Creative Commons license.

How we made an API for BoingBoing in an evening

Thursday, January 27th, 2011

Yesterday the folks over at boingboing.net posted eleven year’s worth of posts as a zipped up XML file. XML is good, but having a searchable database of posts is better. So I (ntoll) am in the process of importing all the data into Fluidinfo. :-)

When finished, every post and author in the boingboing data dump will be represented by an object in Fluidinfo and tagged with useful information. The diagram below shows a representation of what a typical object about a boingboing.net post looks like:

Tags on an object representing a boingboing.net post.

The object (the red blob with a unique ID written inside it) has several tags attached to it (named “boingboing.net/author” and “boingboing.net/comment_count” for example) with associated values (“Mark Frauenfelder” and “53” respectively).

Furthermore, while I was cleaning/preparing the data for upload I made sure to extract every domain name and URL referenced in each post and annotate the publication date as computer friendly values rather than just a human readable date.

An instant win is the ability to query data. For example, you’ll be able to search for all posts that link to techcrunch.com written in 2010 by Cory Doctorow. This is how to write the query in Fluidinfo’s super simple query language:

boingboing.net/domains contains "techcrunch.com" and
boingboing.net/year = 2010 and
boingboing.net/author = "Cory Doctorow"

The result will depend on how you make the query, but let’s assume you’re using a /values based call in Fluidinfo’s REST api and you’ve asked for each post’s title, publication date and a list of domains mentioned. You’ll get back some JSON encoded data that looks something like this:

[
  "results" : {
        "id" : {
            "05eee31e-fbd1-43cc-9500-0469707a9bc3" : {
                "boingboing.net/title" : {
                    "value" : "This is a made up title for illustrative purposes"
                },
                "boingboing.net/created_on" : {
                    "value" : "2010-08-19 13:23:41"
                },
                "boingboing.net/domains" : {
                    "value": [
                        "techcrunch.com",
                        "microsoft.com"
                    ]
                }
            },
            "0521e31e-fbd1-43cc-9500-046974569bc3" : {
               … more results …
            }
        }
    }
  }
]
 


api

Wait a minute..!?!? This is just as if boingboing.net had an API.

Actually, by importing the flat XML file into Fluidinfo they do have an API – for free! Because of Fluidinfo’s open nature anyone can now make use of boingboing’s data via a few simple and easy to construct RESTful calls to Fluidinfo.

But that’s not all..!

Fluidinfo isn’t just openly readable – it’s openly writeable too.

Huh..?

Any user of Fluidinfo can tag data to any object. For example, I control a couple of tags called “ntoll/rating” and “ntoll/comment” which I could attach to any of the objects representing boingboing.net posts. By tagging an object with associated values I’m indicating what I thought about the post.

Importantly, I know which object I want to tag because it has a special unique tag called “about” whose value is the URL to the boingboing.net post in question. Other people who want to add information about this post will know to use the same object as me because the about tag-value tells them, er, what the object is about.

This brings me to the killer point: accessing data from boingboing.net is good, but the facility to annotate, discover and re-use everyone’s data about boingboing.net posts is better. That’s why we sometimes say we’re trying to do to databases what Wikipedia did to encyclopaedias.

Users of Fluidinfo don’t have to retrieve information about boingboing.net posts by building queries using just boingboing.net tags. It’s possible to search using other people’s tags. For example, here’s how to search for posts where I’ve given it a relatively high rating and added a comment:

ntoll/rating > 6 and has ntoll/comment and
has boingboing.net/title

And users don’t have to just ask for boingboing.net related tag-values either. It’s possible to ask objects for all their tags that you have permission to see. For example, you could retrieve a matching post’s title, body, author and any comments I make about the post with the ntoll/comment tag.

I’m only scratching the surface here so I’ll follow up with another post soon with some example code and use cases. In the meantime, if you want to find out more feel free to get in touch with us. We’re more than happy to help.

If you’re a developer and want to play with the boingboing.net data you should take a read of my last post explaining how to explore Fluidinfo’s API with Python.

In case you were wondering, it really was only half an evening’s work to prepare the data and write the import script. :-)

Note: The import is currently running but should be complete later this afternoon. Not all posts will be in Fluidinfo yet (so far we have everything up to the end of September 2008).

Image credits: Diagram generated by abouttag written by Nick Radcliffe and the “API Sign” is © 2006 ulybug under a Creative Commons license.

Exploring FluidDB with fluiddb.py

Friday, January 14th, 2011

FluidDB.py is a Python module based upon work by Seo Sanghyeon. The module has been extracted, extended and unit-tests were added by me (Nicholas Tollervey).

This post leads you from signing up to FluidDB to executing commands and queries using fluiddb.py. It assumes you’re already familiar with the concepts behind FluidDB and that you’re a developer looking to experiment with the API. If you’re not familiar or need a refresher take a look at the following slides:

We’ll be using Python but don’t assume any familiarity with that language. So lets get started…

When you sign up for FluidDB the following steps happen:

In order to access the API with your new credentials you need to use basic HTTP authentication over SSL (i.e. the URI starts with “https”). Obviously, using the raw API with a browser or tools such as curl or wget isn’t practical for most people. Hence the need for fluiddb.py as a wrapper (there are lots of API wrappers for FluidDB in many different languages, check out our list of libraries on our website for more information).

Installing fluiddb.py is as simple as typing:

$ pip install -U fluiddb.py

or

$ easy_install fluiddb.py

or, if you want to install from source:

$ git clone https://github.com/ntoll/fluiddb.py.git
$ cd fluiddb.py
$ python setup.py install

Simply import fluiddb to get started. The following Python terminal session demonstrates what I mean:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fluiddb

The fluiddb.instance variable indicates which instance of FluidDB the module is using (it defaults to the main instance – the sandbox can be used for the purposes of testing/experimentation). Make use of the fluiddb.MAIN and fluiddb.SANDBOX “constants” to change instance as shown below.

>>> fluiddb.SANDBOX
'https://sandbox.fluidinfo.com'
>>> fluiddb.instance = fluiddb.SANDBOX
>>> fluiddb.MAIN
'https://fluiddb.fluidinfo.com'
>>> fluiddb.instance = fluiddb.MAIN

Use the login and logout functions to, er, login and logout (what did you expect..?):

>>> fluiddb.login('username', 'password')
>>> fluiddb.logout()

The most important function provided by the fluiddb module is call(). You must supply at least the HTTP method and path as the first two arguments to a call to the REST API. For example, the following call gets information about Terry Jones:

>>> fluiddb.call('GET', '/users/terrycojones')
({'status': '200', 'content-length': '69', 'content-location': 'https://fluiddb.fluidinfo.com/users/terrycojones', 'server': 'nginx/0.7.65', 'connection': 'keep-alive', 'cache-control': 'no-cache', 'date': 'Fri, 14 Jan 2011 15:15:51 GMT', 'content-type': 'application/json'}, {u'name': u'Terry Jones', u'id': u'05eee31e-fbd1-43cc-9500-0469707a9bc3'})

Notice how call() returns a tuple containing two items:

  • The header dictionary
  • The content of the response (if there is any)

Often it is simply better to do the following:

>>> headers, content = fluiddb.call('GET', '/users/test')

So the response headers get put into the “headers” variable and the actual content of the response is found in the “content” variable.

It is also possible to send the path as a list of path elements:

>>> headers, content = fluiddb.call('GET', ['about','yes/no','test','foo'])

Which will ensure that each element is correctly percent encoded even if it includes problem characters like slash: ‘/’ (essential for using the “about” based API).

If the API involves sending json data to FluidDB simply send the appropriate Python dict object and fluiddb.py will “jsonify” it appropriately for you:

>>> headers, content = fluiddb.call('POST', '/objects', body={'about': 'an-example'})

If the body argument isn’t a Python dictionary then you must be HTTP PUTting a tag-value on an object. In which case, it’s possible to set the mime-type of the value passed in body:

>>> headers, content = fluiddb.call('PUT', '/about/an-example/test/foo', body='<html><body>Hello, World!</body></html>', mime='text/html')

If you’re PUTting a primitive value then the fluiddb.py will automatically provide the correct mime-type for you:

>>> headers, content = fluiddb.call('PUT', '/about/an-example/test/foo', 12345)

To send URI arguments simply append them as arguments to the call() method:

>>> headers, content = fluiddb.call('GET', '/permissions/namespaces/test', action='create')

The “action = ‘create'” argument will be turned into “?action=create” appended to the end of the URL sent to FluidDB.

Furthermore, if you want to send some custom headers to FluidDB (useful for testing purposes) then supply them as a dictionary via the custom_headers argument:

>>> headers, content = fluiddb.call('GET', '/users/test', custom_headers={'Origin': 'http://foo.com'})

Finally, should you be sending a query via the /values endpoint then you can supply the list of tags whose values you want returned via the tags argument. For example, the following call will return the about-tag value and the twitter screen name of those twitter users I have met in the real world:

>>> headers, content = fluiddb.call('GET', '/values', tags=['fluiddb/about', 'twitter.com/users/screen_name'], query='has ntoll/met')

If this walkthrough isn’t enough then check out the three screencasts below. I made them a few months ago so I’m demonstrating an older version of fluiddb.py but it’s pretty much unchanged (only the implementation details described in part two have changed a little).

Using FluidDB’s RESTful API with fluiddb.py (Part 1)

Using FluidDB’s RESTful API with fluiddb.py (Part 2)

Using FluidDB’s RESTful API with fluiddb.py (Part 3)

As always, if you have any question, encounter problems or simply want to give us feedback, get in touch!

Delicious to FluidDB

Monday, December 20th, 2010

In case you’ve missed the brouhaha, Yahoo were rumoured to be shutting down the rather excellent delicious bookmark/tagging service. Since reading this post in the Washington Post and checking from the horses mouth it looks like the rumours are mistaken. Nevertheless a plethora of tools for grabbing and backing-up data from delicious have been posted “just in case”.

Since I (ntoll) have always wanted to use FluidDB (our openly writable shared database – sign up here) as a delicious clone, the rumours prompted me to quickly knock together a script to extract my bookmarks from delicious and store them in FluidDB. I’m not the only person to have thought of this: Fluidinfo advisor Nick Radcliffe described one method for achieving this aim last year. Over the weekend both Nick and I have been thinking hard about how to organise the imported data within FluidDB.

The result is a simple standalone Python script called delicious2fluid that does exactly what its name implies. The source code is hosted at Github and I’ve added it as a package on PyPI (the Python Package Index). The rest of this post explains how to use delicious2fluid then describes some of the benefits of using FluidDB (a flexible schema, simple yet powerful queries, values associated with tags etc).

There are two options for installation:

  1. Download the source code and run the installation script:
    $ git clone git://github.com/ntoll/delicious2fluid.git
    $ cd delicious2fluid
    $ python setup.py install
  2. Use PyPI with pip or easy_install:
    $ pip install -U delicious2fluid or
    $ easy_install delicious2fluid

Once installed you simply need to run the command and answer the questions:

$ delicious2fluid
Delicious username: ntoll
Delicious password:
FluidDB username: ntoll
FluidDB password:
FluidDB path (hit return to default to root namespace: ntoll)
2010-12-17 21:09:12,601 - d2f - INFO - Grabbing bookmarks from delicious
2010-12-17 21:09:29,223 - d2f - INFO - 200 OK
2010-12-17 21:09:29,492 - d2f - INFO - Creating delicious namespace in FluidDB
... etc ...

The username and password for both services are not stored in any way shape or form. As you might have guessed the script pipes a log of what it’s up to to stdout. If you do encounter any problems then the d2f.log file will contain lots of debug information (bug reports and suggestions most welcome!).

The script will ignore private bookmarks since we don’t want it to be responsible for leaking information but it will import all the tags you use even if they’re attached to private bookmarks. It’s important to note that the existence of tags in FluidDB is public since every tag has an associated object with an appropriate “about-tag” value that identifies it as an object about a specific tag (you have been warned!).

After grabbing an XML dump of your bookmarks from delicious the script creates the following tags in your root namespace (override the default location of the tags by providing a namespace path for the final question that the script asks you):

  • USERNAME/title
  • USERNAME/notes

Metadata from delicious is stored with tags created under the delicious namespace:

  • USERNAME/delicious/hash
  • USERNAME/delicious/time
  • USERNAME/delicious/meta
  • USERNAME/delicious/tag

FluidDB stores the tag names as a set of strings in the tag named USERNAME/delicious/tag. Furthermore, each tag you create in delicious will be recreated in FluidDB under your root namespace:

  • USERNAME/TAGNAME

Obviously, “USERNAME” is replaced with your username on FluidDB (i.e. your root namespace if you’ve not overridden the default location). These tags annotate objects representing bookmarks in FluidDB (one object per bookmark). The object’s about tag value is simply the URL that the bookmark references so everyone else can easily find and tag it.

For example, say I (ntoll) only used three tags (“foo”,“bar” and “baz”) then the following tags will be created in FluidDB:

  • ntoll/foo
  • ntoll/bar
  • ntoll/baz

These tags are automatically added to the correct objects to indicate how the original bookmark was tagged. Of course there is nothing to stop anyone from adding more tags and information or creating more objects to represent bookmarks that might not have originated from delicious.

I’ve succeeded in importing all my tags and bookmarks (it took a couple of hours for c2000 tags and 1800 bookmarks). If you’re interested, use the FluidDB Explorer to take a look at a user-friendly view of my delicious tags. Open the tree view on the left hand side and click on the tags to find the associated objects/bookmarks. You’ll also see the query used to generate the results (usually something along the lines of “has ntoll/delicious/tags/FOO”).

You’ll also notice that I’ve actually put all my tags in the ntoll/delicious/tags namespace and ignored the default “schema”. Why have I done this? Three reasons:

  1. It helps to indicate the origin of the data.
  2. It stops my root namespace from getting polluted with (potentially) thousands of tags.
  3. It indicates that all the tags under the “delicious” namespace are to be used just like in the delicious web-application.

But won’t that mean I’ve broken FluidDB since I’ve ignored the precedent set by Nick Radcliffe in the blog posts I mentioned earlier..?

Not at all! One of the strengths of FluidDB is that it works well across different or evolving schema. For example, I can still find interesting bookmarks with queries such as:

has njr/fluidinfo and has ntoll/delicious/tags/fluidinfo

Which leads me to an object with the id f3f80612-7015-4a61-a1ba-94087e9aa582 and fluiddb/about value of “http://paulerb.typepad.com/infosharing/2009/01/is-metadata.html” (a really great blog post, by the way). I’ve used Nick’s visualisation tool to create the following representation of the object:

If you’re eagle-eyed you’ll have spotted that I’ve also added an “ntoll/rating” tag to this object with an associated value of 10 (it’s at the bottom left hand side). This demonstrates several important aspects of FluidDB:

  • I’m not limited to using a pre-defined schema. I can annotate any object with any tag linking it to any type of data – be it a primitive (searchable) value like an integer or something more opaque like a PDF document (contrast this with delicious’s value-less tags).
  • It’s possible to ask the tag for it’s description, in which case this particular one will return “An indication of how I rate something”. Since I am the only person who could have created this tag you know “I” = ntoll.
  • Because the tag is openly readable you can use it in your queries. For example, you might want a list of all the delicious bookmarks to which I’ve tagged a high rating:
    has ntoll/delicious/description and ntoll/rating >7
    (In fact you could use any tag for which you have “read” permission no matter who created it.)

In conclusion, it’s early days for this script and whilst its original purpose as a backup for delicious’s demise is no longer relevant it has provided an opportunity to demonstrate some of the interesting ways in which the openly writable, social and evolutionary approach of FluidDB adds value to a collection of bookmarks.