Archive for the ‘Essence’ Category

Marc Hedlund joins the Fluidinfo board

Wednesday, February 9th, 2011

We’re really happy to announce that Marc Hedlund has joined the Fluidinfo board!

I’ve gotten to know Marc slowly over the last 10 years. We first met very briefly when he was CEO of the Popular Power, a San Francisco start-up. Nelson Minar (Marc’s co-founder) and Derek Smith, two of my close friends who are very close to Marc, were both working there. Nelson and Derek, as well as several others including Fluidinfo investor and advisor Tim O’Reilly have sky-high opinions of Marc. Hearing regular off-the-charts superlatives about Marc over the years always kept me interested to someday know him better.

Marc was present at my first ever (abysmal!) solo VC pitch for Fluidinfo, to the ill-fated Bryce Roberts and Mark Jacobson of OATV in early 2007. During the presentation, Marc interrupted to ask if he could take a photo of my slide titled “Revenue”. I think he wanted it as an example of how not to pitch a VC. I’ve never forgotten. He snapped the pic, resumed his seat, and told me to carry on :-)

Marc has a ton of experience. He founded and led Lucas Online, the internet subsidiary of Lucasfilm, was director of engineering at Organic Online, and was also CTO at Webstorm. After Popular Power he was VP of Engineering at Sana Security, and then Entrepreneur in Residence at OATV, gaining intimate knowledge of the world of venture capital and interacting with hundreds of start-up companies. Marc then co-founded Wesabe where he was Chief Product Officer before becoming CEO. These days he’s Chief Product Office at Daylife in New York.

As you can probably imagine, we’re honored and excited to have Marc involved at Fluidinfo.

Wanted: a UI/UX virtuoso who wants to help change the world

Monday, February 7th, 2011

Image: brandon shigeta

Fluidinfo is an always-writable information layer designed to hold metadata of any type about anything. It has an information model simple enough for everyone to understand – as simple as using Post-it notes. We’ve reached the point in development where we want people as well as applications to be able to create and interact directly with information in Fluidinfo.

Our core and driving passions are based on thinking about how humans work with information: how we find, consume, create, remember, organize, and share. We’re looking for a UI/UX wizard who also thinks and cares deeply about these things, and who has experience and exquisite taste in building elegant and super low-friction interfaces to information. Someone who can tell us what Tufte got wrong, and why.

Most of our computational experiences take place in read-only environments, or ones in which we can only add information in ways that have been anticipated and approved. This read-only world makes working with information using a computer very unlike the way we work with information in the natural world. A read-only world inhibits ad hoc, creative, unanticipated uses of information. It inhibits personalization and customization and therefore effective filtering. It is wrong because it puts information ontology ahead of evolution, inhibiting the natural emergence of information communication conventions like @addressing and hashtags.

With Fluidinfo we’re trying to imagine and build a computational world in which we always have write permission. In which people are free to add information to anything – to personalize and customize, to filter and search on their information and combinations of information. A computational world in which we can collectively make information more valuable by storing it in context.

There’s a huge and challenging UX/UI component to this. We believe humans are actually very good at working with raw information. It’s applications that are confusing. Part of our challenge is to create an interface to anything (i.e., in a web of things sense) and to do so in as transparent a manner as possible.

We’ve been thinking about these ideas for years and have built Fluidinfo as a platform to support this kind of simple and always-writable information storage. Now we want to put a face on it and we’re looking for someone truly great to join us.

About Fluidinfo Inc.

Fluidinfo is an angel and VC backed start-up based in New York, which is where we want you. (Our development team is currently distributed.) We have a world-class set of investors, including: Betaworks, IA Ventures, RRE Ventures, Lerer Ventures, Chris Dixon and the Founder Collective, Esther Dyson, Tim O’Reilly, Joshua Schacter, Michael Parekh, and Andrew Rasiej.

We’ve been quietly working on architecture since our seed funding in 2010, but are now starting to show the world the kinds of things we want to enable. Just for a small taste, read about the writable API we recently built for BoingBoing in a single evening, or how to use Fluidinfo to put metadata onto tweets. To get a flavor of what Fluidinfo is aiming at, see Truly Social Data, Information. Naturally, Fluidinfo as a universal metadata engine, and Kaleidoscope: 10 takes on Fluidinfo.

We also have some exciting news coming up on Feb 14 at the O’Reilly Tools of Change conference in NY, so keep an eye out for that too.

About you

We’re not going to tell you in detail what skills you need, because you’ll hopefully know that better than we do.

As a guideline though, in terms of technology for UI, we’re mainly interested in things built using ubiquitous web tools and standards like HTML5 and Javascript. We’re less keen on Flash, and cannot stomach heavyweight proprietary UI platforms. Anything that requires a clunky download is a non-starter. We do everything with Linux and will be very happy if you know your way around that world too. Experience with mobile or desktop UI/UX will also be valuable.

The three most important things we’re looking for are, in order of importance: 1) brilliance in UX/UI thought and execution; 2) proficiency in building dynamic web content with Javascript / modern HTML etc; 3) graphic design skill or experience working with graphic design teams to convert working prototypes into beautiful products. 1 and 2 are much more important than 3.

Above all though, you need to be able to show us interactive interfaces you’ve built or designed, be passionate about UI/UX, and be able to talk convincingly and in depth about what makes things work and not work.

Hiring process

To apply, send email to jobs at fluidinfo dotcom and include:

  • An outline of why you’d like to join Fluidinfo
  • A CV
  • Pointers to your previous work
  • Names and contact details of at least two references

Hiring will involve initial telephone/skype interviews, to be followed by in-person interview(s).

Dropping FluidDB as a product name, in favor of Fluidinfo

Friday, February 4th, 2011

As of today, we’re dropping FluidDB as a product name and will just use Fluidinfo (which is of course also the name of our company). We’ve obviously put a lot of energy into the FluidDB name so it feels bad to know that some of that will be squandered, and that we might create short-term confusion with the change. But the downsides of a bad name are both real and important, and it’s time to fix the mistake. I’ve run the name change idea past dozens of people over the last two months, with almost universal approval—some of it very enthusiastic. So from now on it’s Fluidinfo, and only Fluidinfo.

Here are the main reasons for the change:

  • Having two names instead of just one was a source of confusion to many people.
  • The term “database” has too much inappropriate baggage. The mindset around databases is that they are used to carefully hoard and protect one’s own information. Fluidinfo is about combining information, about putting it in the same place, about openly writable objects with a different model of control. It’s a completely different mindset. Programmers, especially, have expectations about databases being something they download and incorporate into their application to hold just their data.
  • We were never particularly interested in the NoSQL debate (see also this O’Reilly GMT interview). Being occasionally classed as Yet Another NoSQL Database was inaccurate and led people towards apples vs oranges comparisons and confusion. For example: How does Fluidinfo compare to Hadoop? (note the two problems here – until now Fluidinfo has only been a company name, and FluidDB does not compare to Hadoop in terms of storage).

We very reluctantly began using “db” with the only purpose being to try to help VCs understand what we were building. That was before ideas of “cloud computing” and data as a service became well understood. It was helpful back then, but is not helpful now. Fittingly, after living with the baggage for a long time, the straw that broke the camel’s back was a VC who told me he almost didn’t take our meeting because he assumed from the name that we were just another of those NoSQL databases. When he said: One way or another I’m going to get you to drop that name, I knew we’d gone full circle—an inappropriate name had finally been recognized as being detrimental by the kind of person the name was supposed to be helping.

The name change can already be seen on our funky website (desperately in need of a facelift) and in the Fluidinfo documentation. We’ve also switched our Twitter account to @fluidinfo and are now using the #fluidinfo channel on IRC.

We’ll soon be moving some web pages around behind the scenes to improve their URLs, but will make sure the old links still work. If you’re interested in technical details of the API, please feel free to ask questions below, to join us in the Fluidinfo users mailing list, or to drop by the #fluidinfo channel on irc.freenode.net.

And….. stay tuned, we have some exciting Fluidinfo news right around the corner.

How we made an API for BoingBoing in an evening

Thursday, January 27th, 2011

Yesterday the folks over at boingboing.net posted eleven year’s worth of posts as a zipped up XML file. XML is good, but having a searchable database of posts is better. So I (ntoll) am in the process of importing all the data into Fluidinfo. :-)

When finished, every post and author in the boingboing data dump will be represented by an object in Fluidinfo and tagged with useful information. The diagram below shows a representation of what a typical object about a boingboing.net post looks like:

Tags on an object representing a boingboing.net post.

The object (the red blob with a unique ID written inside it) has several tags attached to it (named “boingboing.net/author” and “boingboing.net/comment_count” for example) with associated values (“Mark Frauenfelder” and “53″ respectively).

Furthermore, while I was cleaning/preparing the data for upload I made sure to extract every domain name and URL referenced in each post and annotate the publication date as computer friendly values rather than just a human readable date.

An instant win is the ability to query data. For example, you’ll be able to search for all posts that link to techcrunch.com written in 2010 by Cory Doctorow. This is how to write the query in Fluidinfo’s super simple query language:

boingboing.net/domains contains "techcrunch.com" and
boingboing.net/year = 2010 and
boingboing.net/author = "Cory Doctorow"

The result will depend on how you make the query, but let’s assume you’re using a /values based call in Fluidinfo’s REST api and you’ve asked for each post’s title, publication date and a list of domains mentioned. You’ll get back some JSON encoded data that looks something like this:

[
  "results" : {
        "id" : {
            "05eee31e-fbd1-43cc-9500-0469707a9bc3" : {
                "boingboing.net/title" : {
                    "value" : "This is a made up title for illustrative purposes"
                },
                "boingboing.net/created_on" : {
                    "value" : "2010-08-19 13:23:41"
                },
                "boingboing.net/domains" : {
                    "value": [
                        "techcrunch.com",
                        "microsoft.com"
                    ]
                }
            },
            "0521e31e-fbd1-43cc-9500-046974569bc3" : {
               … more results …
            }
        }
    }
  }
]
 


api

Wait a minute..!?!? This is just as if boingboing.net had an API.

Actually, by importing the flat XML file into Fluidinfo they do have an API – for free! Because of Fluidinfo’s open nature anyone can now make use of boingboing’s data via a few simple and easy to construct RESTful calls to Fluidinfo.

But that’s not all..!

Fluidinfo isn’t just openly readable – it’s openly writeable too.

Huh..?

Any user of Fluidinfo can tag data to any object. For example, I control a couple of tags called “ntoll/rating” and “ntoll/comment” which I could attach to any of the objects representing boingboing.net posts. By tagging an object with associated values I’m indicating what I thought about the post.

Importantly, I know which object I want to tag because it has a special unique tag called “about” whose value is the URL to the boingboing.net post in question. Other people who want to add information about this post will know to use the same object as me because the about tag-value tells them, er, what the object is about.

This brings me to the killer point: accessing data from boingboing.net is good, but the facility to annotate, discover and re-use everyone’s data about boingboing.net posts is better. That’s why we sometimes say we’re trying to do to databases what Wikipedia did to encyclopaedias.

Users of Fluidinfo don’t have to retrieve information about boingboing.net posts by building queries using just boingboing.net tags. It’s possible to search using other people’s tags. For example, here’s how to search for posts where I’ve given it a relatively high rating and added a comment:

ntoll/rating > 6 and has ntoll/comment and
has boingboing.net/title

And users don’t have to just ask for boingboing.net related tag-values either. It’s possible to ask objects for all their tags that you have permission to see. For example, you could retrieve a matching post’s title, body, author and any comments I make about the post with the ntoll/comment tag.

I’m only scratching the surface here so I’ll follow up with another post soon with some example code and use cases. In the meantime, if you want to find out more feel free to get in touch with us. We’re more than happy to help.

If you’re a developer and want to play with the boingboing.net data you should take a read of my last post explaining how to explore Fluidinfo’s API with Python.

In case you were wondering, it really was only half an evening’s work to prepare the data and write the import script. :-)

Note: The import is currently running but should be complete later this afternoon. Not all posts will be in Fluidinfo yet (so far we have everything up to the end of September 2008).

Image credits: Diagram generated by abouttag written by Nick Radcliffe and the “API Sign” is © 2006 ulybug under a Creative Commons license.

Betaworks is a Strange Attractor

Wednesday, September 22nd, 2010

I’ve just arrived back at Betaworks, who led the investment in Fluidinfo, after a 3-month absence. It’s an amazing environment, somehow buzzing with both intensity and diversity. A lot of people are trying to get an answer to the question “What is Betaworks?” The most obvious (and wrong) answer is that Betaworks is an incubator. I’ve spent time on and off thinking about the question, and I don’t think there’s a single good / conventional answer. Betaworks is something else. So here’s my informal answer, which I think works pretty well:

Betaworks is a Strange Attractor.

From that Wikipedia page: An attractor is a set towards which a dynamical system evolves over time. Of course I don’t mean Betaworks is literally a strange attractor, though their logo looks like an outward-facing version of the attractor on the right (more images). I’ll illustrate what I mean by describing who I ran into at Betaworks the last couple of times I arrived in New York and went into the office.

On June 17, 2010, I flew to NY from Vegas on a very early flight and got to Betaworks mid-morning. Just inside the door of the office I ran into Brady Forrest and Mike Loukides, both of whom work for O’Reilly. I went for coffee with Brady and spent at least an hour talking to Mike. Later in the afternoon Tim O’Reilly showed up. That’s a pretty interesting batch of folks to have run into at Betaworks. But it’s much more interesting than that. Get this: none of the three O’Reilly people knew that the others were going to Betaworks. At least one of the three didn’t know the others were even in town. Think about it. I find it quite extraordinary. NYC is a big place, and though there’s a ton of stuff going on here, there’s apparently only one place to be. Here was the first really tangible evidence for me that Betaworks have become a strange attractor of some kind, and a powerful one.

Second example: Yesterday when I arrived in the office, after several months away, I immediately ran into a bunch of people I know – none of whom works out of the Betaworks offices: Caterina Fake (Flickr, now Hunch), Jyri Engeström (Jaiku, now Google), Reshma Sohoni (Seedcamp), Iain Dodsworth (Tweetdeck), and Mika Salmi (many things). Like me, Mika also lives in Barcelona. But where do I unexpectedly run into him…? At Betaworks. How is that possible? Why were all these great people in the office today? Was there some event going on? No. So what’s going on here?

These are just a couple of days, chosen because they were both first days back in town. There are many similar days. Betaworks is an attractor in other senses too – my examples omit the companies and people who are in the office every day, creating the gravitational pull that’s attracting so many extraordinary visitors.

That’s it for now. I don’t think anyone can pin down exactly what Betaworks is, and it doesn’t really matter. But we can all be strangely attracted.

Anatomy of a funding

Monday, May 24th, 2010

Yes!! Fluidinfo is finally funded. Getting there took a while, and wasn’t easy. In fact, it was the most difficult thing I’ve ever done. Below are a few loosely related comments on how it all went down. I hope this will be of interest to people who are specifically interested in the progress of Fluidinfo, and also to other entrepreneurs. I’ve been waiting years to write it. Predictably, now that I’m in a position to do so, I don’t really have time to do a decent job.

There was a lot of rejection along the way. I blogged about expecting and embracing startup rejection in Nov 2008. We had to ignore all that, and take heart from the strong support of a few people. More on them later.

We were even called "unfundable" (and “world-changing”) in a generous article by Robert Scoble, who is a very careful listener and much smarter than people seem to realize. I was of course determined to change that.

Tim O’Reilly uttered the most memorable fund-raising related sentence along the way: “This could take over the world…. but I don’t see how we could fund you.” Argh! Best, clearest, and most useful advice came from Jeff Bezos who simply said “Never give up”. Most haunting startup article: Avoiding the Cargo Cult by Roman Stanek. Most inspiring and favorite other entrepreneur met along the way: Jason Calacanis.

Several times we could have been funded if I’d accepted very low valuations. I said no to deals that didn’t feel right. I said no, and we pushed on to a release living on fumes, when people told me Fluidinfo was just ideas – as if the years of coding counted for nothing. I even said no to a couple of informal offers to acquire Fluidinfo.

Several times we could have been funded if we’d agreed to just build some kind of application, instead of insisting on working on Fluidinfo as a general storage architecture. I was never willing to give up or even relegate the importance and value of getting the right architecture in place. I think many startups have powerful general ideas but get diverted along the way into building something more specific. They have the intention of getting back to the more general and broadly applicable vision, but they never do. There are many reasons why that happens, among them the fact that once low-level engineering (programming) decisions are baked in, they are very hard to revisit.

So you could say I wasn’t willing to compromise on the vision – to get funded earlier in order to do less. And I felt compelled to push back on the elevator pitch, a posting through which I met Andy Weissman. I don’t think entrepreneurs should obediently infantilize their grand visions just so someone who’s probably not going to fund you can perhaps understand what you’re doing. Sure, if you have a compact idea that you can package up into a tiny pitch, go for it. If after trying to do that, you don’t, go find some potential investors with the patience to listen to you. The Betaworks guys told me after we’d first met that they’d blocked off 3 hours to talk to me – they knew it might take some time to see the scope of what we were attempting, to ask about it and consider it, to let it sink in, to hear why it might work, etc. (It didn’t take 3 hours, BTW, but they were prepared.)

I spent a lot of time pondering the link between obviousness, the creation of value, and passion. I often try to get people to read that article.

I’ve been introduced to many people along the way. A couple of years ago, in order to avoid real work, I started drawing a graph of the introductions I could remember. I filled my entire 3 meter whiteboard. So I wrote a small program to generate input for Graphviz and fed it all the information. You can see the result here. The colors are for people I’ve only talked to on the phone (light green), only emailed (orange), or was introduced to but never got a reply back from (yellow). All the rest (darker green) I met in person. I had to force myself to meet people, at first just telling myself it was part of my job and that if I didn’t do it Fluidinfo would simply cease to exist at some point, and in the end actually coming to enjoy it.

If you know your way around the seed and venture world, you’ll recognize many names in that graph. You can infer that they all – one way or another – found a way to say no, with the exception of Esther Dyson (who is a saint) and the investors who have just funded us. Many never actually bothered to say no, and several simply stopped replying to email. (See also: Pond Scum.) As an entrepreneur without a lot of outside support, it’s good to have things to keep you going. Planning to have the last laugh is sometimes all it takes. :-)

Quality I most enjoyed running into while meeting all those people: intellectual generosity. One thing I learned to stop doing: taking other people’s time and attention for granted.

The total money we spent on hardware over the last 4 years is $2000. That was one laptop, a couple of sticks of RAM, a USB memory, a couple of external hard drives, and a printer. The whole of Fluidinfo was written on two 15″ laptops, with no external monitors.

I can’t stand entrepreneurial cliches. They drive me nuts. There are so many people out there pontificating about startups, funding, VCs and entrepreneurs. If you spend any amount of time in that world you’re going to hear the same old tired cliches over and over and over. Standout exceptions: Chris Dixon, Scott Rafer, Nivi and Naval at Venture Hacks, and Paul Graham. Yes, there’s a bunch of other good stuff out there too.

I made the mistake of talking to VCs too early (see the post on rejection mentioned above). I should have been talking to earlier stage funders, but I was so sure the VCs would find Fluidinfo irresistible, and I wanted a bigger amount of money, that I neglected to talk to the only people (with money) who could really appreciate what we were trying to build.

John Borthwick and Andy Weissman, the founders of Betaworks, are fantastic. Betaworks are changing early-stage financing in New York (and beyond). Look at the intro graph: John is very easy to find. Those are almost all introductions he was making long before they funded us. They invited me to speak at their monthly brown bag lunch. They’ve had us in their office for a couple of months this year, given us keys, let us come to their weekly company-wide meetings, taken me into confidence multiple times, let me be present when sensitive things were discussed, created opportunities, pulled the funding round together, been generous and accommodating on the terms of the deal, and more. Spending time with the other small startups in and around the Betaworks office has been great – there’s a lot of very smart people there, and they’re working hard (while having fun) building all sorts of things. John is very good at his job. Keep your eye on Betaworks.

And never give up :-)

TunkRank scores added to Fluidinfo

Thursday, May 6th, 2010

TunkRank is an influence measure for Twitter users. It was originally proposed by Daniel Tunkelang, and an online implementation was written and is maintained at tunkrank.com by Jason Adams. Given a Twitter user name, TunkRank computes a measure of that user’s influence based on followers of the user, how many people the user’s followers follow, etc. As well as a web interface, TunkRank scores can be obtained through their HTTP API.

Today we’re excited to announce that TunkRank scores can now also be accessed through Fluidinfo. This gives us a great way to concretely illustrate some of the unique properties of Fluidinfo. I’m going to give some examples that use Tickery, so you might want to go play with that a little first, or read our earlier postings, Meet Tickery and Tickery, for programmers.

  1. But wait, TunkRank already has an API
    That’s true, but why stop at one? The TunkRank API provides TunkRank scores in isolation. So a 3rd party application can fetch a score and use it in some way. By also putting TunkRank scores into Fluidinfo though, they become searchable and can be combined with other data in Fluidinfo. For example, here’s a Tickery query showing people I follow who have a TunkRank of over 30. Look at the Fluidinfo query: has twitter.com/friends/terrycojones and tunkrank.com/score > 30. In English: "Hey Fluidinfo, get me all objects that have a twitter.com/friends/terrycojones tag on them and that also have a TunkRank score greater than 30." That’s pretty interesting: a data mashup of Twitter friend information with TunkRank scores. You can’t (easily) do that when the two sources of data are held in isolation, each behind their own API, but it’s trivial if the information is in the same place.
  2. TunkRank’s domain name is on their data in Fluidinfo
    In the query above, the tag name for the TunkRank score is tunkrank.com/score. The tunkrank.com domain is part of the name of the data. With that name comes reputation and trust: if you see a tunkrank.com/score on a Fluidinfo object, you know it came from TunkRank. Want to put your domain name on individual pieces of data? Send us an email and if you can show us you own the domain, we’ll give you the Fluidinfo namespace with that name.
  3. TunkRank continue to control their own data
    The permissions system of Fluidinfo allows TunkRank to control their own data. If Jason Adams wants, he can take permission away from Tickery to read the TunkRank scores. Because Fluidinfo’s permission system works at the level of the tag, not at the level of the Fluidinfo object, you get fine-grained control over the pieces of information that comprise objects.
  4. Tickery was conceived independently of the idea of TunkRank scores
    Tickery doesn’t know anything about TunkRank. It happens to have an advanced tab that allows arbitrary Fluidinfo queries because that’s part of why we built it – to illustrate how other data added to Fluidinfo objects could easily be queried. Think about it – here we have two applications that were built independently of one another able to enhance each other because their information is in the same place.
  5. TunkRank did not ask for permission to add their scores
    Because Fluidinfo objects do not have owners, there was no need for TunkRank to ask for permission to put their scores onto the same Fluidinfo objects that Tickery is using. They just did it. You can too, all you need is a Fluidinfo user name (you can get one here) and then send mail to api@fluidinfo.com requesting an API password. (Fluidinfo is still in private alpha.)
  6. TunkRank and Tickery can move at different speeds
    As described in the Tickery for programmers posting, Tickery is using the Fluidinfo about tag, putting its information onto objects with a fluiddb/about whose values have the form twitter.com:uid:XXXX, where XXXX is a numeric Twitter user id. In Fluidinfo, anyone can create an object that’s "about" something. The upshot of that is that TunkRank does not have to wait until Tickery adds a user, it can just go ahead and put a tunkrank.com/score onto an object with the appropriate about tag – no questions asked. TunkRank doesn’t care or need to know if the object already existed, or whether Tickery had already put its tags there. Think about this too: here we have two (or more!) applications that are following the same convention, putting their own data into the same database, and each can move at its own speed without regard for the other. There’s no coordination beyond the about tag convention, when new data shows up from an application, it naturally goes to the place it belongs.
  7. Anyone can play – that’s the point
    Other tags can be added to the same Fluidinfo objects, and can be queried on using the Fluidinfo query language in Tickery’s advanced tab (or by using the Fluidinfo API directly). To illustrate, Esteve Fernández and I put esteve/met and terrycojones/met tags onto objects to indicate people we’ve physically met. That data can be instantly (and interestingly!) mashed up with other data. For example, here are the people Esteve is following that he has not met, but who I have met, via the query (has twitter.com/friends/esteve and has terrycojones/met) except has esteve/met. And to show Twitter friends info, TunkRank scores, and our met tags in a single query, here are the people I follow who have a high TunkRank but who I’ve not actually met.

If you’ve read all the way down to here, thanks.

If you were having trouble grasping the point of Fluidinfo before, I hope these concrete examples have helped (if not, please let us know in the comments). Considering just Twitter, TunkRank, and Tickery as three related web applications, it’s clear that when their data is in the same place – via something like a Wikipedia for data – that it becomes more valuable.

But Fluidinfo is not about Twitter – the above is just an example of three applications (four, if you count the informal esteve/met and terrycojones/met tags) that are related and which all benefit from having their data co-located. People (and applications) can do more with it – they can search it more easily, they can add to it, they can do things that the original application designers did not and could not have anticipated. This applies to any data anyone cares to add to Fluidinfo.

Finally, Fluidinfo does not have the disconcerting (to some) free-wheeling anarchy property of a wiki. While preserving the essential character of wikis: it is always writable and it has an object for everything, it has three important things that a wiki does not have and which make it suitable for use by applications: 1) a strong permissions system (e.g., only TunkRank can add tunkrank.com/score tags to objects, unless they allow others to too), 2) a query language, and 3) typed data.

Tickery, for programmers

Thursday, January 21st, 2010

Where's the beef?If you’re a programmer and you’ve played around with Tickery, it should be clear that Tickery is functionally very simple when looked at from a traditional database perspective. Tickery looks like an application in its own right. It tries to offer something so simple that anyone (at least any Twitter user) can understand and use the Simple “Enter two Twitter usernames” functionality. But we actually designed and built Tickery mainly as a demo of what’s possible with Fluidinfo (description, API). So here are some first details on how Tickery uses Fluidinfo, and how you can use it too.

Fluidinfo objects

The most important thing to understand about Fluidinfo initially is that it maintains a collection of tagged objects that are not owned. Tags have permissions and a tag on an object can optionally have a value associated with it. Here’s a conceptual view of an object that has two tags on it that were added by Tickery.

The long identifier is the object’s unique id in Fluidinfo. The left column shows tag names, such as twitter.com/users/screen_name, and the right column shows the value (if any) of the tag on the object. Because objects in Fluidinfo are not owned, anyone (which is to say any application) can put additional tags onto this object. I’m going to ignore permissions in what follows – that’s a subject for a separate posting.

Any application can find this object in Fluidinfo, using a simple query, like twitter.com/users/id = 42983 or twitter.com/users/screen_name = "terrycojones".

Now let’s suppose Tickery adds @esteve to Fluidinfo, and wants to indicate that Esteve currently follows terrycojones. Tickery creates a new tag, twitter.com/friends/esteve in Fluidinfo, and adds it to the above object. The object then looks like this:

Similarly, Tickery adds a twitter.com/friends/esteve to the objects representing all the Twitter users Esteve follows. At this point it is easy to retrieve all those users via the Fluidinfo query has twitter.com/friends/esteve (i.e., get me all the objects that have a twitter.com/friends/esteve tag, irrespective of the tag’s value, if any).

Suppose Tickery now adds another Twitter user, @fergusstothart who currently follows me. It adds another tag to the object, resulting in

and also puts a twitter.com/friends/fergusstothart tag onto the objects for the other users that Fergus follows. It finds these objects via a Fluidinfo query (using the Twitter id of Fergus, obtained from the Twitter API). If Tickery needs to tag an object for a user that it hasn’t created yet, it simply creates a new object for that user, and tags it.

Given the above, we’ve seen enough to know how Tickery does most of its work. For example, getting things like the set of people Esteve and Fergus follow in common is just an and query has twitter.com/friends/esteve and has twitter.com/friends/fergusstothart, or the set of people Esteve follows but Fergus does not has twitter.com/friends/esteve except has twitter.com/friends/fergusstothart, etc.

Where we come in

That’s all well and good, but it’s all about Tickery. What if other programmers, who perhaps don’t care or even know about Tickery want to add data and search on it too? In a normal database or with a normal application, you’d probably expect to have to ask permission. Then, supposing it was granted, you could only do the kinds of things that had been anticipated and provided for.

But in Fluidinfo it’s completely different. Any application can come along and put whatever it likes onto the above object, or any other object (that it can find). As a trivial example, Esteve and I have also added tags to the Fluidinfo objects to indicate which of the people we follow we have also met in person. Esteve has an esteve/met tag, and because we’ve met (in fact we built Fluidinfo together), he has put that tag onto the above object:

Think about what just happened. An unknown 3rd party (well, let’s pretend Esteve was unknown) just came along, sometime after the Tickery data already existed in Fluidinfo, and added something completely new and unanticipated to an existing object, without asking for permission, and without in any way disturbing the original content. Esteve, or anyone else who can read his tag, can now do interesting searches, like has twitter.com/friends/esteve except has esteve/met, which shows the people he follows but has not yet met. Further, his searches seamlessly combine the existing Tickery data with his own data, and could also include other tags that other applications add.

That kind of unanticipated use of information, flexibility in representation and search, and change of control is what’s at the heart of Fluidinfo.

Twitter lists

If you think about it, Esteve adding esteve/met tags to the objects for Twitter users is exactly like making a list in Twitter using their new lists function. But it’s more useful for two main reasons.

Firstly, you can query across lists, e.g., has terrycojones/met and has esteve/met will find people we have both met in person, and (has twitter.com/friends/esteve and has terrycojones/met) except has esteve/met will find people Esteve follows that I have met in person but whom he has not. As you can see, querying on lists makes them much more useful.

Secondly, you can use the Fluidinfo permissions system to control who can see or read your tags. So it’s not only possible to have a completely private list, a public one, but you can also have a list that’s visible just to some friends, or one that you let certain other people add to (by giving them write permission on the tag involved).

Permissions in Fluidinfo are very simple and very flexible, and because they apply at the level of the tag (not the object), you can control who can do what to individual pieces (tags) of an object. That’s a subject for another post, as I mentioned. You might like to have a read of the Fluidinfo permissions docs and/or check out Nicholas Radcliffe‘s post Permissions Worth Getting Excited About, plus see the comment by Nicholas Tollervey, who writes Fluidinfo’s "killer feature" is actually its permissions system and the implications thereof. It is so important that I’ll save that topic for its own blog post later.

More Tickery tags

Tickery also saves a few more Twitter user details onto objects in Fluidinfo. The object above has some additional tags:

You can put these into queries too, of course. The Tickery Advanced tab lets you type them in, e.g., I can see which of the people that Jack follows are very popular has twitter.com/friends/jack and twitter.com/users/followers_count > 100000.

Running on ahead of Tickery

Finally, here’s a subtle but very important point. What if you write an application that uses Fluidinfo to store data, and you want it to interact fully with Tickery, but you want to store information about a user that Tickery doesn’t know about yet?

This is crucially important because it’s about information control, and if control is completely in the hands of Tickery, other developers will be less likely to want to add information. Exactly this scenario plays out in many domains: e.g., suppose Amazon released something that let you indicate which books you own, but that you own things that are not in the Amazon database. How can you run on ahead of Amazon to insert your data before they create an object for the book, if ever? How can you do it in a way so that when they finally do create the book your data and theirs are seamlessly joined without anyone having to lift a finger or even be aware of the other party?

This is one area in which the special Fluidinfo “about” tag (full name fluiddb/about) makes all the difference. You can read about it here, and be sure to check out Nicholas Radcliffe‘s blog which is titled, not coincidentally, About Tag.

Tickery uses the Fluidinfo about tag to hold a Twitter user id, like this:

There’s a ton that could be written about this. Very briefly, the about tag is immutable and can only be set on an object when it’s created (in fact the about tag shown above was put there when the object was made). So, if you want to add data to Twitter user that Tickery hasn’t gotten to, just look up the user’s Twitter id (say XXXX) with the Twitter API, create the object in Fluidinfo with about tag twitter.com:uid:XXXX, and put your tags onto that object. If Fluidinfo doesn’t have an object with that about tag, it will make one for you. When/if Tickery gets around to adding its information for that user, it will put it in the same place. Magic.

Convenience API

As a convenience, though note that it’s optional and it’s use is up to you, Tickery provides a small API that you can use to have it put its twitter.com/users/screen_name and twitter.com/users/id tags onto objects for you and give you the Fluidinfo object id its using for a user.

E.g., if you do an HTTP GET on http://tickery.net/api/screennames/terrycojones you’ll see the object id from our examples above. Or if you happened to know my Twitter user id (42983) via the Twitter API, you could do a GET on http://tickery.net/api/uids/42983 and receive the same thing.

Truly social data

This API is just for convenience. Tickery uses the about tag in order to be able to share Fluidinfo objects with other apps – including apps that want to add information about a user that Tickery has not gotten to yet. Just like Fluidinfo, Tickery wants to encourage what we call Truly Social Data. Tickery doesn’t place itself in the center, doesn’t make its data more important than anyone else’s, and doesn’t act as a gatekeeper.

In fact, it gets even better: a normal user can turn around and stop Tickery from reading the data that Tickery stored on the user’s behalf. That’s as it should be. Users should have control over their data, and a choice of application shouldn’t result in lock in.

Getting access

Fluidinfo is still very new, and we’re in a private alpha phase. If you’d like to use the API, there are two steps: 1) reserve a username, and 2) send us email mentioning the name you reserved and a line or two about what you’d like to do. We apologize for this early restriction – please rest assured that we’re planning to open Fluidinfo up to everyone before too long. That’s the whole point.

More soon. Thanks very much for reading!

Meet Tickery

Thursday, January 21st, 2010

TickeryWe’re very excited to present Tickery: a fun tool for exploring sets of Twitter friends and finding new people to follow.

Tickery starts off very simply, letting you see who pairs of Twitter users follow in common. For example, who do Tim O’Reilly and Tim Bray follow in common? Even simple queries like this are interesting, because they’re a great way to find interesting Twitter users you may want to follow too. In addition, on the Tickery page just below where you enter the two user names, you’ll get to see whether the two users follow each other. So if you find yourself asking “I wonder if X follows Y?” you can use Tickery to find out immediately, which beats scrolling through multiple pages on twitter.com.

Please be patient with Tickery if you do a query on a user we haven’t added yet. Tickery uses the Twitter API to get information about users, and there are restrictions on how fast we can make those API calls.

Tickery lets you sign in via Twitter – see the button on the top right of the tab bar. If you sign in you can filter results to show just the people you’re following (or not), you can click to follow new people, and you can send out tweets with links to Tickery results of interest.

Tickery’s Intermediate tab offers a big jump in power. Enter simple queries using Twitter user names, and simple terms like and, or and except. For example, the query (jack and ev and biz) except terrycojones shows me Twitter users that all the three Twitter founders follow, but who I do not. Or get possible hints on which entrepreneurs are being followed by one firm and missed by another: for example who does everyone at Union Square Ventures follow that no-one at Betaworks does?

Tickery also has an Advanced tab, which gives you another big jump in power. I’ll save a description of that for another post, but to whet your appetite, here are users Tickery knows about with a Twitter id less than 1000 and people I follow on Twitter that I have also met in person. Or see the description and examples in the "huh?" button on the Advanced tab.

Powered by FluidDB

The most important and interesting thing about Tickery is that it’s built on top of FluidDB (description, API). Tickery is great fun all by itself, but it was built to show what we at Fluidinfo think the relationship between applications, their users, and their data will come to look like. That’s also the subject of an upcoming post, but here are a few bullet points to give you an idea of why applications like Tickery, written on top of FluidDB, may look normal but are in fact very different. Such applications, in combination with an information architecture like FluidDB will:

  • Leave users in control of their data, which includes letting them use other applications to work on it and, if you’re really serious, being able to turn off access to the application that stored it for you.
  • Make the world writable by default, by allowing anyone to add anything to the underlying data in any way they like.
  • Selectively protect individual aspects of data objects on a user-by-user and application-by-application basis.
  • Allow users and applications to put their name (like an internet domain name) on pieces of data, thereby stamping that data with trust and reputation.
  • Let you combine and organize your data, in isolation or with anyone else’s, via search and tagging.
  • Let other applications add more data, in a compatible and integrated way, without needing the permission or advance knowledge of the original application.
  • Explicitly allow for, and encourage, the flexible evolution of data structure conventions, similar to the way that we see evolution of tagging and hashtags.

You can read about this in the context of Tickery on the About tab.

These are the kinds of ideas that people have recently been writing about as the future of data. For example, see some of these articles: The Future: Operating System And Application-Neutral Data, We need a Wikipedia for data, Can Twitter Survive What is About to Happen to It?, Shared data services – the next frontier?, and Robert Scoble’s Twitter to turn on advertising “you will love” (here’s how: SuperTweet). While you’re at it, you might enjoy Scoble’s article The unfundable world-changing startup, which he wrote about Fluidinfo a year ago.

Stay tuned, there’s much more to come. If you’d like to find out how to write programs that can augment and use the data Tickery has stored in FluidDB, have a read of Tickery, for programmers.

Meanwhile, have fun with Tickery! Check back here, or follow me on Twitter for more news on Tickery, FluidDB, and Fluidinfo.

Putting metadata onto tweets with Fluidinfo

Tuesday, December 1st, 2009

novaVarious articles have recently discussed adding metadata to Twitter tweets – see the posts by Nova Spivack, Robert Scoble, and Dave Winer (who also suggests we need a programming language built into a Twitter client).

These are the sorts of things that Fluidinfo was designed to support, and you can do them today. If you want a password to start playing with the Fluidinfo API, send email to api at fluidinfo dot com and we’ll set you up.

In the meantime, here are some examples. I’m doing this at the iPython command line, using the Fluid Object Mapper (FOM) library, written by Ali Afshar. FOM provides a natural way to work with Fluidinfo objects, namespaces, tags, etc. But you could use any client-side software you like. The Fluidinfo API is just HTTP.

First, let’s get a connection to Fluidinfo:

from fom.session import Fluid

fdb = Fluid()
fdb.db.client.login(‘terrycojones’, ‘PASSWORD’)
fdb.bind()

That last line is a bit of internal FOM magic that makes interactive use simpler in what follows. Ignore it for now.

To put metadata onto a tweet, we’ll first ask Fluidinfo for the object that’s about a particular tweet. Let’s take the one in the image above by @novaspivack. That tweet has a URL of http://twitter.com/novaspivack/status/4999653280. We ask Fluidinfo to give us the object “about” that URL:

from fom.mapping import Object

o = Object()
o.create(‘http://twitter.com/novaspivack/status/4999653280′)
o.uid
>>> u‘ab7fa032-06df-45be-9bb2-859c18c4d342′

The argument in the o.create call is the value of the Fluidinfo about tag. If an object with that about tag already exists, Fluidinfo gives it to us. Otherwise, a new object with that about tag is created. As you can see, the object also has an identifier (o.uid). In case you’re not familiar with Python, the “u” printed in front of the id indicates that the value is a unicode string.

This is a first point of interest. We’ve just created a Fluidinfo object corresponding to an arbitrary tweet. We didn’t ask for permission, we just did it. It’s a bit like a wiki: you can ask a wiki for its page on anything, and if no such page exists, the wiki just makes you a new one. Fluidinfo does the same thing with its objects and about tag. If you want to think about Fluidinfo in all its generality, you should now consider that the about tag above could have been for any tweet, including tweets that don’t exist (or don’t yet exist), for any URL, in fact for any string. We also could have followed Nova’s suggestion and used an about value like "twitter.com/id=4999653280". But we’re getting ahead of ourselves.

Fluidinfo has a simple query language, so let’s quickly confirm that we can find this object with a search:

fdb.objects.get(‘fluiddb/about = "http://twitter.com/novaspivack/status/4999653280"’)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

The 200 is an HTTP OK status telling us the call succeeded, and you can see one object matched the search and that its id is as expected.

So how about some metadata? Let’s say I want to add a rating to the object. Here’s a bit of one-time setup. First I get my top-level namespace (which corresponds to my Fluidinfo user name). Then I create a new tag called rating in that namespace:

from fom.mapping import Namespace

ns = Namespace(‘terrycojones’)
ns.create_tag("rating", "A tag for Terry’s ratings.", False)

The False argument is telling Fluidinfo that I don’t want the tag to be indexed. Ignore that for now.

The magic of FOM lets us directly examine the tag using Python attributes. So you can get the tag and see its description like so:

rating = ns.tag(‘rating’)
rating.description
>>> u"A tag for Terry’s ratings."

At this point we have a new tag, or an abstract tag if you prefer, but we haven’t actually tagged any objects with it. So let’s tag the object we created above for Nova’s tweet:

o.set(‘terrycojones/rating’, 6)

That was pretty easy! The Fluidinfo object that’s about Nova’s tweet now has some metadata on it, a ‘terrycojones/rating’ tag, with a value of 6. Let’s make sure we can get that value back:

o.get(‘terrycojones/rating’)
>>> (6, None)

We get a 2-tuple whose second value is None when the tag’s value is a primitive Python type (in this case an integer).

Let’s do a couple of quick searches for objects with terrycojones/rating tags:

fdb.objects.get(‘terrycojones/rating = 6′)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

fdb.objects.get(‘terrycojones/rating > 4′)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

fdb.objects.get(‘has terrycojones/rating’)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

In each case just that one object is returned, as expected. Note that the last query just tests for the presence of the tag, irrespective of the tag’s value (if any).

So there you have it: arbitrary metadata on tweets, and with a query language to help find things.

But let’s press on and see how things get more interesting.

First of all, you may have noticed that I didn’t have to deal with permissions at all in the above. I was able to create the Fluidinfo object about Nova’s tweet and to tag it without asking permission. In Fluidinfo that’s always the case.

But there is a permissions system. Let’s log in as a different user and try a few things to see how it works. First, I’ll log in as njr another user whose password I happen to know:

fdb.db.client.login(‘njr’, ‘PASSWORD’)

The njr user is actually Nicholas Radcliffe who has written several great introductory articles about Fluidinfo over at About Tag.

Let’s try (as Nick) getting the terrycojones/rating tag from the object for Nova’s tweet:

o.get(‘terrycojones/rating’)
>>> (6, None)

That still works, so we can infer that the terrycojones/rating tag is readable by the njr user. Let’s log in as terrycojones again and have a look at the permissions:

fdb.db.client.login(‘terrycojones’, ‘PASSWORD’)
fdb.permissions.tag_values[‘terrycojones/rating’].get(‘read’)
>>> (200, {u‘exceptions’: [], u‘policy’: u‘open’}

We’ve asked Fluidinfo for read permissions on tag values for the tag terrycojones/rating. The result is a general policy (open), with exceptions (currently empty). Now I’ll put the njr user into the exceptions list, and confirm the result:

fdb.permissions.tag_values[‘terrycojones/rating’].put(‘read’, ‘open’, [‘njr’])
>>> (204, None)
fdb.permissions.tag_values[‘terrycojones/rating’].get(‘read’)
>>> (200, {u‘exceptions’: [u‘njr’], u‘policy’: u‘open’}

The 204 status above is just the HTTP way of telling us that the call succeeded and that the reply has no content (as expected).

Now let’s reconnect as njr and try getting the terrycojones/rating tag again:

fdb.db.client.login(‘njr’, ‘PASSWORD’)
o.get(‘terrycojones/rating’)
>>>

You can see we got nothing back. If FOM handled non-OK HTTP responses a little more carefully, you’d see that this request actually got a 401 (Permission Denied) status. Fluidinfo is now refusing to let njr read the tag.

Nick already has a rating tag, called njr/rating, so let’s go get it, make sure there’s not one already on our object, and then tag our object with it:

ns = Namespace(‘njr’)
rating = ns.tag(‘rating’)
o.get(‘njr/rating’)
o.set(‘njr/rating’, 4)
o.get(‘njr/rating’)
>>> (4, None)

Now things are getting interesting. We have tags from different users on the same object. That’s part of the point of Fluidinfo and its where the value comes from: putting information together allows you to do nice things, like query on it. After re-connecting as terrycojones, I can now do queries like this:

fdb.objects.get(‘terrycojones/rating > 5 and njr/rating > 3′)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

fdb.objects.get(‘terrycojones/rating > 5 and njr/rating < 3′)
>>> (200, {u‘ids’: []})

fdb.objects.get(‘has terrycojones/rating and njr/rating >= 4′)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

fdb.objects.get(‘has terrycojones/rating and has njr/rating’)
>>> (200, {u‘ids’: [u‘ab7fa032-06df-45be-9bb2-859c18c4d342′]})

fdb.objects.get(‘has terrycojones/rating except has njr/rating’)
>> (200, {u‘ids’: []})

There’s a lot more I could do too, like giving Nick permission to add terrycojones/rating tag to objects. By the way, Nick has written some nice articles about the Fluidinfo permissions model. See Permissions Worth Getting Excited About and The Permissions Sketch.

For a final look at metadata, let’s put something totally different onto our object:

ns = Namespace(‘terrycojones’)
page = ns.create_tag("page", "Terry’s page tag.", False)
o.set(‘terrycojones/page’, ‘<html><head><title>hi</title></head><body>Hello there!</body></html>’, ‘text/html’)

I’ve just made a new tag called terrycojones/page and tagged our object with it. What’s different here is that the value is a string, and I’m passing a MIME type with it. If I retrieve the value of the tag on the object, you’ll see the MIME type comes back too:

o.get(‘terrycojones/page’)
>>> (‘<html><head><title>hi</title></head><body>Hello there!</body></html>’,
 ‘text/html’)

and as you might hope, if you go get that tag from that object using a browser, the MIME type is returned in the HTTP Content-type header, so you end up with a real web page, with a predictable URL. Try clicking: http://fluiddb.fluidinfo.com/objects/ab7fa032-06df-45be-9bb2-859c18c4d342/terrycojones/page. We can do the same for any MIME type at all – including ones you invent for your own convenience.

So there you go. That’s metadata on tweets. With a permissions model, with a query language, with user identity, with the freedom to add anything you want, and with typed data. We don’t need a new programming language for doing this sort of thing. What we need is a better data architecture.

Fluidinfo was designed with exactly this kind of use in mind. And it’s not specific to Twitter or tweets, or anything in fact. So you can put metadata onto anything you like, search on it, continue to own/control your own data, combine it as you like, and get data in and out using a simple HTTP API.

This is all live. It’s up and running, you can do this today. I should also add that Fluidinfo is still an early alpha, and is not yet particularly fast. For more information on Fluidinfo, start with the high-level description and if you’re a programmer, read the API docs.

Next time I’ll show you how we’re putting metadata onto Twitter users, and how you can too, of course! I might also start to talk about Tickery, our upcoming Twitter query application.

If you like all this, please pass on this article. We’d love to get the word out about Fluidinfo. It’s a little difficult from Barcelona.