Archive for August, 2009

Information. Naturally.

Friday, August 28th, 2009

Image: Mary Hodder

Image: Mary Hodder

From the Fluidinfo home page:

Humans are diverse and unpredictable. We create, share, and organize information in an infinity of ways. We’ve even built machines to process it. Yet for all their capacity and speed, using computers to work with information is often awkward and frustrating. We are allowed very little of the spontaneity that characterizes normal human behavior. Our needs must be anticipated in advance by programmers. Far too often we can look, but not touch.

Why isn’t it easier to work with information using a computer?

At Fluidinfo we believe the answer lies in information architecture. A rigid underlying platform inhibits or prevents spontaneity. A new information architecture could be the basis for a new class of applications. It could provide freedom and flexibility to all applications, and these advantages could be passed on to users.

Fluidinfo does not attempt to directly model information accumulation and use in the real world. It simply provides an information architecture that is more flexible than the ones we’re used to. It provides a fairly simple answer to the question of how we might work with information more naturally when using a computer. It does not claim to be the final word on the subject, but points out a fruitful direction for advance. And it provides a concrete implementation that can be used today.

The fruitful direction

The computational world is too read-only, and too tightly controlled. Most of the time we spend using a computer, we are either 1) strictly in read-only mode or 2) using an application that allows us to write, but only in predetermined ways. In contrast, in our normal dealings with the natural world, when using our brains, we are never in read-only mode. We are constantly processing information and adding (i.e., writing) to our mental models. I’m talking about everyday things, like noticing something and remembering that you noticed. Or seeing something you like, and being able to recall that fact later. Even in these trivial acts we are in some sense writing—laying down memories that can later be recalled, sorted amongst, shared, organized, merged, or put aside for long periods or even forever.

In thinking about this extreme fluidity, I find it illustrative to consider how we work with concepts. As I wrote in Kaleidoscope: 10 takes on Fluidinfo:

Concepts are very fluid: they don’t have owners, you don’t ask for permission to add to them, they have no formal structure or central piece of content, they can be organized in many ways, and they have no pre-defined set of qualities or attributes. Exactly the same can be said of Fluidinfo objects.

The fruitful direction—and the mission of Fluidinfo, if I may be so grandiose and dramatic—is to engineer an information architecture with a fluidity similar to that of concepts, in order to make the world more writable. The question (and the point of this posting) is how?

Objects without owners

The answer is actually very simple: Fluidinfo provides support for information objects that do not have top-level owners. These objects are comprised of tags (with values), for which there is a flexible and secure permissions model. Because the objects don’t have owners, anyone, or any application, can add tags to any object it can find. These objects have all the nice properties of concepts mentioned above.

That’s it?

While there’s a lot more to Fluidinfo than having objects with no owners, this single change is the key to the architecture and is responsible for its generality and flexibility. It’s almost embarrassingly simple.

It takes a while for the implications of this twist to sink in. I’m not going to go into the details in this post. I’ll just point out that a simple change in representation can have a surprisingly profound effect. I’ve already written about this, though without giving details of Fluidinfo.

I’m fascinated by representation and its role in problem solving. How can such a simple piece of mental jujitsu result in fundamental change? I’ll describe the consequences in later posts. There are several of them, and I think they’re important. For now though, if you’re interested, please follow the link above to see some simple examples of the power of changing an underlying representation.

Kaleidoscope: 10 takes on Fluidinfo

Tuesday, August 25th, 2009

I’ve been asked what Fluidinfo is hundreds of times. I’ve never really known how to answer because it can be looked at from many different angles. As I try to answer, I often feel like I’m holding up a large opaque crystal in front of me, turning it this way and that, until I find an angle that makes sense for this particular listener. I came slowly to the realization that there is no perfect answer, and that Fluidinfo can be many things to many different people. It’s like looking through a kaleidoscope: keep turning it until you see it in a way that’s attractive.

I’ll try to explain some of these points of view in later posts. For now, I’ll just give the flavor of some of them.

The many possible views of Fluidinfo do not mean that it is complex. It’s actually very simple. But it has a flexibility and generality that obviously make it difficult to grasp. Even when you do understand it, it’s common to be able to imagine two or three ways you might use it to solve a particular problem. I’m going to save a description of the object model for later, too.

In no particular order then, here are 10 ways of thinking about what Fluidinfo might be, or become.

1. A database with the heart of a wiki: The wiki analogy is strong in some ways: Anyone can add data (but see below), applications can collaborate, data put in a shared place is more valuable, and abstractly there are Fluidinfo objects for every purpose just like there are wiki pages for every subject – all waiting for someone to create them. But the analogy is very poor in others: Fluidinfo has a strong permissions system that can prevent others from changing or even seeing your data, there is a query language, and content is typed. So Fluidinfo has the flavor of a wiki, but when you get right down to it almost everything is quite unlike a wiki.

There is an interesting related question here. The world of encyclopedias was tightly controlled, and very few people were allowed to write – the encyclopedia was the ultimate read-only authority. Comically, it seemed, Wikipedia was the exact opposite. Yet in the space of a few years, the unthinkable had happened: Wikipedia had eclipsed even the mighty Encyclopedia Britannica. Can Fluidinfo do for applications and traditional databases what Wikipedia did for humans and traditional encyclopedias? You don’t have the rigid tables or schema, anyone can write, content can evolve, and there is no top down control.

2. A metadata store for everyone and everything: Fluidinfo has a special about tag. You can use it to ask for the object that’s about something, like a URL or an ISBN number. That gives applications an easy shared place to put things about other things – i.e., to store metadata. The metadata can be users’ customization or personalization information, ratings, opinions, whatever.

3. A store of concepts: Concepts are very fluid: they don’t have owners, you don’t ask for permission to add to them, they have no formal structure or central piece of content, they can be organized in many ways, and they have no pre-defined set of qualities or attributes. Exactly the same can be said of Fluidinfo objects.

4. A platform for mashups: When a programmer makes a mashup, combining information from different sources to create information about something, where should that new information be stored? The usual answer today is to put the new information in a database, behind a new API, to document it, to get a server, to keep it running. In effect this is just making another hoop for future programmers to jump through to make an additional mashup. In Fluidinfo you can put the new information with the old—because objects don’t have owners. And that’s where it’s probably most valuable, because it’s then immediately available to be mashed up with other data on the same object, and search can target heterogeneous (i.e., mashed up) data on the same object.

5. A way of storing social graphs: Because users each have a Fluidinfo object associated with them, it is very easy to build social graphs. For example, user Andy might put an andy/i-follow tag onto the object for user Betty. If you have a few people doing that, interesting queries are then immediately possible—both within and across social networks.

6. A new way of organizing information: When we organize things, we are creating new information. Normally we store that new information elsewhere. When you can store it on the objects that are being organized, lots of nice things happen. I have an upcoming blog post that I’m tempted to title “Multiple simultaneous non-conflicting dynamic sharable organizations.” A bit of a mouthful, but nevertheless true.

You can also build all data structures from tags in Fluidinfo. They’re slower to use than data structures in a typical program, but where you lose in speed you gain in flexibility.

7. Something that frees us from APIs/UIs: APIs and UIs are usually regarded in a positive way: they make getting to information easy for programs and people. But they also control us. We can only do what they allow and what they anticipate. Tight limits are imposed on us in getting to our own data. Fluidinfo can change this: you can own your own data, you can always add data and customize, and you can directly search on anything you like.

8. A communication system: You can look at Fluidinfo objects as places for cooperating applications to exchange information. The information could be messages, jobs and results, etc. Nicholas Radcliffe, who has understood Fluidinfo for years, today found a new pleasing angle to look at it from, as a Twitter for data.

You could easily use a Fluidinfo object as a voting box, with some nice properties (e.g., retract or change your vote, verify that someone voted without being able to read their vote). And you can do more complex things, too.

9. An evolutionary data system: Fluidinfo allows reputation, trust, and convention to evolve. Its namespaces, tags, and users all have objects, and these give natural places to accumulate fitness information. Conventions will evolve for naming and tag values, just as they do for tags and hashtags. Selection pressure will take care of fixing ambiguities exactly to the extent that it’s important and worthwhile to fix them.

10. An alist on everything: One of the oddest moments ever in trying to explain Fluidinfo came when talking to Paul Graham. After at least 10 minutes of trying to find an angle for him, he finally said “oh, I get it. It’s an alist on everything.” I smiled, breathed a sigh of relief, and said yes. Well why didn’t you say so? replied Paul. Just goes to show you can never have too much background in computer science.

Cloud storage: the real opportunity

Monday, August 24th, 2009

What’s the real opportunity in online cloud storage?

It’s not that it might be cheaper, or easier, more scalable, or more backed up. It’s not building data silos in the clouds.

The real opportunity, the place where there’s the potential to realize value that’s never before been unlocked, is in using cloud storage to share data. And the best way to share data is to put it together. Because if you put it together, you can do useful things—for example, search across it.

Google and Wikipedia both showed us the value of putting disparate but related data in the same place. Can the same thing be done, at least in part, for databases? Fluidinfo is betting the answer is yes. See my last post for more detail.

That’s the opportunity cloud storage offers. That’s what’s new and valuable. Not just doing more of the same old same old, except in the clouds.

Truly social data

Monday, August 24th, 2009

This is the first in a series of posts that will describe why we’ve built Fluidinfo and what we think it’s good for.

Fluidinfo exists, in part, to address an increasingly apparent mismatch. Humans are extremely social, almost inescapably social. I won’t go into evolutionary history, though. Many of us also use computers that are connected to the internet. Sitting in Barcelona, I can now connect to a machine on the other side of the world in milliseconds. We all can.

Put billions of intensely social humans in front of computers connected to a global network that ties them all together, and what do you get? Humans trying to be social using computers.

The mismatch—and the missing piece—is social data.

I don’t think we can get to truly social data while applications maintain tight control over their data. Even calling it their data is likely wrong, as much of the data ingested and stored by applications comes from users and might be in a sense owned by the user. But even that is wrong: how can you own information? You can only think you own it.

These days applications are increasingly open. But things are still far too locked down. The standard way to open one’s data today is to provide an API to let others get at it. But custom APIs are not the answer to truly social data. An API, like a user interface, only lets you do to information what the people controlling access have decided to allow you to do. You can only do what’s been anticipated. You often have to ask permission. You can’t add things to the data as you please.

Data will only be truly social when you can work with it in the kinds of ways we work with information in the real, non-computational, world. In the real world we don’t ask for permission to have an opinion on something, to add to the ball of information surrounding a concept. Our needs don’t have to be anticipated by programmers. We can share information as we please. For example, nobody owns the concept of Barcelona. If I want to essentially “tag” Barcelona as being hot, or noisy, or beautiful, I just do it. I can keep my opinion private, I can share it with certain others, I can hold conflicting opinions, I can organize things in multiple ways at the same time and give things many names.

Fluidinfo lets you do all of the above, and then some.

The main way in which it does this is by changing the control over information. In Fluidinfo, objects (which can correspond to anything – web pages, files, people, movies, ideas, etc) do not have an owner. Any application or user is free to add information to any object. There’s a strong and flexible permissions system—but permissions are applied at the level of the tags (with values) on objects, not at the level of the object itself.

The reason this is so different and much more social is that Fluidinfo gives applications and their users a world in which they can always contribute information. It can take us from a default read-only world to one in which we can all write. Without stopping to ask if it’s ok, and without anyone having to anticipate what we might one day want to do.

In a world of truly social data, any user will be able to customize or personalize anything. You’ll be able to say “I ate there” or “that’s cool” or “that sucks!” or “I know that person” or “I want one of them” or “I’ve read that” or “Hey mum, look at this” or …. or do pretty much whatever you want. I can think of hundreds of examples, making them up at will—you only have to think of the kinds of things you’re used to doing in the real world. Your contributions will be just as important as any other. You’ll be able to search based on your data, or any selection of your friends’ data. You’ll be able to combine your information with heterogeneous information created by others. You’ll be able to augment, organize, and selectively share information as you please.

The nonexistence of truly social data is the huge missing piece in the puzzle of today’s computer applications. It’s the pain we’re all feeling, but which we’re so used to that we don’t realize it. The problem won’t be solved at the level of the application. Truly social data will be the foundation of a new class of applications that all benefit by storing at least some of their information into a common social data architecture. The ones that don’t will be left behind. Because when you think about it, all data is social.

And you can guess the rest: That’s what Fluidinfo is all about. Stay tuned.

More libraries and proto-apps, a tracker and a sandbox

Saturday, August 22nd, 2009

sandboxIt’s been another long day with a ton of progress.

First off, several more client side libraries were published. We’re now up to seven, written in just 4 days. Today’s additions were .Net and Java. You can see the full list here. These are being produced extremely quickly by a group of people hanging out on IRC.

There are several more projects in the offing. There’s the beginnings of a PHP FluidDB browser by @paparent (sounds like Marbl.es, for those in the know). There’s a very clean and API-faithful Python library coming together from Ali Afshar (here‘s a snippet, putting blood sugar levels onto objects). Nicholas Radcliffe built out his additions to Seo Sanghyeon‘s library to make a set of command-line tools for talking to FluidDB, and then used them to import his 1500+ Delicious tags into FluidDB.

There’s a ton going on, with the #fluiddb channel on IRC. In fact there’s so much going on that we can barely keep up with it. That’s why I’m blogging this at 5am, after about 16 hours of intense FluidDB work. Ross Jones said “Not only has FluidDB got me writing open source code, it is making me blog too – unheard of :) ” and wrote of FluidDB: “I’ve also heard it called Delicious on steroids, both are pretty accurate.”

We fixed a few FluidDB issues today. We also set up a public bug tracker at http://bugs.fluidinfo.com. And we’ve just set up a sandbox instance of FluidDB at http://sandbox.fluidinfo.com for programmers to fool around in without fear of error. The sandbox is running the latest patches, and these are being confirmed as working by people on #fluiddb. We have about half a dozen more tickets we’d like to close before pushing the changes into the main instance.

We’ve also started to work on a little app of our own, which should offer some pretty nice functionality to enhance one of our favorite web sites… More on that later :-)

Finally, another €10K was sent to the Fluidinfo bank account today by a couple of friends on Twitter, on top of €7K yesterday from a neighbor here. This is really a grassroots effort, in many ways.

An amazing first 72 hours

Friday, August 21st, 2009

pencilsWe’ve had an amazing first 72 hours following the FluidDB launch early Monday evening.

Initially, we didn’t give out any passwords for real FluidDB API access. That was partly to give ourselves a chance to recover, partly because we wanted to take it slowly to see how FluidDB usage would go, and partly because we really did launch early – we have a lack of monitoring and admin tools to prove it.

On Monday night I think I was more mentally exhausted than ever before. Apart from having very little sleep, the emotional side and stress factor was high. I’d told many people we would launch on the 17th, and really didn’t want to have to postpone. On the weekend Esteve and I had found two problems, one seemingly serious, and fixed them. I built the final Amazon EC2 images on the Monday morning. Esteve had been putting in long hours to roll the 0.3 version of txAMQP. Xavi was putting the final touches on the web registration process and we were moving domains and URIs around hastily after realizing there was an overlap. It was all pretty calm, but there was a lot going on.

On Tuesday I figured I’d take it easy, just keep an eye on things, and even take the kids to the pool. But it didn’t work out exactly as planned… We found that people were getting into FluidDB, and exploring using the anonymous user (the user you become if you don’t send a username). The anon user has very limited abilities, but can do a few things. Several people had shown up in the #fluiddb IRC channel we’d made and were discussing how to program to the API, and what they had learned so far.

And we were stunned to see that within 12 hours of the release, Seo Sanghyeon in Korea had blogged about us, had written a small Python library for talking to FluidDB, and had even written glue to make a FUSE filesystem backed by FluidDB. Who is that guy? we all wondered.

Since then it’s been one great surprise after another. We’ve been having 20-35 people in #fluiddb on IRC. Four client-side libraries have been written or begun (for Common Lisp, PHP, and at least a couple for Python). The people on #fluiddb have pointed out a few shortcomings of our HTTP REST interface. They’ve run into several (small, thankfully) bugs. They’ve helped to improve the documentation in multiple ways. They’ve been teaching each other. They’re sharing code. They’ve helped clarify RFC2616 and our sometimes marginal interpretation or plain misuse. They’ve discovered and intuited most of the internal organization of FluidDB. We are now using the client libraries written by these incredible people from all over the world; they’re in Korea, South Africa, Italy, the US, Uruguay, Spain, Canada, France, the UK, and probably several other countries. It’s amazing.

There’s also been a ton of support and a lot of nice comments on Twitter and Friendfeed. Some examples: 1, 2, 3, 4, 5, 6, 7, and 8. You can search Friendfeed or search Twitter to see a whole lot more.

I couldn’t have imagined a nicer way to launch FluidDB. Hanging out in IRC with a bunch of smart, enthusiastic, and energetic hackers going over the ins and outs of FluidDB is like a dream come true. It makes me smile to think of the number of times I’ve been dubiously asked why I was so sure other programmers would like FluidDB. So if you’re wondering what FluidDB is all about and how to use it, please head over to #fluiddb on irc.freenode.net and say hi. And to everyone already there: thanks so much :-)

A “private alpha” launch

Monday, August 17th, 2009

babyWe’re launching FluidDB in a private alpha phase. In a way though, that makes no sense. if it’s launched, then how come it’s still private?

FluidDB is launched in the sense that it’s up and running, and we’ve opened up the documentation and discussion groups. Also, anyone can reserve a FluidDB username.

By a private alpha, we mean that you’ll have to get individual approval to make FluidDB API calls. To do that, please email us. In other words, we’re for now restricting the number of people who can write programs that use FluidDB.

We’re doing that because we need to watch FluidDB carefully as programmers begin to use it. We’ve taken the “release something as soon as you possibly can” approach, and so there’s a lot that we’re planning to do to help FluidDB mature.

Think of the private alpha phase as being similar to what happens when a child is born. Only close family members are invited to the hospital to have a first look. Once home, a wider group of friends starts dropping by. With care the child grows stronger and is able to cope with the world at large. FluidDB is still something like an infant. It’s very real: it’s up and it works, but it needs some monitoring and supervision before we can let it go out to play with the world.

We hope this makes sense. Thanks for your understanding and patience.

FluidDB Launches!

Monday, August 17th, 2009

Image: Mark von Minden

Image: Mark von Minden

We’re very happy to announce that FluidDB is launching into a private alpha phase today.

We’ll use this blog for a series of posts describing the ways in which FluidDB is different, what it changes, how it can be used, what it’s good for, ideas for and examples of applications, programming with FluidDB, and much more. In the meantime, there are some links below to get you going.

If you followed @FluidDB on Twitter to reserve a username, that has been created for you. We’ll send you a direct message on Twitter when your account is also enabled for API access. We wont be creating accounts for Twitter followers any longer, because you can now reserve a username directly.


Mailing lists

  • FluidDB Discuss is a mailing list for all sorts of discussion about FluidDB.
  • FluidDB Users is a mailing list for programmers who are using the FluidDB API to write applications.

Documentation

Follow us on Twitter or read our personal blogs