I think the appearance of Google App Engine and the “massively scalable Google Data Store” buzz took a lot of web app developers in the GAE direction.
GAE has a free level and I think Amazon has been forced to offer a free level due to market pressures.

The major reason for using SDB or GDS for that matter is the data model. Both of these allow dynamic attributes which are very important in modern semi-structured data.

I am working on a project to modernize technology for managing academic bibliographic items – this is an NSF funded project and the core need here is dynamic attributes because on of the core formats ‘bibtex’ is an extensible format. Also the need for a data base which is a key value store where values can be lists == keys can have multiple values.

While you can model this in a relational database using a key-value table, unifying the dynamic and static attributes in queries gets ugly fast.

Both GAE (through its Expando model) and SDB just natively, allow attributes to be added at run time and for keys to have multiple values. If your model needs this and you are not at the bleeding edge of data size then these datastores buy you a lot.

Currently GAE has some obscurely expressed limit on resource cost of a query and web request and it times out when you exceed it but you really don’t have a good idea how to tune it expect to try to do “less bad things”

SDB on the other hand tells you that if your query exceeds a certain well defined CPU usage it will time out – this suggests that if your queries are the equivalent of table scans or are above some level of computational cost you SDB will not work.

One significant disadvantage that GAE has is that the database is not exposed via REST’ish URL’s or via web services of any kind. It is up to the app you write to do that.

I have noticed that AWS continues to incrementally improve each of their web services in feature set and reduced cost as they scale up. I think the SDB data model takes getting used to and also early users of EC2 and S3 often experienced outages so SDB may be in “growing pain stage”

I am currently using GAE for my bibliographic item store and will be experimenting with SDB as well on this real world app and will keep you informed.

Nitin

The costs and benefits list is quite long and complex so I won’t get into it here, but overall I think that only a tiny fraction of applications will find it preferable to MySQL.

That’s very interesting. I’ve spent many years thinking about and building the product that Fluidinfo (my company) is now working to release. For a long time I resisted the “database” label. But I had enormous trouble getting people (mainly potential investors) to understand what we were trying to do. Finally I relented and started calling it FluidDB. I find the “database” part of the name slow and boring, and now you’ve pointed out that it might even be seriously misleading. We’ve cycled through a few names (one of the advantages of having something that takes a while to build is that you can play with different names), none of them particularly satisfying. It’s a product (initially) for programmers, so giving it a name that they can recognize seems sensible. But if they recognize it and think “that’s not a database”, it’s not good.

Hmmmm…..

Thanks,
Terry

yes and no.

Yes in terms of marketing. Amazon have completely blundered this area, and really should drop the “DB” tag as quickly as possible. It is not a DB in the sense that most developers recognize and here in lies the confusion.

We are sold the wrong product to begin with.

Instead it is a global registry; nothing more, nothing less. A registry is not a database by any stretch.

We use SimpleDB to manage all our instances running in the cloud, in a variety of different of different providers, while we power some of the largest sites in Europe. Without it, we would have to be doing a lot of extra management, and without a doubt it is a very slick and useable service.

The problem is, people don’t know what to use it for; they think database, and are left with effectively a hashmap in the cloud.

We do the same: EC2 + S3 + Postgresql

And thanks!

Terry

@Terry Jones – Good luck with FluidInfo.

It’s interesting that nobody has mentioned here ThruDB, which uses S3, and they say is faster and cheaper than SDB.

That’s a good question and one that has certainly entered my mind. In addition to the lack of web content that you cite, the simple db forums at amazon are much less active than the s3 and ec2 forums.

Simple db is very different from what people are used to in a database and the benefits are not as obvious as are the benefits of s3 and ec2.

I think it may be a marketing and positioning issue. Focusing on the “simplicity” would probably be a better fit if there were official tools that made it simple to import and export data and perform other functions that database folks are used to.

Third party tooling continues to grow and with the massive price cuts, the “free trial” and the public beta I think that adoption will grow. But until tooling makes it truly simple to administer, I don’t know how sharply that adoption curve will rise.

FWIW: I think simple db is great technology, I use it and I hope that the answer to the question turns out to be “no”.

The 1K limit hasn’t been a big problem for us. We have an ORM layer that allows large fields to be stored in S3. Of course, you can’t search on that data, they are just blobs.

If you think about a typical web app and the objects you would have (e.g. users, groups, etc.) most have a relatively small number of metadata fields (e.g. username, email address, etc.) that easily fit within the current limits of SimpleDB. At least, that’s been our experience.

Thanks! You say “The biggest failing… the 1k field limit. SDB is effectively a document store” but it feels like you meant to say “a metadata store”?

Also, you remind me of a question I had when I read the docs originally. I meant to write something to test it, but perhaps you know the answer. Is the 256 attribute limit imposed at the domain level? I know a doc can have 256 attributes, but can two docs in the same domain have mutually independent attribute sets with the total being over 256? Am I making sense?

So why do I think people aren’t using it? Probably for the same reasons I’m no longer using it. A flat or document oriented data model is fast for development but often lacks flexibility. Couple this with a lack of export/import tools and you can get into trouble when its time for new features. The biggest failing… the 1k field limit. SDB is effectively a document store. A 1k field limit removes it from the running for most apps that are interested in this model (256 attribute pairs is also kinda small).

What is it good for? Directory data… with not too many fields… and not too big. :)

A summary of my SimpleDB presentation @ etech: http://bit.ly/CdUq

And, no, I don’t think SimpleDB has been a flop. There have been people on the waiting list for ever since I’ve been involved. But the price change and the move to provide a SQL-like SELECT query language are clearly meant to draw more people in.

Actually I was asking a question, and looking for input/comments/opinions, not making a statement. There’s even a question mark in the title. Did you need an invite to try out SimpleDB? I just remember it “launching” a year ago, but never had an interest in using it. I knew a couple of people who had NDA access prior to the launch (I just mailed one, asking him to comment).

Even if it’s in entirely private beta, I still find it very odd that those Google counts are essentially zero. If Google can find and index this web page in well under an hour, you’d think they could have found some other coder/blogger writing about SimpleDB in the last year. Not one single blogger saying “we are using SimpleDB” etc., in a whole year???

I’m building stuff on top of S3 and EC2 too. Every single day.

Thanks a lot for the comment in any case. BTW, who is “we”?

I work for a small start-up in the social space. We’re based entirely in AWS. We leverage EC2, SQS, S3 and SimpleDB. SimpleDB is the foundation of what we’ve built and we love it.

Your statement that it’s a flop is entirely unfounded and comes across as quite ignorant.

Amazing.