Terry Jones

Right now I am using AWS EC2 and S3 with PostgreSQL just because I am quite satisfied with it and PostgreSQL is really reliable.

I'll back again for sure, thanks for great article :D

I like the idea behind SimpleDB, but I wish there was a service like this for MySQL or relationship databases in general.

Hi Terry,

I think the appearance of Google App Engine and the “massively scalable Google Data Store” buzz took a lot of web app developers in the GAE direction.
GAE has a free level and I think Amazon has been forced to offer a free level due to market pressures.

The major reason for using SDB or GDS for that matter is the data model. Both of these allow dynamic attributes which are very important in modern semi-structured data.

I am working on a project to modernize technology for managing academic bibliographic items – this is an NSF funded project and the core need here is dynamic attributes because on of the core formats ‘bibtex’ is an extensible format. Also the need for a data base which is a key value store where values can be lists == keys can have multiple values.

While you can model this in a relational database using a key-value table, unifying the dynamic and static attributes in queries gets ugly fast.

Both GAE (through its Expando model) and SDB just natively, allow attributes to be added at run time and for keys to have multiple values. If your model needs this and you are not at the bleeding edge of data size then these datastores buy you a lot.

Currently GAE has some obscurely expressed limit on resource cost of a query and web request and it times out when you exceed it but you really don’t have a good idea how to tune it expect to try to do “less bad things”

SDB on the other hand tells you that if your query exceeds a certain well defined CPU usage it will time out – this suggests that if your queries are the equivalent of table scans or are above some level of computational cost you SDB will not work.

One significant disadvantage that GAE has is that the database is not exposed via REST’ish URL’s or via web services of any kind. It is up to the app you write to do that.

I have noticed that AWS continues to incrementally improve each of their web services in feature set and reduced cost as they scale up. I think the SDB data model takes getting used to and also early users of EC2 and S3 often experienced outages so SDB may be in “growing pain stage”

I am currently using GAE for my bibliographic item store and will be experimenting with SDB as well on this real world app and will keep you informed.

Nitin

Amazon SimpleDb has a lot of problems. Reliability is not up to snuff and it doesn’t really do what it should with performance. There are probably some vectors in which it scales better than MySQL (probably parallel reads/writes) but there are certainly others in which it does not scale as well (queries across hundreds of megabytes of data in a single table/domain). With MySQL, when you do a query that takes a “long time” it gets logged and you can improve its performance later. With SimpleDb, it croaks.

The costs and benefits list is quite long and complex so I won’t get into it here, but overall I think that only a tiny fraction of applications will find it preferable to MySQL.

Hi Alan

That’s very interesting. I’ve spent many years thinking about and building the product that Fluidinfo (my company) is now working to release. For a long time I resisted the “database” label. But I had enormous trouble getting people (mainly potential investors) to understand what we were trying to do. Finally I relented and started calling it FluidDB. I find the “database” part of the name slow and boring, and now you’ve pointed out that it might even be seriously misleading. We’ve cycled through a few names (one of the advantages of having something that takes a while to build is that you can play with different names), none of them particularly satisfying. It’s a product (initially) for programmers, so giving it a name that they can recognize seems sensible. But if they recognize it and think “that’s not a database”, it’s not good.

Hmmmm…..

Thanks,
Terry

You ask the question is SimpleDB a complete flop?

yes and no.

Yes in terms of marketing. Amazon have completely blundered this area, and really should drop the “DB” tag as quickly as possible. It is not a DB in the sense that most developers recognize and here in lies the confusion.

We are sold the wrong product to begin with.

Instead it is a global registry; nothing more, nothing less. A registry is not a database by any stretch.

We use SimpleDB to manage all our instances running in the cloud, in a variety of different of different providers, while we power some of the largest sites in Europe. Without it, we would have to be doing a lot of extra management, and without a doubt it is a very slick and useable service.

The problem is, people don’t know what to use it for; they think database, and are left with effectively a hashmap in the cloud.

Hi Engago

We do the same: EC2 + S3 + Postgresql

And thanks!

Terry

We use AWS EC2 and S3 with PostgreSQL.
Why because we know PostgreSQL and its’ reliability.
Why bother about and take a risk with SimpleDB as you have PostgreSQL?

@Terry Jones – Good luck with FluidInfo.

Another show stopper in SimpleDB is the low limit of attributes in a query – either 10, or 15, I don’t remember exactly. Lack of full-text search is also a very limiting factor. And last, but not least, inserts are slow – sometimes one takes 1-2 seconds (although there’s a new mass update capability now). Oh, also, the lack of integration with Ruby on Rails models is also a problem as many startups use Rails. On the contrary, AppEngine integrates well with Django and this is where Amazon should focus on – a drop-in replacement in existing applications.

It’s interesting that nobody has mentioned here ThruDB, which uses S3, and they say is faster and cheaper than SDB.

Amazon SimpleDB a complete flop?

That’s a good question and one that has certainly entered my mind. In addition to the lack of web content that you cite, the simple db forums at amazon are much less active than the s3 and ec2 forums.

Simple db is very different from what people are used to in a database and the benefits are not as obvious as are the benefits of s3 and ec2.

I think it may be a marketing and positioning issue. Focusing on the “simplicity” would probably be a better fit if there were official tools that made it simple to import and export data and perform other functions that database folks are used to.

Third party tooling continues to grow and with the massive price cuts, the “free trial” and the public beta I think that adoption will grow. But until tooling makes it truly simple to administer, I don’t know how sharply that adoption curve will rise.

FWIW: I think simple db is great technology, I use it and I hope that the answer to the question turns out to be “no”.

Each item in the domain can have a total of 256 attributes and/or values associated with it. Each item is completely independent of the other items in the domain (look ma, no schema!).

The 1K limit hasn’t been a big problem for us. We have an ORM layer that allows large fields to be stored in S3. Of course, you can’t search on that data, they are just blobs.

If you think about a typical web app and the objects you would have (e.g. users, groups, etc.) most have a relatively small number of metadata fields (e.g. username, email address, etc.) that easily fit within the current limits of SimpleDB. At least, that’s been our experience.

Hi Jay

Thanks! You say “The biggest failing… the 1k field limit. SDB is effectively a document store” but it feels like you meant to say “a metadata store”?

Also, you remind me of a question I had when I read the docs originally. I meant to write something to test it, but perhaps you know the answer. Is the 256 attribute limit imposed at the domain level? I know a doc can have 256 attributes, but can two docs in the same domain have mutually independent attribute sets with the total being over 256? Am I making sense?

Look at the feature set. Distributed, indexed, replicated data storage with a query language and client interface (http); that makes it available from production server and laptop alike. Add to that a simple single table (they call it a domain) approach and it’s pretty easy to build data intensive applications quickly. All this without a dba or sysadmin (people are usually the major expense, not hardware/disk).

So why do I think people aren’t using it? Probably for the same reasons I’m no longer using it. A flat or document oriented data model is fast for development but often lacks flexibility. Couple this with a lack of export/import tools and you can get into trouble when its time for new features. The biggest failing… the 1k field limit. SDB is effectively a document store. A 1k field limit removes it from the running for most apps that are interested in this model (256 attribute pairs is also kinda small).

What is it good for? Directory data… with not too many fields… and not too big. :)

A summary of my SimpleDB presentation @ etech: http://bit.ly/CdUq

We have been using SimpleDB for almost two years. We were in the original private beta and have continued on since then. I love it but if you look at it expecting to find a hosted MySQL, you will be disappointed. They don’t call it simple for nothing. But, what it does it does very well and we have built and run a number of apps for us and for other people that run completely in EC2/S3/SimpleDB/SQS. For me, I’d rather give up some features to gain scalability and I love that I don’t have to maintain the thing. It just works.

And, no, I don’t think SimpleDB has been a flop. There have been people on the waiting list for ever since I’ve been involved. But the price change and the move to provide a SQL-like SELECT query language are clearly meant to draw more people in.

@Jessie

Actually I was asking a question, and looking for input/comments/opinions, not making a statement. There’s even a question mark in the title. Did you need an invite to try out SimpleDB? I just remember it “launching” a year ago, but never had an interest in using it. I knew a couple of people who had NDA access prior to the launch (I just mailed one, asking him to comment).

Even if it’s in entirely private beta, I still find it very odd that those Google counts are essentially zero. If Google can find and index this web page in well under an hour, you’d think they could have found some other coder/blogger writing about SimpleDB in the last year. Not one single blogger saying “we are using SimpleDB” etc., in a whole year???

I’m building stuff on top of S3 and EC2 too. Every single day.

Thanks a lot for the comment in any case. BTW, who is “we”?

Quite honestly, SimpleDB has been in private beta for the last year. Only today was it made publicly accessible. The same can’t be said for S3 or EC2. They’ve been in public beta for 2+ years.

I work for a small start-up in the social space. We’re based entirely in AWS. We leverage EC2, SQS, S3 and SimpleDB. SimpleDB is the foundation of what we’ve built and we love it.

Your statement that it’s a flop is entirely unfounded and comes across as quite ignorant.

I think the issue is that SimpleDB is, as of yet, neither here or there. By that I mean that either people using Amazon services are happy building standard LAMP-style apps on top of a combination of EC2, S3 and open source relational databases like MySQL, or if they’re early adopters wanting column-oriented stores, are building apps using HBase/Hadoop running on EC2. IMO, SimpleDB gives you a lot of the functionality you would want with an easy-to-adopt API, but with some restrictions that are procrustean relative to HBase. Hence the lack of early-adopter uptake.

Ha – some of the results in my table are now wrong, because Google has indexed this page. I now have the only instance of “recommend SimpleDB” on the web, according to the big G. Likewise, this is apparently the only page on the web containing the text “we are using SimpleDB”.

Amazing.