The Right Tool For The Right Job: Choosing The Best Data Storage Option

When architecture teams discuss data storage options, religious wars almost always follow suit. Now more than ever there are dozens of different data storage options available to teams. But instead of looking at data storage as X vs Y, it’s often better to consider X and Y. Learn how one startup ended up selecting five separate storage engines, and is still happy with the decision more than two years later.

This session will look at the following project/technologies/cloud offerings and outline the pros and cons as they related to the needs of the speaker’s startup, BrowserMob:

• On-premise MySQL
• EC2-based MySQL
• Amazon RDS (managed MySQL run by Amazon)
• Amazon SimpleDB
• Google App Engine (Master/Slave & High Replication)
• MongoDB
• Cassandra
• CouchDB
• Amazon S3
• In-memory caching
• Flat files (!)

Through trial-and-error and real-world production experiences, BrowserMob eventually settled on a mixture of on-premise MySQL, Amazon RDS, Amazon SimpleDB, Amazon S3, and even flat files. The final system can now easily handle the processing of 20TB+ per hour!

In addition to giving code examples and use cases for all of the popular storage techniques, this session will also cover the specific reasons why these technologies were chosen and how they were the best fit for BrowserMob.