Database Scalability Patterns: Sharding for Massive Growth

These days every database supports some form of replicated read slaves to enable horizontal scaling for read operations. That’s good, but what happens when you’re read traffic doesn’t spread uniformly across all nodes? What happens when you’re ability to write data and push it out amongst these nodes outstrips outstrips the capabilities of the master node? In these cases people turn to sharding, but how best to accomplish that isn’t quite the simple solution many people would like it to be.

In this talk we’ll focus on how to solve this problem; sharding data across multiple nodes to handle both read and write traffic, even in cases where that traffic might not be uniform. We’ll also cover how to keep things running; schema changes, index builds, and failover scenarios will all be covered, discussing the trade-offs and constraints that various database will impose upon you.

We’ll also discuss the other half of the equation; how this affects the applications that need to interface with the database. Sharded architectures add complexity to any environment, and navigating those waters can be tricky. We’ll cover some of the most common hurdles we’ve run across, and show how to stretch the patterns based on your needs.