Scribe - Moving Data at Massive Scale

In just six years Facebook has grown from an experiment in a dorm room to a service used by over 400 million people worldwide. This explosive growth has presented many challenges in systems software and required the development of many custom solutions to handle the unique problems that arise at this scale.

One piece of software we’ve found to be particularly useful in scaling our site is Scribe, an open source system for aggregating massive amounts of logging data from thousands of machines, or more generally moving around large amounts of data in an asynchronous and mostly-reliable way. We initially wrote Scribe for just a couple of use cases, to replace special-purpose systems that were busting at the the seams as we grew. It turned out to be so useful that it now handles hundreds of use cases and delivers over 150 billion messages a day. Scribe is also now being used by Twitter.

In this talk I’ll describe the design philosophy and goals for Scribe and the decisions made in its implementation. I’ll discuss some of the lessons learned as we brought it into production and scaled it with Facebook, and what our plans are for the future.