Building an A/B Testing Framework for Web Applications

We’ll start with an overview of the roles that A/B testing can play in a web application (design optimization, feature ramp-up, backend migrations and others) and how it relates to other techniques (e.g., side-by-side testing). We’ll then outline the various approaches for implementing A/B testing and discuss their pros & cons.

In the second part of the talk, we’ll present one particular implementation, Etsy’s A/B framework (it’ll be open sourced by OSCON time) in detail. We’ll look at our PHP API and how we integrate it with a variety of other tools (Apache, Hadoop, Splunk…). We’ll include a few notes on experiment design along the way.

The last part of the tutorial will be hands-on: PHP developers can try out the Etsy framework; developers who are using or interested in other stacks can take the ideas from the talk and code up the rudiments of an A/B framework in their favorite language. We hope to take a look at some of these efforts and discuss how one might be able to build on them.