AMQP in Production: Building and Scaling High-Performance Compute Clusters

Cloud
Location: D139-140
Average rating: ***..
(3.90, 10 ratings)

The explosion of cloud services over the last 2 years has demanded new techniques for simple, fast inter-server communication and AMQP has been highly effective at addressing this need. Most non-trivial web services require mission-critical data processing behind the scenes but there are few clear tutorials on setting up, managing, and scaling compute clusters. This aims to be one such guide.

In this presentation, we’ll explore what AMQP is and how it can be applied. We’ll dive in to installing and configuring RabbitMQ and setting up simple message producers and consumers in Ruby.

With the basics covered, we’ll address more interesting challenges specific to putting a compute cluster in production:

  • Designing worker servers to be 100% autonomous
  • Delegating jobs to specific task servers
  • Collecting real-time stats for performance analysis
  • Throughput monitoring
  • Worker failure recovery
  • Automatic scaling under load

By the end of the presentation, you should have a strong understanding of how to architect compute clusters for your application and what to watch out for along the way.

Photo of Nicholas Silva

Nicholas Silva

Box

Nick is a software engineer at Box developing distributed systems in Ruby, Erlang, PHP, and ActionScript. He has spent most of his life developing web applications, and could type LOAD ”*”,8,1 before he could read. He has presented at RubyConf, BarCamp, and numerous user groups in Boston and California. He enjoys fine cheeses.

Sponsors

For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at (707) 827-7065 or scordesse@oreilly.com.

View a complete list of OSCON contacts