AMQP in Production: Building and Scaling High-Performance Compute Clusters

The explosion of cloud services over the last 2 years has demanded new techniques for simple, fast inter-server communication and AMQP has been highly effective at addressing this need. Most non-trivial web services require mission-critical data processing behind the scenes but there are few clear tutorials on setting up, managing, and scaling compute clusters. This aims to be one such guide.

In this presentation, we’ll explore what AMQP is and how it can be applied. We’ll dive in to installing and configuring RabbitMQ and setting up simple message producers and consumers in Ruby.

With the basics covered, we’ll address more interesting challenges specific to putting a compute cluster in production:

• Designing worker servers to be 100% autonomous
• Delegating jobs to specific task servers
• Collecting real-time stats for performance analysis
• Throughput monitoring
• Worker failure recovery
• Automatic scaling under load

By the end of the presentation, you should have a strong understanding of how to architect compute clusters for your application and what to watch out for along the way.