Crash Reporting: Mozilla's Open Source Solution

Nothing exercises the limits of software like turning it over to the end users. While in-house testing is invaluable, the end users can often find the neglected edge cases or subtle race conditions in a more reliable manner. However, once the software is in the end user’s hands, postmortem crash information is usually lost. Even if it is retrieved, the users were not using the version instrumented with debug information. Because of those problems, valuable trends about crashes can go unnoticed, and opportunities for improved software robustness are forfeited.

By taking the Google Breakpad project and enhancing it with Mozilla’s own Socorro project, Mozilla has imbued its latest generation of products with the ability to overcome these problems. The client library component of Google Breakpad lives in the application to collect machine independent crash dump data. When a crash occurs, it captures the state of the application (frames, threads, modules, etc) and transmits that information to a waiting Socorro Server. The Socorro Server can handle multiple simultaneous submissions and will filter and schedule them for further analysis. The Socorro Server can be scaled simply by adding more instances, which allows the Mozilla installation to accept more than a million crash reports a day. Socorro Server uses Google Breakpad libraries to generate useful stack traces from crash reports. The data ends up in a PostgreSQL database overseen by the Socorro UI, which is a PHP web application that provides information on specific crash reports and trends.

In this talk, the three primary Mozilla developers in charge of the three parts of the project will discuss the architecture, deployment, and use of the Socorro/Google Breakpad system. While the path has not been entirely smooth, this Open Source project has proven invaluable to Mozilla.