Building a Business on Open Source Distributed Computing

Visible Technologies is a Social Media Analysis and Monitoring company. They collect millions of blogs and other forms of social media, performing data mining and heavy semantic analysis. Visible uses this data to provide advanced Business Intelligence to many clients, including several Fortune 100 companies.

Visible’s MS-exclusive “2.x” platform is being replaced with a “3.x” infrastructure built on software such as Hadoop, Nutch, Lucene, Mahout, Solr, Zookeeper, HBase, ActiveObjects, and more. It’s provided amazing agility and leverage for the R&D investment.

Bradford Stephens the Lead Engineer on the Data Platform team, which is in charge of Visible’s distributed computing core.

We’ll be discuss the lessons learned along the way so others can avoid going down the wrong path we once did (aka “RDBMS Hell”).

Presentation topics may include: -The importance of “scale-out” over “scale-up” from business and technology perspectives -BI without SQL Server – it can be done! -Overview of a collection, data processing, and document storage system -Leveraging Hadoop, Lucene, Mahout, Nutch, Lucene, Zookeeper, HBase, and more -MapReduce and the importance of high-latency, high-bandwidth computing -Semantic Analysis on Social Media with Distributed Systems and Machine Learning

Apache’s “Cloud Stack” is enabling a new generation of companies to process “Google-scale” data, promising to change the way we analyze and use the web forever.