Natural Language Processing, Advanced Analytics, and Entity Resolution at Massive Scale

We’ve spent the past year helping the intelligence community (IC) build a brand new system from the ground up that mines massive amounts of structured and unstructured data (high tens to 100+ million documents), stores the results in a highly flexible semantic meta-model that’s massively distributable, lends itself to fierce analysis by the men and women who protect our country and we heavily use Java-based open source software to do it.

This presentation begins with an overview of the problem domain and our experiences with performing semantic analysis on massive amounts of messy structured and unstructured data, but quickly transitions to a discussion of how we’ve leveraged open source software to create a powerful solution stack for the IC. Specifically, we’ll discuss:

• NLP – How we’ve leveraged MALLET, MAchine Learning for LanguagE Toolkit, and other open source AI technology to build a natural language processor and entity extractor that’s competitive with the state of the art

• NoSQL – How we’ve augmented the traditional Hadoop stack to use Cassandra–a Java-based eventually consistent storage system–instead of HBase to build a massively distributable, fault-tolerant infrastructure that runs on commodity hardware

• UI/UX – How we’ve used Flare–an open source visualization toolkit inspired by Prefuse–to build a powerful Flex-based user interface that lends itself to advanced analysis.

A working demonstration using publicly available data will accompany the presentation (and hopefully be available at a public URL for users to tinker with during/after the presentation.