Map Reduce All the Way Down

Russell Branca (Cloudant)
Data
Location: Portland 256
Average rating: ***..
(3.10, 10 ratings)

Map reduce opens up a number of interesting possibilities in data processing, especially in the commonly talked about areas of big data, but there are number of other less explored applications of map reduce. One such possibility is pipelined map reduce operations gradually funneling from the database down to the browser. This allows for device manageable levels of data, while providing different levels of filtering.

On the database level, CouchDB’s incremental map reduce will be used to turn large volumes of data into an efficiently indexed structure allowing a variety of dynamic queries. These queries be aggregated on different levels, providing efficient queries in either raw form, or aggregated down to a desired data size.

The server level will be powered by a Javascript implementation of map reduce, that will allow fully dynamic post processing of multiple map reduce queries and other operations. This functions primarily as a staging area for combining data sets and other processing that are too large for the browser, or require more custom and dynamic request patterns.

The final map reduce level will happen client side in the browser. This is where low latency filtering on top of funneled data will provide fast interactive feedback for user queries and visualizations.

The other goal, and primary motivation, for shared Javascript on the client and server here, is to facilitate dynamic selection of the execution environment. This allows the map reduce job to happen locally on your desktop or mobile device, and then kick back out to the server for larger data processing dynamically. This is the big idea of the talk, exploring paradigms for data processing in dynamic environments. For instance, a high powered desktop can handle larger volumes of data processing than a mobile phone, but that doesn’t mean they shouldn’t be able to ask the same questions, or that the desktop should sacrifice performance and power for mobile clients, and where the server runs of memory, the database steps in to handle even larger volumes of data.

This talk will take a deep dive into exploring the “map reduce all the way down” paradigm, and examine how to progress through the different levels of the pipeline.

Russell Branca

Cloudant

I’m Russell Branca, you may know me as Chewbranca. I’m a programmer at Cloudant, CouchDB aficionado, enjoyer of food, and father to a red headed tornado of a toddler.

Comments on this page are now closed.

Comments

Picture of Jon Guidry
Jon Guidry
07/25/2013 12:12pm PDT

Hi there – do you know if OSCON has the slides for this presentation yet? Thanks :)

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Sharon Cordesse at (707) 827-7065 or scordesse@oreilly.com.

Contact Us

View a complete list of OSCON contacts