A little known initiative called\u00a0XDATA\u00a0has spawned a collection of technologies that could bring advances to the world of Big Data. The program is funded by the sometimes mysterious, sometimes fanciful\u00a0DARPA\u00a0(Defense Advanced Research Projects Agency). In this case, it looks like DARPA\u2019s imaginings could bring some tangible benefits to the real world.<\/p>\n
A few days ago, XDATA put up a\u00a0catalog\u00a0of open source software that has come out of the effort. Though many of the applications are research-grade works still in development, some are being used in production in some of the web\u2019s Big Data cornerstones like Twitter and Facebook and are serious enough to have formed the core technology of promising startups (Trifacta and Databricks). The projects each address key challenges in Big Data are likely to bring substantial improvements in our ability to make sense of massive amounts of data. I\u2019ll try to highlight some of the major software packages in terms of some over-arching themes.<\/p>\n
Big Data involves volumes of data that exceed the capacity of any one machine to store and process. Systems for managing Terabytes and Petabytes of information therefore have to coordinate data processing across multiple machines \u2013 sometimes in the thousands of them. Hadoop was one of the first widely available platforms for getting computers to collectively process data at this scale.<\/p>\n
Mesos\u00a0is an XDATA funded platform that could be a successor to Hadoop. Roughly speaking, it extends Hadoop by enabling a more diverse set of systems to participate in the data processing. Where Hadoop is setup to support one kind of processing (called MapReduce), a collection of computers are able to adopt different patterns of processing that allow a departure from a \u201cone size fits all\u201d approach. Mesos was initially developed by graduate students at Berkeley interested in expanding Hadoop\u2019s capabilities.<\/p>\n
Twitter\u2019s backend engineering was one of the first groups to grasp the potential of this system and Mesos is now widely used for Big Data inside Twitter. The Mesos team eventually spawned another effort called\u00a0Spark, which is also being adopted in the Big Data community. Spark allows Big Data software to trade off memory and disk in ways that allow increased flexibility and performance, in some cases allowing workloads to complete 100 times faster than Hadoop MapReduce.<\/p>\n
BlinkDB is a new kind of database that allows queries that balance accuracy against speed.\u00a0BlinkDB\u00a0leverages the idea behind opinion polling. A pollster doesn\u2019t have to ask the entire population of a state how they\u2019re likely to vote to get a sense of the way things are going to go in an election: if they can get a representative sample of a population then they can give a prediction with a margin error based upon the response of people in the sample.<\/p>\n
Similarly, BlinkDB builds small samples of a given data base and can respond with query results that have a given margin of error for a query result. In cases where there\u2019s historical data to base the sampling on, BlinkDB can give responses 200 times faster than traditional querying. Where this really helps is in real time applications, where getting a \u201cclose enough\u201d sense of what a population is doing can feed real time response. Suppose that you are a have an online store and you\u2019d like to tie promotions to real time measures of customer sentiment \u2014 quickly enough impact purchasing trends. BlinkDB could save you the effort of having to wait overnight for a full analysis to run across all your data. If it could assess sentiment quickly with a 10% accuracy, that might be good enough to help with hour to hour marketing decisions.<\/p>\n
Beyond storing the data and running queries, the goal of data science is to extract meaning from Big Data. To return to the store example, there are complex algorithms that could be used to predict customer behavior or to understand their feelings about the things they\u2019ve already purchased. Data scientists typically rely on special statistical packages like R and SAS for designing these algorithms. These tools tend to be the platforms of record for data science \u2013 most researchers use them to develop state of the art algorithms, they provide a breadth of coverage that is unparalleled, the and the implementations have been validated by the world\u2019s statistical community.<\/p>\n
The problem is that these tools tend to be very slow primarily because of design decisions around memory management: a skilled programmer could probably get 10 to 100 times improvement for a given R algorithm by rewriting it in a compiled language like C++. Further, these tools are ill-equipped to deal with the huge data sets that are now critically relevant to business. The R installation documentation recommends data sizes no larger than 20% of available memory.<\/p>\n
XDATA awardees have developed new modules for R that allow processing of large data sets using a variety of techniques including integration with Hadoop (the RHIPE project) and memory management modules that support large data sets (bigmemory). Another XDATA recipient is the\u00a0Julia\u00a0statistical language project. Julia is a new computing tool which is both fast \u2013 for some calculations 100 times faster than R \u2013 and has the statistical libraries needed to support serious data science.<\/p>\n
Perhaps the most important outcome of the XDATA effort will be the development of new tools to \u201csee\u201d the data.\u00a0Bokeh\u00a0is a graphics library being developed by Continuum Analytics allowing data scientists and BI professionals to deliver stunning graphical displays of data without much low level coding.\u00a0Vega\u00a0is another tool that provides a high level visualization language for laying out complex visualizations and\u00a0imMens\u00a0supports visualization of large networks.<\/p>\n
The great thing about all these tools is that they are free, open source efforts that will no doubt support the development of the next generation of data science tools. In posts to come, I\u2019ll step through usage of some of these platforms.<\/p>\n