Analytics Bottleneck: Battling the (Unfortunate) Shape of Big Data

Consider that every data source requires some unique degree of handling, processing and/or technical infrastructure for end users to harvest a meaningful amount of information from it. (For the record, TABB Group believes that maximizing the rate of data-to-information conversion requires premium versions of this handling, processing and infrastructure.) Sure, various market data sources will have some commonalities among them, but then vary by geography, latency, or product specifications, and many other factors. Various news sources will also have overlaps among them, but may vary by language or format, for instance. To some degree, every data source is different from every other data source.

Now, consider that every data source may also be different from itself when temporal issues come into play. The treatment of an end of day (EOD) version of a data source is most likely going to be different – due to adjustments for corporate actions like dividends, share splits, or survivor bias - than streaming or intra-day versions of that same data source. Historical versions of a data source will most likely need to be modified to reflect ongoing changes to the underlying data and the technical specifications of the feed itself.

Finally, consider that even the simplest trading firm today may rely on hundreds of data sources – market data, fundamental, transactional, and others – and the most complex global banks and asset managers are likely to rely on thousands of data sources (plus the temporal permutations of each of them). It is no wonder that the imminent deluge of new digital-era data sources has most capital markets participants wide-eyed and frozen in place.

Everyone knows that harvesting big analytics from big data is a big challenge today. In many ways, most players are still struggling with the management of internally generated data from yesterday, and therefore are ill-equipped to even consider the mountains of data that are gathering externally. What may be a competitive advantage for a few right now—using the latest tools, technologies, infrastructure and processing methods to harvest greater intelligence from increasingly bigger data—is sure to become a competitive necessity sooner than anyone expects.

In this 15-page, 5-exhibit TABB Group Focus Note, Analytics Bottleneck: Battling the (Unfortunate) Shape of Big Data, we showcase how big data issues are just now impacting the global markets landscape; the extreme growth of data for the foreseeable future distorts conversion rates of that data into new (and in some cases, desperately needed) analytics; the increasing complexity of big data for capital markets use cases means that virtually no firm will possess the capability to manage big data challenges without the help of content aggregation and platform management partners; and how a much more collaborative community of (both internal and external) specialists is representative of a new enterprise data management (EDM) strategy going forward.