<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=24166&amp;fmt=gif">

AtScale Blog

BIG BI:  Running with the Red Queen: BI is Here to Stay

Posted by on
Find me on:


Do you remember those word problems on tests like the SAT or the ACT?

They would go something like this:  Train A leaves the Station at 1:15PM, gradually accelerating for 7 minutes until reaching a speed of 52mph for a distance of 76 miles. Train B leaves 12 minutes later, accelerating to a speed of 47mph in 11 minutes. After one hour, how far behind Train A was Train B?

This sort of problem reminds me of the Business Intelligence (BI) business, alternately known as Decision Support, Analytics, Reporting, Data Discovery, etc. It seems that no matter how fast Train B can go (organizations implementing BI), they can only, at best, keep up with Train A (the relentless march of technology). 

In biology, this is called the ‘Red Queen Effect’. named after poor Alice in 'Through the Looking Glass', where the faster she runs with the Red Queen, the faster the landscape moves with them so they have to go as fast as they can to merely keep up.

Historically, most information technology emerges faster than organizations are able to adopt and implement it. BI also tends to lag behind the technology curve, both in enhancement to products as well as acceptance by clients. Therefor we can say with confidence that BI is subject to the Red Queen Effect too.

Why is this? Because BI doesn’t exist in a vacuum. From its inception, depending how you define it, BI relied on a series of periphery technology. It started with proprietary mainframe technology, then client server, then relational databases, then PC’s, then exotic massive parallel architecture, the Web, the explosion of external data from eBusiness, and finally today, cloud and big data. Each of these innovations has its own benefits and advantages; but where BI was concerned, there was always a sense that there was a moving target to get current. The Red Queen.

Challenges of the BI Red Queen Effect

BI has provided some amazing innovation over the years.  The problem hasn’t been lack of enthusiasm adopting and putting it to good use. Instead, the challenge enterprises faced was always how to port those beneficial features from previous architectures to new ones without losing the investment, skills and features in the process. The success record is mixed. BI has been extremely valuable to business analysts, but it  became somewhat unfashionable in recent years; Not to the businesses that relied on it, but to the vendors, the journalists and the industry analysts who considered ‘BI’ passé and always try to stay one step ahead of the innovation, not one step behind it.

The Red Queen and Big Data

Something not often discussed in the current focus around Big Data and data science is that the results of investigations using advanced analytics and quantitative methods (aka BI) have to land somewhere where analysts, decision makers and others  can review and understand them. In other words, the results of models built by data science have to be rendered in presentations for the non-numerates. In most cases, those presentations, reports and visualizations are generated by BI. Without BI there would be no Data Science.

When it comes to BI, one hugely valuable, and often dismissed capability that is crucial for enterprises to get the most value out of their Big Data as possible is OLAP; which we touched on in the previous blog 'Data isn't a Process, It's an Asset'. In a future article we’ll look even more closely at OLAP, its power, its foibles and the wide variation of the technologies. OLAP is a part of BI, but its most distinguishing features are its multi-dimensional model and interactive navigation using the multi-dimensional model. We will also examine how BI gets super-charged working within the Big Data ecosystem in terms of scale and complexity, but in order for that to happen, there is another technology needed, a uniform semantic model of the data. Semantics go beyond metadata, it exists to give meaning and structure to data. Not just one structure, but many, so different contexts and points of view can exist simultaneously using the same physical data.

In Summary

When we think about BI running with the Red Queen to stay current with Big Data, the semantic model is crucial because for BI to be successful, it has to become more agile and more dynamic. Data warehouses have a place for curated, consistent historical data, but they lack the flexibility for business today. Virtual structures defined by robust semantics provide the needed functionality.

In future posts I’ll dive in deeper to explain why and how you can get ahead of the Red Queen with your BI strategy.  Thanks for checking in. Subscribe to this blog and stay tuned for more.

~ Neil Raden.

Don't miss the first post in this series 'Big BI: Data isn't a Process, It's an Asset'.


About the Author: Neil Raden is an author, consultant, industry analyst and founder of Hired Brains Research, based in Santa Fe, NM. He has a passion for analytics based on decades of experience and strives to express it through his work in writing, speaking and advising clients.  Neil began his career as an actuary with AIG. In 1985, he started Archer Decision Sciences, consulting on analytics projects for Fortune 500 companies. Archer was one of the first to develop large-scale data warehouses and BI environments. In 2003, Neil expanded into a role as an industry analyst, publishing over 50 white papers, hundreds of articles, blogs, keynote addresses and research reports. He is also co-author of the book, Smart (Enough) Systems, about decision automation systems driven by predictive analytics. Please feel free to contact Neil directly at nraden@hiredbrains.com

Topics: Hadoop, Business Intelligence, Big Data

Learn about BI & Hadoop

The AtScale Blog is the one-stop shop for cutting edge news and insights about BI on Hadoop and all things AtScale.

Subscribe to Email Updates

Recent Posts