Forbes - Innovation

Data Plumbing From Stovepipes To Firehoses

Data is everywhere, but we know that already. What we perhaps don’t always know or think about is the plumbing that goes on beneath the data services that we all plug into every day.

In the current era of smart toasters and intelligent doorbells, we are in very real terms drinking from a firehose in terms of the amount of data being ingested. What matters now is that we don’t try and drink from the data firehose if it’s being fed by a ‘stovepiped’ lower-level information architecture; the potential for pipes to crack and so create a flood or gush that drowns us is (or at least should be) too great a risk to run.

What is stovepipe technology?

So what do we mean by this term stovepipe technology? In both real world engineering and software engineering, a stovepipe system is a sort of single chimney that is only capable of pushing its flow (smoke, water or data) through one single channel. A stovepipe functions on its own, individually. It shares no connection with other pipes, flumes or exhausts, so it is thus more prone to being overloaded, cracked or otherwise compromised.

If you’re going let your applications drink from the firehose of modern data ubiquity, then you don’t want them being exposed to all that power with the fragility of an archaic system beneath.

“Stovepipes are a fine way to support unique missions, for instance the automation of a single business process or the reporting of one business team’s activities. But to be truly insightful, you need the complete picture across all your business processes and teams. In the world of cloud, the web and modern applications, the mantra I’m afraid needs to be (and cover your ears if you must) stovepipes be damned,” said Shawn Rogers, vice president analytics strategy at TIBCO.

The real problem here for contemporary technology platforms is the fact that many organizations will be running legacy relational databases with stovepipe architectures. In contrast, the cloud computing model of virtualized and abstracted interconnected systems is almost the antithesis.

Stovepipes have a single channel. By contrast, the cloud has a potentially infinite number of distributed scale-out nodes and channels throughout its architectural make-up.

“While the cloud has an inherent technology advantage, the data traversing across it must also be harmonized. Take customer data in sales and marketing or accounts and receivables. If those data sources are stovepiped, then they sit in the dark. To delight your customers however and whenever you engage with them, you must first harmonize your customer data using master data management to provide a clear channel through which business can breathe,” added TIBCO’s Rogers.

MORE FOR YOU

An impedance mismatch

The end result of stovepiped legacy data structures attempting to serve higher-level cloud-based expansiveness is an impedance mismatch, both constructs can not exist in the modern IT universe without leakage, blockage or calamitous breakage.

This subject is close Jim Walker’s heart. In his role as VP of product marketing at Cockroach Labs, Walker is vocal on the use of cloud data orchestration technologies, such as the much-famed Kubernetes.

Walker reminds us that the legacy data systems of old were not constructed to work in the distributed information universe that we exist in today. Worse still, although organizations will attempt to modify, bolt-on and change these legacy systems, the inevitable outcome is the creation of a bottleneck, or worse, a single point of failure for the application.

“Running a NoSQL database on Kubernetes is a better alignment that can help overcome some of the challenge here, but organizations will still mostly likely experience transactional consistency issues. The stovepipe architecture of our legacy relational databases contradicts the distributed scale out architecture of Kubernetes, as they weren’t built with the same architectural primitives,” said Walker.

A contained answer to the challenge

Much of the tech industry has looked to containerization as a means of building the next era of cloud-native technologies, but what everyone has realized is that when applications are containerized (i.e. built from smaller discrete individual component pieces), our databases must be too. Are legacy databases built with the kind of modern modular interchangeability and connectedness that contemporary software application development engineers are able to use? Well, in a word, no.

Cockroach Labs’s Walker advises that ultimately, a forward-looking future-proofed database should look and feel like a traditional database, while simultaneously taking advantage of all the benefits of cloud infrastructure.

“Like pieces of a car, you can replace certain parts but the rest of the car stays the same. This is the type of architecture that developers and enterprises need to think about to develop, deploy and maintain software containers efficiently,” he said.

From stovepipes to Internet pipes

Thinking about the way forward here, we can use a lot of what we learned in the stovepipe era (the last couple of decades of the last millennium… and possibly further back in some cases) because the absolute core logic of data movement is still there. In tech circles, we do still refer to the Internet a pipe and we need it to be every bit as pipe-like as a stovepipe, we just need it to be smarter and stronger.

In future, we can perhaps think about opening wide and drinking from the data firehose with bigger gulps, the stovepipes will crumble and new plumbing is on the way.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button