What is data management? Data management is the ongoing process of ingesting, storing, arranging and maintaining data your organisation creates or...
How Large Organisations’ Data Warehouses Harm Business Agility
Large scale data warehouses—for all their utility as a precursor to powerfully sophisticated data retrieval and analytics
The notion of ‘agility’ makes its fair share of appearances incorporate literature oriented around competitive adaptability. Not least in the spheres of data management (DM), business intelligence (BI) and enterprise performance management (EPM).
Among those things sits the data warehouse—one of the stepping stones from DM to BI en route to EPM. Without the humble data warehouse, the business agility and data-to-decisions potential or having real-time insight wouldn’t be possible.
However, as larger organisations migrate more capacity into cloud-based microservices and architectures, it’s the data warehouse itself—until now a source of business agility—that risks becoming the very thing that threatens it.
Data Mesh Architectures and the Need to Rethink the Data Warehouse
A ‘data mesh’ is neither technology nor a platform; it’s a paradigm. One with the potential for helping large-scale organizations unlock huge data potentials in AI, ML, knowledge sharing, applications, and services.
The goal of creating a ‘data mesh’ architecture of data products is to allow separate business domains to provide a set of standard interfaces for ubiquitously and scalably querying data belonging to that business domain. This may sound familiar if you’ve encountered Domain Driven Design that models software according to input from that specific domain's experts. Similarly, the ‘data mesh’ is decentralised, with each domain benefiting from its own resources and implementation, without cross-influence.
What are the benefits of data mesh architecture?
Microservices running in the cloud focus on a single business domain that can be implemented as fully independent deployable services.
- The Data Platform team doesn’t need broad experience relating to the domains but should be skilled in Software Development and Data Engineering, acting also as a support for the domain in terms of technical knowledge.
- Because consumers stay close to the sources, they require little or no help from Platform teams for implementation or metadata integration. The idea is ‘self-service’, so bottlenecks become rare.
- Ownership is clearer. Domain Teams are responsible for reliable data provisions to consumers while Platform Teams support integration.
How the data warehouse hinders data mesh architecture utility
The inconvenient truth is this.
Large scale data warehouses—for all their utility as a precursor to powerfully sophisticated data retrieval and analytics—suffer from the same problem as large scale business applications; they’re monolithic. That is to say, they’re inflexible.
While microservices and mesh architectures are increasing agility in the application space, the data and analytics space is being left behind.
Thanks to the monolithic nature of legacy data warehouses, access to the data produced by agile microservices is being hundred, by the traditional data warehouse monolith, limiting the utility of microservice architectures.
What’s the answer? How Can Large Organizations With Data Warehouses Harness Data Mesh Architecture?
Director of emerging technologies at ThoughtWorks in North America,
Zhamak Dehghani, sums up the problem and solution nearly.
“Many enterprises are investing in their next-generation data lake, with the hope of democratizing data at scale to provide business insights and ultimately make automated intelligent decisions. Data platforms based on the data lake architecture have common failure modes that lead to unfulfilled promises at scale.
To address these failure modes we need to shift from the centralized paradigm of a lake or its predecessor data warehouse. We need to shift to a paradigm that draws from modern distributed architecture: considering domains as the first-class concern, applying platform thinking to create self-serve data infrastructure, and treating data as a product.”
Summing up: Rethinking the Utility of Your Data Warehouse
As organisations move to agile implementations with microservices, data architectures need to keep pace if maximum utility is to be gained—that means comprehensive data engineering able to prepare corporate data environments to capture those benefits.
Partner with EWX and get data-mesh ready
Thanks to a highly skilled team of experienced data engineers, EWX is helping change the way large organisations approach the future of data utility. Speak to us about partnership and our experience in designing microservices architectures and we’ll speak to you about modernising your data warehouse and developing your data-mesh architecture.