info@meradia.com
What is data mesh? How did this evolve?
You cannot avoid hearing the term, but what is implied by the term ‘data mesh’? It is the latest step in architecture, governance, and data management evolution. We have lived (or view the history) through the databases to data warehouses, data virtualization, and then to data lakes and now lake-houses. Companies in our industry acknowledge the shortcomings of these frameworks, including accrued technical debt, ongoing and rising costs, and sprawling data lakes, which have not delivered the promised land to analysts and operational business users.
Data mesh is about four core concepts:
- Business domain ownership.
- Data as a product.
- Data self-service.
- Federated and decentralized data governance.
Data mesh has finally merged the best interests of the business with technology. It embodies the most modern way to manage, govern and provide access to data within a company. A key element is the focus of data mesh: It’s about the consumer.
Data mesh is an evolution where data is genuinely seen as a product, not unlike how products are available for consumers at a store. People can go into any store, find what they need relatively easily, then ‘check out’ and leave with their products. This works because stores have federated governance and domain ownership – specific departments own their displays and often have a say in designing product mix.
These internal departments, close to consumers, know the customers and how they wish to view and acquire the products. Newer stores by Amazon even allow consumers to avoid checkout and walk out with the products. Consumers want convenience and ease when viewing and purchasing products. Investment data should be no different. Our data consumers should be able to go into a ‘data marketplace’ and find what they need and are entitled to with relative ease, then experience a frictionless ‘check out.’
Data Mesh is a mechanism to do this. The core concept of data as a product could allow a fixed-income risk analyst to find a fixed-income portfolio attribution and performance data set (product) and another product of portfolio holdings. Merging these two data sources would give the analyst nearly all the data they need to run the analytics in their scope. Finding the data is one thing. Acquiring it is the next step. The other tenant of data mesh – self-service infrastructure would allow the analyst to access the data quickly and consume it for their purposes. The level of effectiveness and fit of the data would be ideal because the data ownership for the domain in question would be local to their subject area and tailored to their liking by product owners that know the analyst’s needs and preferences. The last tenant decentralized, or federated data governance, would ensure that the policies surrounding the storage and transformation of the data for the analyst were best aligned with the scope of their job.
Wouldn’t every firm want these benefits for their analysts and developers? The benefits of easier discovery of and access to data, faster time to insights from data, and faster time to develop and mature models for trading or reporting are all part of the package. Data Mesh embodies the marriage of technology and business, integrating people, processes, and systems. In this context, there is nothing to fear with data mesh.
But is data mesh achievable? Or is it the proverbial pie-in-the-sky ideal? Each of the four principles of data mesh is important, yet we have seen firms hampered by the data governance culture needed or the localized data ownership. The realities in the investment industry can be challenging. In our next piece, we will review insights we obtained at a recent symposium and share whether data mesh in investment management is fact or fiction.
Download Thought Leadership Article Process Design & Change GIPS Advisory Asset Managers Andrew Jacob