To exploit the strategic business potential embedded in data, companies must enable their business users to realize its value. While the preparation of data is accomplished centrally, value realization is conducted across the organization in decentralized and cross-functional teams. This means that business users play a crucial role as data consumers, connecting the dots between data and business outcomes. Yet the importance of business users has been consistently neglected in the past by proposed enterprise data architecture concepts.
Addressing Business Users
Many organizations are investing heavily in their next generation data landscape with the hope of maximizing the value of data at scale to provide business insights, automate intelligent decisions, and ultimately monetize data. These efforts were the driving forces for the transition from data warehouses to data lakes. Today, the shortcomings of monolithic data lakes as the single source of truth with central teams for data management are becoming more and more notable: they are slow, the data is locked in, and it is becoming very expensive to maintain the data pipelines. The gap between data and business users as data consumers has been artificially closed by expert data teams. Without their intervention, business users were unable to use, understand, or apply data (see The Challenge of Becoming a Data-Driven Company). In the meantime, the demand for new data architecture concepts is becoming harder to overlook.
Other areas of IT management have adopted a more business-oriented approach decades ago. For example, operating systems are aligned to business capabilities, thus easily justifying their need and existence. Similarly, efforts in agile IT management focus on products with the end user (customer) in mind. Data-driven companies have deeply rooted the idea of data as a product in their organizational culture. Their goal is to democratize data and give power to the business departments to govern their data and to provide consumable data products to other units. All of this has given rise to the emerging concept of data mesh.
What is Data Mesh?
Data mesh is a concept for designing enterprise data architecture. It provides a modern approach to managing distributed architectures for analytical data. The main objective is to enable end users to easily access and query data where it lives without first transporting it to a data lake or data warehouse. Through this, data mesh fosters thinking of data as a product and offers a decentralized strategy to distribute data ownership to domain-specific teams that govern and manage data assets. Data mesh empowers data consumers and makes data available and accessible at scale, and reduces the dependency of data consumers on highly specialized data teams. It is not a single technology or solution, but rather a pattern for enterprise data architecture, building on top of prevailing data lakes and data warehouses. However, this approach of allowing a domain-driven, self-service, and decentralized management of data as a product requires a shift in data culture, thinking, and governance.
The design of enterprise data architectures in accordance with data mesh should follow four principles:
Data should be domain-driven, i.e., locally owned by the team responsible for collecting and/or consuming the data;
Domain data should be considered as a product;
Self-service data infrastructure as a platform should be prioritized, as this helps to overcome the challenge of duplicate efforts required to operate data pipelines for domains;
The management of computing resources should be federated, acknowledging the reality that in a distributed environment there will be multiple interdependent data products that must interoperate.
Data Assets as Data Products
Most notably, the data mesh concept proposes the convergence of data and product thinking, which enables data governance and management to become more aligned to the business structure of an organization. In addition, domain-driven offering of data to products brings new opportunities for business users for self-service consumption of data products. This results in shorter timeframes for achieving business outcomes and insights. Domain teams view data assets as data products, offering them to other business units. Furthermore, data products can be offered to external entities for data monetization. The characteristics of data products include the following: discoverable, addressable, trustworthy, self-describing, interoperable, and secure.
With the dawn of data fabric and data mesh concepts, new opportunities arise to connect business users and data, especially when cataloging data assets in complex architectures. It is clear to modern businesses that domain-oriented, decentralized data ownership and architecture must be leveraged. Self-service data infrastructures as a platform allow data consumers to access the full bandwidth of data and analytics products.
Find out how Assefy applies data mesh concepts to empower business users to find, understand, and use more data in their daily work.