In recent decades the use of digital data has become an important element of conducting business. Technological innovations in the domains of business intelligence, big data, and analytics, as well as in artificial intelligence and machine learning, have further accelerated this trend. No sphere of business will remain untouched by these developments. However, most companies face challenges in maximizing value from data. Especially when it comes to scaling the value of data initiatives, concerns arise related to making data findable, accessible, and understandable. Additionally, as concepts like data mesh have emerged, promoting the perception of data as a product and the use of a self-service model of data consumption, the reusability and interpretability of data have become vital (see also FAIR data). Aiming to address these challenges, companies might look into data catalogs and/or data marketplaces. But what exactly are these, and how do they differ from each other?
What is an (Enterprise) Data Marketplace?
The term “data marketplace” has been around for a couple of years. The standard data marketplace is a space in which third-party data can be bought and sold. A data marketplace that is created to serve only a company’s internal data shoppers and invited participants is referred to as an enterprise data marketplace. The data sets offered can be enriched with commercially available data from external sources. Furthermore, it is possible to make data sets available for external (commercial) exchange.
What is a Data Catalog?
A data catalog is an inventory of data assets, which is created through the discovery, description and organization of distributed datasets. A data catalog allows data consumers (i.e., business users) to search for data sets that they may be able to use to strengthen their businesses objectives. The more modern data catalogs, which incorporate machine learning, feature automation that makes using them very straightforward. Tasks related to data cataloging, including metadata discovery, ingestion, enrichment, and the creation of cross-references between metadata, are all automated in this type of data catalog. These newer data catalogs can act as a strong foundation for businesses’ metadata management projects.
Now, What’s the Difference?
While both data marketplaces and data catalogs aim to accelerate the value generated from data initiatives, the marketplace has a greater focus on exchanging data for something in return, while the catalog primarily focuses on creating and maintaining an inventory of data assets. Can both co-exist? Yes, they can. Cataloging data and having an inventory of data assets is a prerequisite for a well-run enterprise data marketplace. Depending on the company culture and use cases for data value realization / data monetization, a data catalog can be extended with marketplace functionalities to allow data exchange and a shopping experience for data consumers. The data exchange can be realized as a financial transaction or through other forms of exchange of valuable items, e.g., coins and benefits.
Fostering Collaboration
The purpose of both a data catalog and data marketplace is to connect the data provider and consumers by making data findable, understandable and accessible. First and foremost, the ambition should be to foster collaboration and establish a data-driven culture. A collaboration platform like Assefy applies the advantages of both concepts to cultivate data as an asset; allow data collaboration; and make data findable, accessible, and understandable. The diagram below illustrates the core components of a data collaboration platform that incorporates a data catalog for data governance and inventory and metadata management, as well as marketplace functionalities via the storefront, giving data consumers a “shopping” experience.
The data pipeline describes acquisition of the data and the process of transformation from data sources to curated data sets. Furthermore, the pipeline incorporates the processes used to curate the data sets. These processes are also well documented and curated to make them reusable and promotable at the storefront. The data catalog comprises the data governance policies and practices for data ownership, decision rights, and marketplace ground rules. The data inventory holds and maintains information about all data assets. Metadata management is conducted in the data catalog end-to-end, from data sources to storefront. Data consumers access the data products and data services as products through the storefront.
What is the Best Solution for Us?
It depends. Your previous data initiatives, and with that your technology stack (enterprise architecture), and most importantly your data culture, are important when it comes to making the best decision. Understanding your current situation will allow you to make informed decisions about your next steps. You should ask yourself: What do I want to achieve? What is my target state?
According to recent research findings, people are the biggest roadblock when it comes to organizations becoming more data-driven. With regard to your target state, you should consider how you want to foster collaboration to establish a data-driven culture. Connecting the data provider with the data consumer and emphasizing the critical role of business users is crucial for your success. But as they say, Rome wasn’t built in a day, so an incremental approach might be worth considering. Expanding the collaboration capabilities of your organization, use case by use case, offers certain advantages.
Strong data governance and metadata management are the backbone of any data-driven organization. These capabilities might be a good starting point. Talk to our experts today for advice.