Unified Data Catalog Registry for Consistent Cross-Catalog Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems for enterprises face challenges in maintaining accurate, consistent, and up-to-date metadata across disparate data catalogs located in various physical or network locations, leading to inefficiencies and non-compliance with data governance policies.
Innovation Solution
A unified data catalog system utilizing platform and vendor-agnostic APIs to collect and manage metadata, ensuring metadata accessibility and accuracy, and providing a seamless view of enterprise data through data aggregation, processing, and domain definition, with features like data linkage and quality assessment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If metadata is stored across multiple disparate data catalogs in various locations, then data coverage and accessibility are improved, but metadata consistency and accuracy deteriorate
Solution Approach 1:
The patent introduces a metadata registry as an intermediary component that sits between multiple data catalogs and users. The registry stores mapping data that links information assets to their metadata locations across different catalogs, enabling centralized coordination without consolidating all metadata in one place. This mediator resolves the contradiction by providing unified access (improving accessibility) while maintaining the distributed architecture (preserving consistency through centralized mapping).
Solution Approach 2:
The system segments metadata management into two distinct parts: (1) the actual metadata stored in various data catalogs, and (2) the mapping information stored in the metadata registry. This segmentation allows each catalog to maintain its own metadata independently while the registry provides a unified view, thus improving accessibility without compromising consistency.
2Adaptability or versatility
If a unified data catalog system is implemented across multiple locations, then metadata integration is improved, but system complexity increases
Solution Approach 1:
The metadata registry acts as a simplified intermediary that manages the complexity of integrating multiple data catalogs. Instead of creating a complex direct integration between all catalogs, the registry provides a single point of coordination that stores mapping relationships, thereby reducing system complexity while achieving unified metadata access.
3Adaptability or versatility
If metadata is distributed across multiple data catalogs, then data storage flexibility is improved, but metadata retrieval efficiency deteriorates
Solution Approach 1:
The system performs preliminary action by pre-storing mapping relationships in the metadata registry that indicate where metadata for each information asset is located. When a retrieval request occurs, the system can immediately query the registry for the location rather than searching across multiple catalogs, thus maintaining storage flexibility while dramatically improving retrieval efficiency.
Data Source
AI summary
A computing system is described that includes a memory storing a plurality of information assets and a processing system of an enterprise, the processing system comprising one or more processors implemented in circuitry. The processing system is configured to receive a query for metadata for an information asset stored in one or more data catalogs of the enterprise. The processing system determines a location of the metadata using a metadata registry that stores mapping data that maps information assets to locations of metadata for the information assets, the locations corresponding to the data catalogs of the enterprise. The processing system retrieves the metadata from the determined location and provides the metadata in response to the query.