Persistent Metadata Catalog for Unified Data Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In networked computer environments, data consumers face difficulties in discovering and accessing data from multiple third-party sources due to varying data formats, locations, and accessibility methods, requiring them to independently investigate each source for schema and access information.
Innovation Solution
A metadata catalog service is implemented to store and manage metadata about data sources, providing a centralized repository that allows producers and consumers to add, modify, and access metadata, enabling uniform discovery and access across different data sources and formats, and optimizing query performance by caching updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data consumers independently discover each data source and its format, then data sources can be accessed, but the complexity and time required for data discovery increases significantly
Solution Approach 1:
The patent introduces a metadata catalog service as an intermediary between data consumers and multiple data sources. This service maintains a centralized metadata repository that stores location, format, structure, and accessibility information for various data sources. Instead of consumers independently discovering each source, they query the metadata catalog service which provides unified access information, thereby reducing discovery complexity and time.
2Adaptability or versatility
If each data publisher provides data and definitions in different locations and manners, then data sources can be published independently, but the difficulty for consumers to discover and access third-party data increases
Solution Approach 1:
The metadata catalog service provides a universal interface that handles multiple data sources with different formats and locations. It performs multiple functions: storing metadata from various publishers, standardizing access information, and providing unified query capabilities. This allows independent data publishing while maintaining simple consumer access through a single standardized interface.
3Reliability
If metadata is frequently updated to reflect current data sources, then data accuracy improves, but the overhead for maintaining and synchronizing metadata increases
Solution Approach 1:
The system performs preliminary actions by maintaining a persistent metadata catalog that is proactively updated and synchronized before consumers need the information. The metadata catalog service pre-processes and stores access information, so when consumers query, the metadata is already current and ready, eliminating the need for complex real-time synchronization during consumer operations.
Data Source
AI summary
A metadata catalog may store metadata about data sources. The metadata may define table, column, partition and/or data formats, as well as title information, textual descriptions, modification information, data schemas and access information, such as location and connection information. A metadata catalog service may provide a shared, persistent, central repository for the metadata. The metadata of the catalog may be published via multiple access mechanisms and programmatically obtained and processed by other programs via the metadata catalog service. Metadata consumers may utilize the metadata catalog service to perform various metadata tasks, such as metadata retrieval, creation, addition, modification and/or deletion. The metadata catalog service may be directly integrated into existing data analytic tools as a persistent and durable metadata store allowing users to utilize the broad set of data processing tools.