Dynamic Data Catalog Service for Distributed Storage Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity and cost of managing and accessing data stored across multiple locations, as data sets are distributed across different storage platforms optimized for various purposes, making it challenging to identify and access the appropriate data sets for operations.
Innovation Solution
Dynamic generation of data catalogs that provide a centralized location for structural data, including schema and access information, allowing clients to easily access data sets by discovering storage locations and updating the catalogs to adapt to changes in data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data sets are distributed across multiple storage locations to optimize storage and reduce costs, then storage efficiency and cost-effectiveness are improved, but the complexity of accessing and locating data sets increases
Solution Approach 1:
The patent introduces a data catalog service as an intermediary component that mediates between clients and distributed data sets. The catalog service maintains metadata about data set locations and provides a unified interface for accessing distributed data, thereby reducing access complexity while preserving the benefits of distributed storage architecture
Solution Approach 2:
The system implements feedback mechanisms where the data catalog service continuously discovers and updates information about data set locations across multiple storage systems. This feedback loop ensures the catalog remains current with data movements, enabling efficient access without clients needing to track storage locations manually
2Productivity
If data sets are distributed across multiple storage locations, then optimal storage utilization is achieved, but the time required to locate and access data sets increases
Solution Approach 1:
The data catalog service performs preliminary actions by proactively discovering and cataloging data set locations before clients need to access them. The system continuously monitors storage locations and pre-computes access information, so when clients request data, the location information is already available, eliminating search time
Solution Approach 2:
The system creates copies of metadata (catalog information) about data set locations and characteristics, storing this information centrally in the catalog service. This copying approach allows clients to access location information without querying multiple storage systems directly, significantly reducing data location time
3Ease of operation
If a centralized data catalog is implemented to simplify data access, then data accessibility is improved, but the system complexity increases due to additional infrastructure
Solution Approach 1:
The data catalog service is designed to perform multiple functions: it catalogs data locations, discovers new data sets, updates metadata, and provides access interfaces. This multi-functionality consolidates what would otherwise require multiple separate systems into a single universal service, reducing overall system complexity while maintaining ease of data access
Data Source
AI summary
Dynamic generation of data catalogs may be implemented for accessing data sets in different storage locations. Data sets may be accessed in order to extract portions of data. Structure recognition techniques may be applied to the extracted data in order to determine structural information for the data sets. The structural information may then be stored as part of a data catalog for the data sets. Requests to access the data catalog from different clients may be received and the requested structural data supplied so that the clients may access different data sets utilizing the supplied structural data. Data catalogs may be updated as changes to data sets are made.