Dataset Multiplexer Catalog for Storage-Independent Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern data processing systems face challenges in efficiently managing and accessing large numbers of datasets stored in diverse data stores without requiring applications to be modified when storage locations or formats change.
Innovation Solution
A dataset multiplexer is employed to create and maintain a catalog of logical datasets, enabling applications to access physical datasets dynamically, with automatic updates in response to storage changes, allowing applications to operate independently of physical storage formats and locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If applications are directly connected to physical datasets in data stores, then data access is straightforward, but applications require modification when storage locations or formats change
Solution Approach 1:
The patent introduces a dataset multiplexer as an intermediary component between applications and physical datasets. The multiplexer maintains a catalog that maps logical dataset names to physical dataset locations and formats. When an application requests data, the multiplexer translates the logical dataset reference into the appropriate physical dataset access parameters, thereby shielding applications from storage location and format changes without requiring direct application modification.
Solution Approach 2:
The system segments the data access functionality into separate components: the dataset multiplexer handles catalog maintenance and translation, while applications focus on data processing logic. This segmentation allows the multiplexer to absorb the complexity of managing storage changes independently, preventing this complexity from propagating to applications.
2Ease of operation
If a catalog of logical datasets is introduced to enable flexible access, then application independence from physical storage is achieved, but system complexity increases
Solution Approach 1:
The dataset multiplexer implements self-service mechanisms by automatically maintaining the catalog through event-driven updates. When physical datasets change (addition, deletion, format modification), the multiplexer detects these changes via events and automatically updates the catalog entries, eliminating the need for manual catalog management and reducing operational complexity despite the presence of the catalog.
Solution Approach 2:
The system incorporates feedback mechanisms where the multiplexer monitors events related to physical dataset changes and uses this feedback to dynamically update the catalog. This closed-loop approach ensures the catalog remains synchronized with the actual state of physical datasets without requiring external intervention, simplifying catalog management.
3Reliability
If dynamic updates to dataset catalog are implemented in response to storage changes, then data accuracy is maintained, but processing overhead increases
Solution Approach 1:
Instead of continuous monitoring, the system uses event-driven periodic updates where the catalog is refreshed only when specific events occur (dataset addition, deletion, or format change). This approach maintains data accuracy by updating the catalog at appropriate intervals triggered by actual changes, while minimizing processing overhead by avoiding unnecessary continuous synchronization.
Data Source
AI summary
A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.