Dataset Multiplexer Catalog for Changing Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in managing large numbers of datasets stored across various data stores, requiring applications to be modified frequently due to changes in data storage, leading to errors and high costs.
Innovation Solution
A dataset multiplexer maintains a catalog of logical datasets with entries that automatically provide access to corresponding physical datasets, enabling applications to operate without knowledge of data store formats or locations, and dynamically updates these entries to accommodate changes in storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If applications directly access physical datasets in data stores, then data access is straightforward, but applications require frequent modifications when data storage changes
Solution Approach 1:
The patent introduces a dataset multiplexer as an intermediary component between applications and physical data stores. The multiplexer maintains a catalog that maps logical dataset identifiers to physical dataset locations and formats. When an application requests data, the multiplexer translates the logical identifier into the appropriate physical access parameters, automatically adapting to storage changes without requiring application modifications.
2Reliability
If applications are modified frequently to accommodate data storage changes, then data access remains functional, but errors increase and costs rise
Solution Approach 1:
The patent implements a catalog maintenance mechanism that proactively tracks and updates information about physical dataset locations, formats, and accessibility. Before applications need to access data, the multiplexer ensures the catalog reflects current storage conditions. This preliminary action prevents access errors and eliminates the need for reactive application modifications, thereby improving reliability and development productivity.
3Adaptability or versatility
If applications operate without knowledge of data store formats or locations, then applications are more portable, but a mechanism is needed to map logical to physical datasets
Solution Approach 1:
The dataset multiplexer serves multiple functions within a single component: it maintains the catalog of physical dataset information, translates logical dataset identifiers to physical access parameters, manages different data store formats, and handles location mapping. This multi-functional approach enables application portability while consolidating the necessary access mechanism complexity into a dedicated infrastructure component rather than distributing it across applications.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.