Versioned Relational Dataset Management via Unified Table Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing relational database management systems (RDBMS) lack efficient mechanisms for storing and managing multiple versions of datasets, which limits access to historic data and collaborative data manipulation.
Innovation Solution
The proposed solution involves an apparatus and method for managing versioned datasets, which includes a processor and a non-transitory storage medium. This system receives structured queries to manipulate specific versions of a dataset, applies these queries to create an updated version, and stores it while keeping previous versions intact. It also updates a versioned dataset catalogue with lineage information and generates a database view based on affiliation encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple versions of datasets are stored using traditional RDBMS methods, then data integrity and version accessibility are maintained, but storage space and memory requirements increase significantly
Solution Approach 1:
The patent merges multiple dataset versions into a single unified table structure where different versions coexist through version identification columns rather than requiring separate physical tables. This combining approach maintains all version data in one storage unit, reducing overall storage requirements while preserving data integrity through the unified schema.
Solution Approach 2:
The unified table structure serves multiple functions simultaneously: it stores historical versions, enables version comparison, supports temporal queries, and maintains data integrity all within a single database object. This multi-functionality eliminates the need for separate storage mechanisms for each version while preserving all necessary capabilities.
2Ease of operation
If complete copies of each dataset version are stored, then full access and manipulation of each version is enabled, but device complexity and storage requirements increase
Solution Approach 1:
The patent segments the version identification into distinct columns (e.g., version number, timestamp, author ID) within the unified table, allowing the system to selectively filter and manipulate specific versions through SQL queries without requiring separate physical table structures. This segmentation enables complex version management through simple query operations.
3Loss of information
If historical versions are retained for lineage tracking, then data provenance and audit capabilities are improved, but storage requirements and data processing complexity increase
Solution Approach 1:
Lineage information is merged into the same unified table structure as the dataset versions themselves, with additional columns tracking parent-child relationships, transformation operations, and metadata about each version's origin. This eliminates the need for separate lineage tracking tables while maintaining complete provenance information.
Data Source
AI summary
A versioned dataset is managed. An updated versioned dataset is created by applying a structured query to a specific version of multiple versions of the versioned dataset stored in the backing dataset. Data of the versions including the specific version remain unchanged. A versioned dataset catalogue is updated to include in association with the updated dataset version, the structured query and lineage indicating that the updated versioned dataset includes a successor of the specific version. An affiliation encoding associated with the updated versioned dataset indicating which data of the updated versioned dataset matches data of the specific version and which data of the updated versioned dataset is deleted relative to data of the specific version, is updated. A database view of the updated versioned dataset is generated by selecting rows and projecting columns of the backing dataset corresponding to the updated versioned dataset according to the affiliation encoding.