On-Demand Data Schema Modifications via History Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data schema changes in large data sets are costly and inefficient, as they often require creating duplicate data sets for online access, leading to increased storage and processing costs, and typically involve offline changes that disrupt data accessibility.
Innovation Solution
Implementing on-demand data schema modifications that apply schema changes as data is accessed, using a storage engine to process requests and store schema history, allowing for incremental application of changes without additional storage space, enabling near-instantaneous schema updates while maintaining data accessibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional data schema changes are implemented, then data schema can be updated to reflect new requirements, but storage costs and processing costs increase significantly due to creation of duplicate data sets
Solution Approach 1:
The patent extracts the schema change metadata from the data storage itself and stores it separately in a schema history table. This allows schema changes to be tracked and applied without creating duplicate data sets, as the actual data remains in place while only the schema definitions are updated and versioned.
Solution Approach 2:
Instead of copying the entire data set when schema changes are needed, the patent creates a virtual copy of the schema by storing schema versions in a history table. This allows multiple schema versions to coexist without duplicating the actual data, significantly reducing storage requirements.
2Adaptability or versatility
If offline schema changes are performed, then schema can be modified completely, but data accessibility is disrupted during the change process
Solution Approach 1:
The patent implements dynamic schema changes by allowing the schema to evolve over time through versioning. Multiple schema versions can coexist, and the system can dynamically switch between versions without disrupting data accessibility. This transforms the static, all-or-nothing schema changes into dynamic, incremental updates.
Solution Approach 2:
The patent performs preliminary actions by pre-creating schema versioning infrastructure and maintaining schema history before actual schema changes are needed. This allows schema changes to be prepared and applied in a controlled manner that maintains continuous data accessibility during the transition.
3Manufacturing precision
If schema changes are applied to large data sets, then data structure can be optimized, but processing time and costs increase significantly
Solution Approach 1:
The patent segments the schema change process into discrete, manageable steps: creating schema versions, storing them in history, and applying them incrementally. This segmentation allows schema changes to be processed in smaller batches rather than all at once, reducing processing time and enabling parallel operations.
Solution Approach 2:
The patent maintains continuity of useful action by allowing data access to continue uninterrupted during schema changes. The system can read from and write to data using current schema versions while changes are being prepared and applied in the background, eliminating downtime and improving overall efficiency.
Data Source
AI summary
A data store may implement on-demand data schema modifications. Request to change a schema for a data set in a data store may be received and a description of the change recorded as part of a schema history for the data set. The request to change the schema may then be acknowledged. When access requests directed to the data set are received at the data store, the schema history for the data set may be evaluated. If the schema history indicates that data that is to be accessed in order to service the data store needs to include one or more changes in the schema history, then a version of the schema that includes the one or more changes may be applied to the data.