Binary Hierarchical Data Versioning with Snapshots and Deltas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database applications face performance issues when dealing with very large data sets due to cache management challenges and decreased response times during data storage and retrieval.
Innovation Solution
Implementing a hierarchical data management system that uses binary encoding and versioning techniques, including immutable snapshots and delta records, to efficiently store and manage large data hierarchies, reducing the need for changing or deleting existing records and optimizing data transfer between storage and memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional relational database tables are used to store hierarchical data, then data can be stored and retrieved, but cache management becomes difficult and response time increases when dealing with very large data sets
Solution Approach 1:
The patent segments hierarchical data into immutable snapshots and delta records. Immutable snapshots contain stable historical data while delta records capture only changes. This segmentation allows the system to load only necessary data into memory rather than entire hierarchical data sets, improving cache management and response time for large data sets.
Solution Approach 2:
The patent extracts only the necessary data from hierarchical data sets by using delta records to load only changes since the last snapshot. This extraction approach reduces the quantity of data loaded into memory and cached, thereby improving database response time while maintaining access to complete hierarchical data through the combination of snapshots and deltas.
2Ease of operation
If complete hierarchical data sets are loaded into memory for processing, then data can be accessed, but cache management issues arise and storage space requirements increase
Solution Approach 1:
The patent segments data into immutable snapshots (containing historical data) and delta records (containing changes). This segmentation enables the system to store only necessary data in memory and on disk, reducing storage space requirements while maintaining full data access capability through the combination of snapshots and deltas.
Solution Approach 2:
The patent uses immutable snapshots as a preliminary data structure that stores historical data in a compressed format. By loading snapshots first and then applying only the necessary delta records, the system reduces the volume of data that needs to be stored and cached, thereby reducing storage space requirements while maintaining ease of data access.
3Adaptability or versatility
If traditional data storage methods are used, then data can be stored, but versioning flexibility and performance decrease when handling large data sets
Solution Approach 1:
The patent segments versioning into immutable snapshots (representing historical versions) and delta records (representing changes between versions). This segmentation provides versioning flexibility by allowing the system to load only the necessary snapshots and deltas for a particular version query, thereby maintaining high retrieval performance even for large data sets while supporting flexible versioning operations.
Data Source
AI summary
A database manager is disclosed that retrieves database records having binary encoded data from a database and instantiating objects in an in-memory database. Binary encoding compresses data, allowing many subrecords to be stored a single blob field of a database record. Retrieving chunks from storage reduces transfer time by reducing the size of data and the number of operations needed to retrieve all the subrecords.The database manager receives database access requests from a database application. Changes made to the database objects and committed by the application are written back to the persistent database as versioned delta records. In a subsequent session, loading the database from storage includes first loading the most recent snapshot record, then applying changes to the data stored in delta records. The changes stored in the delta records are applied to the data in the snapshot record in the order in which they were made.