Versioned In-Memory Units for Non-Disruptive Database Servicing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in maintaining transactional consistency and efficiency due to lag in replication mechanisms between in-memory and persistent data, leading to unpredictable artifacts and increased latency during query processing, especially in environments with high DML activity.
Innovation Solution
The approach involves generating and maintaining multiple in-memory units (IMUs) for a mirrored data set, each with a snapshot time, and using staleness metadata to determine which data items are stale, allowing the database server to access up-to-date data from persistent storage when necessary, thereby ensuring accurate query results without the need for frequent IMCU repopulation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single in-memory unit (IMU) is maintained for a mirrored data set, then memory usage is minimized, but query availability and performance degrade during repopulation operations
Solution Approach 1:
The patent divides the single in-memory unit into multiple versioned IMUs (first IMU, second IMU, etc.), each containing snapshots of data from different time periods. This segmentation allows the system to maintain multiple historical versions in memory simultaneously, enabling queries to access appropriate versions without triggering repopulation operations, thus improving query throughput while managing memory through structured version retention.
Solution Approach 2:
The patent pre-loads multiple versions of the in-memory unit into volatile memory before they are needed for query processing. By maintaining multiple pre-populated IMUs with snapshot times, the system eliminates the need for time-consuming repopulation operations during query execution, thereby improving query availability and performance while the memory management system handles the quantity trade-off.
2Reliability
If in-memory data is frequently repopulated to maintain freshness, then data accuracy improves, but latency increases during repopulation operations
Solution Approach 1:
The patent pre-populates multiple versions of the in-memory unit (first IMU, second IMU, etc.) with snapshot data from different time periods before they are needed. This preliminary action ensures that when queries need historical or current data, the appropriate version is already available in volatile memory, eliminating repopulation latency while maintaining data accuracy through versioned snapshots.
Solution Approach 2:
The patent creates multiple copies of the in-memory unit at different snapshot times and maintains them as separate versioned IMUs. Instead of repeatedly repopulating a single IMU, the system copies data at snapshot moments and preserves these copies, allowing queries to access accurate historical data without triggering time-consuming repopulation operations.
3Reliability
If multiple versions of in-memory units are maintained, then query availability during repopulation improves, but device complexity increases
Solution Approach 1:
The patent segments the in-memory data into distinct versioned IMUs (first IMU, second IMU, etc.), each with clearly defined snapshot times and data ranges. This segmentation provides a structured framework for managing multiple versions, making the complexity tractable through organized version control rather than unstructured data management.
Solution Approach 2:
The patent implements a version management mechanism that tracks which IMU versions exist, their snapshot times, and their validity ranges. This feedback system allows the database server to automatically determine which version to use for any given query, managing the complexity of multiple versions through systematic version tracking and selection logic.
4Reliability
If in-memory data is kept synchronized with persistent data through replication, then transactional consistency is maintained, but performance degrades due to replication lag
Solution Approach 1:
The patent creates snapshot copies of persistent data at specific points in time and stores them as versioned IMUs in volatile memory. These copies are taken at controlled intervals or events rather than being continuously synchronized through replication, eliminating replication lag while maintaining consistency for historical queries through the versioned snapshot mechanism.
Solution Approach 2:
The patent performs data copying from persistent storage to in-memory units at predetermined snapshot times before queries need the data. This preliminary copying action ensures that historical versions are already available in fast volatile memory when needed, eliminating the need for real-time replication synchronization and its associated performance penalties.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Techniques for non-disruptive versioning of in-memory units in a database are provided. A database server generates and maintains a first IMU that reflects changes made to a mirrored-data-set up to a first snapshot time, and a second IMU that reflects changes made to the mirrored-data-set up to a second snapshot time. During a first period, the database server responds to updates to first data items in the mirrored data by storing first staleness metadata that indicates that the copies of the first data items in the first IMU are stale. During a second period, the database server responds to updates to second data items in the mirrored data by storing second staleness metadata that indicates the copies of the second data items in the second IMU are stale. The database server responds to a request by accessing the first IMU or the second IMU.