Main-memory database checkpointing via transaction log interface
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Main-memory databases face inefficiencies in snapshot schemes, particularly in processing overhead due to locking mechanisms and external data references, leading to resource wastage and incomplete data recovery during system crashes.
Innovation Solution
The system generates checkpoint data with timestamps for inserted and deleted data, manages storage resources by collapsing and merging data files, and restores the in-memory database state by filtering and replaying transaction logs, reducing storage consumption and improving recovery efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are used to assign transactions to files and reserve storage space in snapshot schemes, then data durability is improved, but processing overhead increases making snapshot creation less efficient
Solution Approach 1:
The patent extracts the locking mechanism from the snapshot creation process. Instead of using locks to assign transactions to files and reserve space, the system uses a lock-free approach where transactions are assigned to files based on timestamp ranges and hash functions, eliminating the processing overhead associated with locking mechanisms while maintaining data durability through the use of write-ahead logs and checkpoint validation
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of a transaction log and checkpoint metadata that mediates between transaction submission and snapshot creation. This intermediary layer allows transactions to be recorded and tracked without requiring locks during snapshot operations, as the metadata structures provide a coordinated view of database state without blocking operations
2Quantity of substance
If external data references are used to determine when and how snapshots are reduced or merged, then snapshot storage efficiency is improved, but processing overhead increases
Solution Approach 1:
The patent implements self-service by embedding all necessary information for snapshot reduction and merging directly within the checkpoint metadata and data files. Instead of referring to external data, each checkpoint contains self-descriptive information including timestamp ranges, transaction identifiers, and dependency relationships that enable automatic determination of when and how snapshots should be reduced or merged without external coordination or additional processing overhead
3Stability of the object's composition
If entire database is snapshotted even if only a portion has been changed, then data consistency is improved, but resource utilization becomes inefficient
Solution Approach 1:
The patent segments the database into multiple data files organized by timestamp ranges and data types. Instead of creating a snapshot of the entire database, the system creates incremental checkpoints that only capture changes to specific segments since the last checkpoint. This segmentation allows selective snapshotting of only the portions of the database that have changed, maintaining data consistency through segment-level versioning while significantly reducing the resources required for snapshot operations
Data Source
AI summary
The present invention extends to methods, systems, and computer program products for main-memory database checkpointing. Embodiments of the invention use a transaction log as an interface between online threads and a checkpoint subsystem. Using the transaction log as an interface reduces synchronization overhead between threads and the checkpoint subsystem. Transactions can be assigned to files and storage space can be reserved in a lock free manner to reduce overhead of checkpointing online transactions. Meta-data independent data files and delta files can be collapsed and merged to reduce storage overhead. Checkpoints can be updated incrementally such that changes made since the last checkpoint (and not all data) are flushed to disk. Checkpoint I/O is sequential, helping ensure higher performance of physical I/O layers. During recovery checkpoint files can be loaded into memory in parallel for multiple devices.