Micro-Journal Transaction Logging for Persistent Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional transaction logging methods using a singular write-ahead/append-only log face challenges such as lengthy recovery processes, the need for CPU/memory-intensive log compaction, and potential system-wide corruption due to malformed transactions, especially when dealing with large data volumes.
Innovation Solution
Implementing micro-journaling techniques that utilize a pool of small, fixed-size logs (micro-journals) stored in byte-addressable persistent memory, where each transaction is recorded in a dedicated micro-journal, allowing for efficient recovery by replaying only in-flight micro-journals and eliminating the need for log compaction, while reducing the risk of corruption through transaction isolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a singular write-ahead/append-only log is used for transaction logging, then data consistency is ensured, but recovery time increases significantly for large data volumes
Solution Approach 1:
The patent divides the singular append-only log into multiple micro-journals, each capable of being independently committed and recovered. This segmentation allows the system to recover by processing only the relevant micro-journals rather than replaying the entire log, significantly reducing recovery time while maintaining data consistency through the same write-ahead logging mechanism.
2Reliability
If a singular append-only log is used, then transaction history is maintained, but CPU and memory resources are consumed for periodic log compaction
Solution Approach 1:
The patent implements a mechanism where micro-journals are discarded (freed) after their transactions are successfully committed and recovered. This eliminates the need for periodic log compaction operations, freeing CPU and memory resources while preserving the necessary transaction history in the active micro-journals.
3Device complexity
If all transactions are recorded in a single sequential log, then logging is simplified, but one malformed transaction can corrupt the entire transactional history
Solution Approach 1:
The patent segments the single sequential log into multiple independent micro-journals. This isolation ensures that a malformed or buggy transaction in one micro-journal cannot corrupt entries in other micro-journals, thereby protecting the integrity of the overall transactional history while maintaining relatively simple logging operations within each micro-journal.
4Quantity of substance
If the log continues to grow without compaction, then all available storage space is eventually consumed, but compaction operations cause throughput and latency degradation
Solution Approach 1:
The patent implements automatic discarding of micro-journals after their transactions are successfully committed. This continuous reuse of micro-journal space prevents storage exhaustion without requiring periodic compaction operations, thereby maintaining consistent throughput and latency performance while efficiently utilizing available storage space.
Data Source
AI summary
Techniques for using commit coalescing when performing micro-journal-based transaction logging are provided. In one embodiment a computer system can maintain, in a volatile memory, a globally ascending identifier, a first list of free micro-journals, and a second list of in-flight micro-journals. The computer system can further receive a transaction comprising a plurality of modifications to data or metadata stored in the byte-addressable persistent memory, select a micro-journal from the first list, obtain a lock on the globally ascending identifier, write a current value of the globally ascending identifier as a journal commit identifier into a header of the micro-journal, and write journal entries into the micro-journal corresponding to the plurality of modifications included in the transaction. The computer system can then commit the micro-journal to the byte-addressable persistent memory, increment the current value of the globally ascending identifier, and release the lock.