Fixed-Size Log Destaging for Fast Recovery and Record Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log management techniques in computing systems are inefficient due to the lack of optimization in accessing and maintaining log records, particularly in identifying the tail of the log during recovery and managing fixed-size versus growable log portions.
Innovation Solution
Bifurcating the log into a fixed-size and growable log portion, where recent log records are stored in the fixed-size portion for optimized access and recovery, using log marking data to quickly identify the tail of the log during recovery, and destaging older records to the growable portion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If log records are stored in a single growable log portion, then the log can accommodate all log records, but accessing recent log records becomes inefficient and complex
Solution Approach 1:
The log is divided into two distinct portions: a fixed-size log portion for recent log records and a growable log portion for older log records. This segmentation allows optimized access paths for each portion, with the fixed-size portion enabling deterministic location calculation and the growable portion providing unbounded storage capacity.
Solution Approach 2:
Different access strategies are applied to different portions of the log. The fixed-size log portion uses deterministic location calculation based on log sequence numbers for fast access, while the growable log portion uses traditional sequential access. This local optimization tailors the access method to the specific characteristics of each log portion.
2Speed
If the entire log is made fixed-size, then access to recent log records is optimized, but the log cannot accommodate growing log data over time
Solution Approach 1:
The log is segmented into a fixed-size portion and a growable portion, allowing the system to simultaneously achieve fast access to recent records and unlimited capacity for historical records. The fixed-size portion maintains performance optimization while the growable portion provides adaptability.
Solution Approach 2:
The solution adds a temporal dimension to the log structure by separating recent records (fixed-size) from older records (growable). This dimensional separation allows the system to optimize for both speed and capacity without compromise.
3Reliability
If log marking data is used to identify the tail of the log, then recovery efficiency is improved, but the complexity of log management increases
Solution Approach 1:
Log marking data is extracted and confined to only the fixed-size log portion, separating the recovery mechanism from the entire log structure. This extraction simplifies the overall system by applying complexity only where necessary (fixed-size portion) while leaving the growable portion simpler.
Solution Approach 2:
Log marking data is applied locally to the fixed-size log portion rather than the entire log. This localized application provides sufficient recovery capability for recent records while avoiding the complexity overhead across the entire log structure.
4Productivity
If readers must search through the entire log to find log records, then all log records can be accessed, but the time to find records increases significantly
Solution Approach 1:
The log is segmented into fixed-size and growable portions, with deterministic location calculation applied to the fixed-size portion. This segmentation enables direct access to recent log records without searching, significantly improving retrieval productivity for the most frequently accessed records.
Solution Approach 2:
The deterministic location calculation method is pre-configured for the fixed-size log portion, allowing readers to directly compute the location of log records based on their log sequence numbers. This preliminary setup eliminates the need for searching during actual retrieval operations.
Data Source
Figure 1
Figure 2~3
Figure 4A
AI summary
The bifurcation of a log into a fixed-size log portion and a growable log portion. Log records are first written to the fixed-size log portion. At some point, the older log records within the fixed-size log portion are destaged to the growable log portion. Destaging may involve copying the log records from the fixed-size log portion and tacking the log record onto the end of the growable log portion. This destaging might occur one sub-portion (e.g., one virtual log file) at a time. The more recent log records are within the fixed-size log portion. Accordingly, new writes to the log occur to the fixed-size log portion. Furthermore, a large portion of the reads are of the more recent log records within the fixed-size log portion. Thus, optimizing performance within the fixed-size log portion results in significant improvement in the performance of the overall log.