Log Segment Directory Shadow Paging for Database Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data recovery processes are slowed down due to the need to scan all log segments for metadata information when restarting a database, especially when log segment metadata is stored at a well-defined position and changes occur, requiring modifications that affect the entire recovery process.
Innovation Solution
The implementation of a log segment directory that loads pages into memory, removes redundant pages, and modifies metadata by updating page generation values and checksums, allowing for efficient access and management of log segments during data recovery, including the use of shadow paging to handle incomplete operations and free up storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all log segments are scanned for metadata information during database restart, then data recovery completeness is ensured, but recovery time increases significantly
Solution Approach 1:
The patent creates and maintains a log segment directory in advance that indexes metadata for all log segments. This directory is updated continuously as log segments are created, closed, or modified. During recovery, the system loads this pre-built directory into memory and uses it to directly identify which log segments contain committed transactions, eliminating the need to scan all log segments and significantly reducing recovery time while maintaining complete data recovery.
Solution Approach 2:
The patent creates a copy of log segment metadata in the form of a directory structure that is maintained separately from the actual log segments. This directory contains indexed information about each log segment including their status and location. During recovery operations, the system uses this copied metadata directory to quickly determine which log segments need to be processed, avoiding the time-consuming operation of scanning all log segments while ensuring no committed changes are missed.
2Ease of operation
If log segment metadata is stored at a well-defined position, then access simplicity is improved, but modification complexity increases when changes occur
Solution Approach 1:
The patent separates metadata management from log segment storage by creating a distinct log segment directory structure. Each log segment's metadata is stored as a separate entry in this directory, which is maintained independently from the actual log segment data. This segmentation allows metadata to be accessed simply through the directory index while modifications can be made to individual directory entries without affecting the physical structure of log segments, reducing modification complexity.
3Loss of substance
If shadow paging is implemented for log segment directory, then storage space management is improved, but memory usage increases
Solution Approach 1:
The patent implements shadow paging where multiple versions of the log segment directory are maintained in memory. When a page is modified, the system creates a shadow copy of the old version and gradually transitions to the new version. Pages that are no longer needed are identified and discarded, with their memory space recovered and reused. This approach improves storage space efficiency by eliminating redundant data while managing memory usage through systematic allocation and deallocation of page versions.
Data Source
AI summary
Replay of data transactions is initiated in a data storage application. Pages of a log segment directory characterizing metadata for a plurality of log segment are loaded into memory. Thereafter, redundant pages within the log segment directory are removed. It is then determined, based on the log segment directory, which log segments need to be accessed. These log segments are accessed to execute the log replay. Related apparatus, systems, techniques and articles are also described.