Log-Structured Key-Value Store Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log-structured storage systems face challenges in proper recovery from failures during garbage collection, as existing methods do not effectively manage incomplete transactions and broken back chains, leading to potential data inconsistencies and increased recovery time.
Innovation Solution
A method for recovery from failures during garbage collection in a chronologically ordered log-structured key-value store is implemented, where a processor records specific offsets within a garbage collection target slot and performs transactional garbage collection, skipping records until the end of the slot, and rolling back the garbage collection transaction by inserting abort records and fixing broken back chains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If garbage collection is performed in log-structured storage systems, then storage space is reclaimed and efficiency is improved, but system failures during garbage collection can cause data inconsistencies and extend recovery time
Solution Approach 1:
The system performs preliminary actions by recording the starting offset of each garbage collection operation in a transaction log before the actual garbage collection begins. This preliminary recording enables the system to resume or rollback the operation consistently after a failure, preventing data inconsistencies without sacrificing the efficiency benefits of garbage collection.
Solution Approach 2:
A transaction log serves as an intermediary between the garbage collection process and the storage system state. This intermediary records the progression of garbage collection operations, allowing the system to verify completion and maintain consistency after failures, thereby resolving the contradiction between reclamation efficiency and data consistency during failures.
2Loss of substance
If traditional garbage collection methods are used, then storage space is reclaimed, but recovery time increases due to incomplete transactions and broken back chains
Solution Approach 1:
The system implements feedback by continuously recording the state of garbage collection operations in a transaction log, including starting offsets and completion status. This feedback mechanism allows the recovery process to quickly determine whether garbage collection operations were completed or need to be rolled back, significantly reducing recovery time while maintaining effective space reclamation.
Solution Approach 2:
By preliminarily recording operation metadata before garbage collection executes, the system prepares the necessary information for rapid recovery. This preliminary action eliminates the need for time-consuming analysis of incomplete transactions during failure recovery, thus reducing recovery time while preserving storage space reclamation effectiveness.
3Productivity
If garbage collection operations are performed without transactional control, then storage efficiency is improved, but data consistency cannot be guaranteed upon failure
Solution Approach 1:
The transaction log serves multiple functions: it records garbage collection operations for efficiency tracking, provides rollback capability for consistency, and enables recovery acceleration. This multi-functional approach allows the system to maintain storage efficiency through garbage collection while simultaneously guaranteeing data consistency through transactional control mechanisms.
Solution Approach 2:
The transaction log acts as an intermediary that bridges the gap between efficient garbage collection operations and data consistency requirements. By mediating between the garbage collection process and the storage system state, it enables both storage efficiency and reliability to coexist through proper transaction management and failure recovery mechanisms.
Data Source
AI summary
One embodiment provides a method for recovery from failures during garbage collection processing in a system including recording, by a processor, a specific offset within a garbage collection target slot of a log structure associated with a garbage collection transaction. Each transaction record of the garbage collection transaction includes a garbage collection target slot, a victim slot and a beginning offset in the garbage collection target slot.