Tree Garbage Collection via Page Mappings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Persistent key-value stores face significant operational costs due to the high processing power and time required for regular garbage collection, which is necessary to manage the storage of old values and maintain system efficiency in internet-scale services.
Innovation Solution
The implementation of a storage engine with a key-value engine that employs a page mapping data structure and garbage collection methods, including delta and base page streams, to efficiently manage page mappings and reduce the frequency and extent of garbage collection, utilizing extent usage data structures to track valid data and adaptively defragment pages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If garbage collection is performed regularly to release storage spaces occupied by old values, then storage efficiency is improved, but processing power and time consumption increase significantly
Solution Approach 1:
The patent applies preliminary action by maintaining page mapping data structures that track the location and validity of pages before garbage collection is needed. The system proactively manages page mappings and identifies obsolete pages through version tracking, so that when garbage collection occurs, it can quickly identify and clean up old values without extensive scanning, thus reducing processing power consumption while maintaining storage efficiency
Solution Approach 2:
The patent replaces the traditional mechanical garbage collection process with a more efficient system using page mapping data structures and version tracking. Instead of scanning entire storage spaces to identify old values, the system uses metadata structures (page mappings) to directly locate valid and invalid pages, substituting the heavy mechanical scanning process with lighter metadata-based identification, thereby reducing processing power requirements
2Reliability
If garbage collection frequency is increased to maintain storage efficiency, then storage efficiency is improved, but operational cost increases
Solution Approach 1:
The patent implements feedback mechanisms through page mapping data structures that continuously track page validity and storage space usage. The system monitors the state of pages and their mappings, providing feedback information about which pages are still valid and which can be garbage collected. This feedback enables the system to perform garbage collection only when necessary and to optimize the timing and extent of collection operations, thereby maintaining storage efficiency while reducing operational costs associated with frequent garbage collection
3Reliability
If append-only storage is used to store new values without overwriting old values, then data integrity is improved, but storage space consumption increases
Solution Approach 1:
The patent applies the extraction principle by separating the management of old and new values through page mapping data structures. Instead of mixing old and new values in the same storage space, the system extracts and tracks the locations of old values using page mappings, allowing new values to be appended to storage spaces while old values are identified and removed through the mapping structure, thus maintaining data integrity while controlling storage space consumption
Solution Approach 2:
The patent implements discarding and recovering by using page mapping information to identify obsolete pages that can be safely discarded. The system tracks which pages contain old values through version information in the page mappings, allowing it to discard these pages and recover their storage space for new values, thereby maintaining data integrity (by preserving valid pages) while reducing overall storage space consumption
Data Source
AI summary
A key-value engine may perform garbage collection for a tree or hierarchical data structure on an append-only storage device with page mappings. The key-value engine may separate hot and cold data to reduce write amplification, track extent usage using a restricted or limited amount of memory, efficiently answer queries of valid extent usage, and adaptively or selectively defragment pages in snapshots in rounds of garbage collection.