Multilevel Hash Index Hardening with Dynamic Intermediate Buckets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in amortizing write operations to on-drive storage, which are costly in terms of drive wear and time, due to the frequency and impact of metadata writes relative to user data writes.
Innovation Solution
Implementing a multilevel hash table structure with an in-memory hash table level, an intermediate on-drive hash table level, and a final on-drive hash table level, where the intermediate level dynamically expands in size according to an arithmetic series, distributing the write operations across multiple levels to optimize metadata writes with user data writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If dirty index entries are hardened directly from in-memory hash table level to on-drive hash table level, then the simplicity of the system is maintained, but the amortization of write operations is insufficient leading to excessive drive wear and time consumption
Solution Approach 1:
The patent segments the hash table structure into three distinct levels: in-memory hash table level, intermediate on-drive hash table level, and final on-drive hash table level. This segmentation allows write operations to be distributed across multiple levels, with the intermediate level serving as a buffer that accumulates dirty index entries before they are hardened to the final level, thereby amortizing the expensive on-drive write operations.
Solution Approach 2:
The intermediate on-drive hash table level acts as an intermediary between the in-memory and final on-drive hash table levels. It temporarily stores dirty index entries that need to be hardened, allowing multiple small writes to be batched and consolidated before being written to the final level, thus reducing the frequency and impact of expensive metadata writes to persistent storage.
2Productivity
If the intermediate on-drive hash table level is made larger to improve amortization, then write operation frequency is reduced, but the device complexity and initial storage requirements increase
Solution Approach 1:
The intermediate on-drive hash table level is implemented with dynamic sizing capability. The number of buckets at this level can be adjusted based on system conditions and workload characteristics, allowing the system to optimize amortization efficiency while adapting to changing storage requirements and avoiding excessive complexity for fixed workloads.
Solution Approach 2:
The patent allows modification of key parameters including the number of buckets at each hash table level, the fullness threshold for triggering hardening operations, and the arithmetic series expansion factors. These parameter changes enable the system to balance between amortization efficiency and device complexity based on specific operational requirements.
3Loss of time
If the number of buckets at intermediate level is increased according to arithmetic series, then the number of write operations is reduced, but the storage space requirements and system complexity increase
Solution Approach 1:
The intermediate on-drive hash table level pre-allocates storage space and structures buckets in advance according to an arithmetic series progression. This preliminary action allows the system to batch multiple dirty index entries before hardening them to the final level, reducing the frequency of write operations and the time spent on hardening operations, while the space overhead is managed through the structured expansion pattern.
Data Source
AI summary
Techniques for improving amortization when hardening index entries across a multilevel hash table. The techniques include, in a first hardening cycle, in response to a first group of index entries filling a bucket at an in-memory hash table level L1, hardening the bucket at L1 to an initial bucket at an intermediate on-drive hash table level L2. The techniques include, in subsequent successive hardening cycles, incrementally increasing the number of buckets at L2 to a final number of buckets according to an arithmetic series, and, in response to a next group of index entries up to a last group of index entries filling the bucket at L1, hardening the bucket at L1 across the incrementally increased number of buckets at L2. The techniques include, in response to the final number of buckets at L2 being filled, hardening the buckets at L2 across buckets at an on-drive hash table level L3.