Flat Hash Table Mapping for Large-Scale Block Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing block storage systems face challenges with memory usage and performance due to large mapping tables that either cannot fit in memory or require frequent disk access, leading to latency and low throughput, especially in large-scale cloud-based storage environments.
Innovation Solution
A hybrid architecture using a flat hash table with multiple hash functions and a usage bitmap to efficiently map uncompressed to compressed block addresses, reducing memory requirements and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large mapping table is used to track uncompressed to compressed block addresses, then mapping accuracy is improved, but memory usage increases and the table cannot fit in memory
Solution Approach 1:
The patent divides the mapping table into multiple segments or partitions, each stored in separate memory regions. This segmentation allows the large mapping table to be distributed across available memory resources, preventing any single memory module from being overwhelmed while maintaining complete mapping coverage.
Solution Approach 2:
The patent introduces a hierarchical dimension to the mapping table structure, organizing mappings across multiple levels or tiers. This dimensional transformation enables efficient memory utilization by allowing frequently accessed mappings to reside in faster memory while less frequently accessed mappings are stored in slower, larger-capacity storage.
2Speed
If a large mapping table is stored in memory, then mapping speed is improved, but memory capacity requirements increase
Solution Approach 1:
The patent applies local quality optimization by identifying and prioritizing frequently accessed mapping entries, then allocating them to faster memory regions. Less frequently accessed mappings are placed in slower storage, creating a differentiated memory hierarchy that optimizes overall mapping speed while reducing peak memory capacity requirements.
3Quantity of substance
If the mapping table does not fit in memory, then memory usage is reduced, but disk access frequency increases causing latency
Solution Approach 1:
The patent implements preliminary action by pre-loading anticipated mapping entries into memory before they are actually needed. This predictive pre-loading strategy reduces the frequency of disk accesses during normal operation, thereby reducing latency while maintaining a manageable memory footprint.
4Device complexity
If existing mapping schemes are used, then implementation simplicity is maintained, but throughput decreases due to frequent disk access
Solution Approach 1:
The patent creates a universal mapping table architecture that can adapt to different workloads and storage configurations. This multi-functional design allows the system to optimize for either speed or memory efficiency depending on conditions, maintaining implementation simplicity while improving throughput through automated adaptation rather than complex manual configuration.
Data Source
AI summary
A flat hash table includes a plurality of entries, and each entry includes a hash function index and a usage bitmap. A method for block device level compression mapping using the flat hash table includes compressing uncompressed data to compressed data, retrieving an entry of the flat hash table using an uncompressed block address of the uncompressed data, determining a compressed block address of the compressed data by executing at least one hash function and by determining a hash function in the at least one hash function for mapping the uncompressed block address to the compressed block address that corresponds to a space in a block storage device, storing the compressed data to the space that corresponds to the compressed block address, and updating the hash function index of the entry of the flat hash table with an index indicative of the hash function.