Transparent Block Compression With Hash-Table Mapping for Scalable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The use of flat mapping tables for transparent block device compression in cloud-based block storage systems results in high memory costs and performance reduction due to the large size of the mapping table, which cannot fit in DRAM and requires frequent reads from storage devices, leading to scalability issues as storage space grows.
Innovation Solution
A transparent block device compression mapping scheme that uses multiple predetermined hash functions to determine mapping candidates for compressed data blocks, allowing the compressed data to be stored in multiple locations based on available storage capacity, with metadata tracking uncompressed block addresses and allocation information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a flat mapping table is used to map uncompressed block addresses to compressed block addresses, then the mapping can be maintained and tracked, but the mapping table becomes huge in size, demanding high memory cost and reducing performance
Solution Approach 1:
The patent divides the mapping system into two parts: a smaller mapping table that fits in DRAM and a hash table stored on the block storage device. The mapping table handles frequent mappings, while the hash table stores the complete mapping information for all uncompressed block addresses. This segmentation allows the system to maintain accurate mappings while reducing the memory footprint to a manageable size.
Solution Approach 2:
The patent introduces a hash table as an intermediary between the uncompressed block addresses and the compressed block addresses. The hash table, stored on the block storage device, acts as a mediator that resolves address mappings when the smaller mapping table in DRAM does not contain the required mapping. This intermediary approach enables the system to handle large storage spaces without requiring a proportionally large mapping table in memory.
2Reliability
If a flat mapping table is used to map uncompressed block addresses to compressed block addresses, then the mapping can be maintained, but frequent reads from storage devices are required, leading to performance reduction
Solution Approach 1:
The patent segments the mapping system into a hot cache (mapping table in DRAM) and a cold storage (hash table on block storage device). The mapping table in DRAM contains the most frequently accessed mappings, allowing the system to resolve the majority of address mappings without accessing the slower block storage device. Only when a mapping is not found in the DRAM cache does the system need to read from the block storage device, significantly reducing I/O operations and improving performance.
Solution Approach 2:
The system pre-loads frequently accessed mappings into the DRAM-based mapping table before they are actually needed. By anticipating which mappings will be required and loading them in advance into faster memory, the system minimizes the need for slow reads from the block storage device during actual data access operations, thereby improving overall performance.
3Reliability
If a flat mapping table is used for transparent block device compression, then mapping can be maintained, but scalability is reduced as storage space grows
Solution Approach 1:
The patent segments the mapping storage into a small DRAM-based mapping table and a large block storage-based hash table. As storage space grows, the hash table on the block storage device can expand to accommodate the increased number of mappings, while the DRAM mapping table maintains a manageable size. This segmentation allows the system to scale to large storage capacities without being constrained by the limited capacity of DRAM memory.
Solution Approach 2:
The patent moves part of the mapping storage from the memory dimension (DRAM) to the storage dimension (block storage device). This dimensional change allows the system to leverage the vast capacity of block storage for storing the complete mapping information, while using DRAM only for the most frequently accessed mappings. As a result, the system can scale to accommodate growing storage spaces without being bottlenecked by memory capacity limitations.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Systems and methods for accessing block storage devices are provided. In particular, a computing device may receive (204) a write request including an uncompressed data and an uncompressed block address associated with the uncompressed data, generate (206) compressed data by compressing the uncompressed data, determine (208) a plurality of mapping candidates of compressed data blocks in the block storage devices based on the uncompressed block address, select a compressed data block from the plurality of mapping candidates that has sufficient capacity to store the compressed data, write the compressed data to the selected compressed data block, update metadata of the selected compressed data block to link the uncompressed block address to a compressed block address of the selected compressed data block, and write the selected compressed data block back to a respective block storage device of the block storage devices.