Transparent Block Compression Using Hash-Table Mapping to Reduce DRAM Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The use of flat mapping tables for transparent block device compression in cloud-based block storage systems results in high memory costs and performance reduction due to the large size of the mapping table, which cannot fit in DRAM and requires frequent reads from storage devices, leading to inefficient data access.
Innovation Solution
A transparent block device compression mapping scheme that uses multiple predetermined hash functions to determine mapping candidates for compressed data blocks, allowing efficient storage of compressed data while maintaining metadata to track uncompressed block addresses, and handles overflow by distributing data across multiple blocks if necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a flat mapping table is used to map uncompressed block addresses to compressed block addresses, then the mapping can be maintained and tracked, but the mapping table becomes huge in size, demanding high memory cost and reduction in performance
Solution Approach 1:
The patent divides the mapping table into multiple segments or levels. Instead of using a single flat mapping table that requires storing all possible uncompressed block address mappings, the system segments the mapping into hierarchical levels where only frequently accessed mappings are stored in memory. This segmentation reduces the memory footprint while maintaining complete mapping coverage through multiple levels of indirection.
Solution Approach 2:
The patent transitions from a one-dimensional flat mapping table to a multi-dimensional hierarchical mapping structure. By introducing additional dimensions such as level indicators, segment identifiers, and indirect indexing mechanisms, the system achieves comprehensive mapping coverage without requiring proportional memory increases. This dimensional transformation allows sparse population of mapping entries while maintaining full address space coverage.
2Reliability
If a flat mapping table is used to map uncompressed block addresses to compressed block addresses, then the mapping can be maintained and tracked, but frequent reads from storage devices are required, leading to performance reduction
Solution Approach 1:
The patent implements preliminary action by pre-loading frequently accessed mapping entries into high-speed cache memory before they are actually needed. The hierarchical structure allows the system to anticipate and prepare mapping data at higher levels of the hierarchy, reducing the need for frequent storage device reads. Hot mappings are cached in advance, while cold mappings remain in lower-level storage.
Solution Approach 2:
The patent introduces intermediary caching layers between the storage devices and the processing units. These intermediary caches store recently accessed mapping entries, acting as mediators that satisfy mapping requests without requiring direct reads from slow storage devices. This intermediary layer significantly reduces storage device access frequency while maintaining mapping accuracy.
3Quantity of substance
If compression is applied to reduce data size, then storage efficiency improves, but the mapping between uncompressed and compressed addresses becomes non-linear and more complex
Solution Approach 1:
The patent segments the compression mapping into manageable hierarchical levels, where each level handles a specific range or portion of the address space. This segmentation transforms the complex non-linear mapping into a series of simpler, localized mappings that can be processed independently. Each segment maintains its own mapping rules, reducing the overall complexity while preserving storage efficiency benefits.
Solution Approach 2:
The patent introduces intermediary data structures and translation layers that simplify the non-linear compression mapping. These intermediaries act as buffer layers that absorb the complexity of non-linear transformations, presenting simplified linear or near-linear mappings to upper-level processes. The intermediary structures handle the complex address translations internally while maintaining storage efficiency.
Data Source
AI summary
Systems and methods for accessing block storage devices are provided. In particular, a computing device may receive a write request including an uncompressed data and an uncompressed block address associated with the uncompressed data, generate compressed data by compressing the uncompressed data, determine a plurality of mapping candidates of compressed data blocks in the block storage devices based on the uncompressed block address, select a compressed data block from the plurality of mapping candidates that has sufficient capacity to store the compressed data, write the compressed data to the selected compressed data block, update metadata of the selected compressed data block to link the uncompressed block address to a compressed block address of the selected compressed data block, and write the selected compressed data block back to a respective block storage device of the block storage devices.