Fixed-Length Compression Mapping for Lower Storage Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques in distributed storage systems suffer from inefficiencies due to high read and write amplifications, leading to increased latency, resource consumption, and reduced storage media lifespan, as they generate irregular-sized compression results that require additional data-intensive operations and result in suboptimal usage of storage resources.
Innovation Solution
A system and method that compresses data by applying a sliding window to input segments, aligning the compressed data with physical block addresses to generate fixed-length segments, and uses a mapping table to efficiently read and write data, reducing read/write amplification and optimizing storage resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing data compression techniques are applied, then storage capacity is saved and data transfer is reduced, but read/write amplification increases leading to higher latency and resource consumption
Solution Approach 1:
The patent segments input data into fixed-size input blocks that are processed independently through the compression engine. Each input block is compressed to produce a fixed-size output block, creating a segmented processing pipeline that avoids the need to buffer and reprocess data, thereby reducing write amplification while maintaining storage efficiency.
Solution Approach 2:
The patent changes the parameter of output block size from variable to fixed. By ensuring that compression always produces output blocks of a predetermined size matching the storage medium's block size, the system eliminates the need for additional read/write operations to handle variable-sized outputs, thus reducing read/write amplification while preserving storage capacity benefits.
2Productivity
If compression techniques process variable-sized output blocks, then compression efficiency is achieved, but additional data-intensive operations are required
Solution Approach 1:
The patent performs preliminary action by pre-defining fixed input and output block sizes before compression processing begins. This preliminary structuring allows the compression engine to operate in a streamlined manner where each input block directly maps to an output block of known size, eliminating the need for complex post-compression operations to handle variable-sized outputs.
Solution Approach 2:
The patent introduces dynamics by using a sliding window that moves through the input data stream in fixed-size increments. The sliding window dynamically processes input blocks sequentially, with each window position representing a complete input block that transforms into a complete output block, creating a dynamic yet predictable processing rhythm that reduces operational complexity.
3Quantity of substance
If irregular-sized compression results are generated, then storage capacity is optimized, but alignment with physical block addresses becomes suboptimal
Solution Approach 1:
The patent changes the parameter of output block size from variable to fixed, where the fixed size matches the storage medium's physical block size. This parameter change ensures that compressed output blocks align perfectly with physical block addresses, eliminating misalignment issues while maintaining efficient storage capacity utilization through consistent compression ratios.
4Productivity
If high read/write amplification occurs, then compression processing is performed, but latency and resource consumption increase
Solution Approach 1:
The patent segments the compression process into independent fixed-size block operations that can be processed in parallel or pipelined efficiently. Each input block is compressed independently to a fixed-size output block, allowing the system to minimize buffering requirements and reduce the time data spends in the compression pipeline, thereby reducing latency while maintaining processing throughput.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining a continuous flow of fixed-size input blocks through the compression engine to produce continuous fixed-size output blocks. This continuous processing eliminates idle time and buffering delays, keeping the compression pipeline full and productive, which reduces overall latency while sustaining high compression processing rates.
Data Source
AI summary
The system can receive data to be written to a non-volatile memory in the distributed storage system. The received data can include a plurality of input segments. The system can assign consecutive logical block addresses (LBAs) to the plurality of input segments. The system can then compress the plurality of input segments to generate a plurality of fixed-length compressed segments, with each fixed-length compressed segment aligned with a physical block address (PBA) in a set of PBAs. The system compresses the plurality of input segments to enable an efficient use of storage capacity in the non-volatile memory. Next, the system can write the plurality of fixed-length compressed segments to a corresponding set of PBAs in the non-volatile memory. The system can then create, in a data structure, a set of entries which map the LBAs of the input segments to the set of PBAs. This data structure can be used later by the system when processing a read request including a LBA.