Symbol Pair Encoding for Higher SSD Compression Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional two-stage data compression algorithms, such as those using LZ variants and Huffman coding, often fail to completely remove inter-symbol redundancies, leading to suboptimal compression efficiency, especially in flash-based storage devices like SSDs, where write amplification and storage space utilization are critical concerns.
Innovation Solution
Implementing a two-stage compression algorithm that calculates the frequencies of occurrence of symbols and symbol pairs in the output of the first stage, and encodes redundant symbol pairs as 'super symbols' in the second stage using Huffman coding, allowing for additional compression efficiency gains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional two-stage compression algorithms (LZ variants followed by Huffman coding) are used, then basic compression is achieved, but inter-symbol redundancies remain causing suboptimal compression efficiency
Solution Approach 1:
The patent segments the compression process into three distinct stages: first stage compression (LZ variant), second stage compression (Huffman coding with extended alphabet), and third stage compression (handling symbol pairs). This segmentation allows each stage to target specific types of redundancies, with the second and third stages specifically addressing inter-symbol redundancies that the first stage misses, thereby improving overall compression efficiency and reducing write amplification.
Solution Approach 2:
The patent transitions from single-symbol encoding to pair-symbol encoding in the third compression stage. By treating symbol pairs as extended alphabet elements in the second stage and further encoding their frequencies in the third stage, the system adds a dimensional layer to the compression process. This dimensional change enables capturing correlations between adjacent symbols that single-symbol approaches cannot exploit, thus improving compression efficiency.
2Quantity of substance
If conventional two-stage compression algorithms are used, then processing speed is maintained, but storage space utilization is suboptimal due to remaining redundancies
Solution Approach 1:
The patent performs preliminary frequency calculations for symbol pairs during the second compression stage, preparing an extended Huffman code table that includes pair symbols. This preliminary action of pre-calculating and pre-organizing the extended alphabet and its frequencies enables the third stage to efficiently encode pair correlations without adding significant runtime complexity, thus improving storage space utilization while managing algorithm complexity.
Solution Approach 2:
The patent introduces an intermediate extended alphabet in the second compression stage that serves as a mediator between the first stage output and the final compression. This intermediate representation includes both original symbols and symbol pairs, allowing the system to gradually incorporate pair correlations without directly complicating the final encoding stage. The intermediate alphabet acts as a bridge that systematically handles the increased complexity.
Data Source
AI summary
Methods, apparatuses, and computer-readable media for compressing data for storage or transmission. Input data is compressed in a first stage utilizing a first compression algorithm and the frequencies of occurrence of symbols and symbol pairs in the output from the first stage is calculated. The output from the first stage is then encoded to a final compressed bit string in a second stage utilizing a second compression algorithm based on the calculated frequencies of occurrence of the symbols and the symbol pairs.