Storage System Two-Stage Compression for IO Speed and Capacity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression technologies in storage systems face challenges in achieving high compression rates while meeting IO performance requirements and maintaining a manageable circuit scale for data compression.
Innovation Solution
The proposed solution involves a storage system with a controller that performs in-line compression using a data compression/decompression circuit. This system executes a two-stage compression process: first, it applies a standard dictionary compression, followed by a second stage where the compressed data is further processed using a more advanced dictionary compression with enhanced character string search capability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If in-line compression is performed using a hardware circuit (accelerator) to meet IO performance requirements, then data compression speed is improved, but the compression rate is limited due to circuit scale constraints
Solution Approach 1:
The compression process is divided into two distinct stages: in-line compression performed by a hardware circuit (accelerator) for high-speed processing, and post-compression performed by a processor (CPU) for high compression rate. This segmentation allows each component to optimize for its specific function without being constrained by the other's limitations.
Solution Approach 2:
The in-line compression is performed as a preliminary action before data is stored, providing immediate compression benefit while maintaining IO performance. The post-compression then further improves the compression rate on already-stored data without affecting the original write performance.
2Loss of substance
If the logic scale of the FPGA accelerator is increased to improve compression rate, then compression performance is improved, but the device complexity and circuit scale increase
Solution Approach 1:
The complex dictionary compression function that would require large FPGA logic scale is extracted from the hardware circuit and implemented as software on the processor. This allows the processor to handle the computationally intensive dictionary compression while the simple hardware circuit focuses only on high-speed range encoding and basic compression operations.
Solution Approach 2:
The patent replaces complex hardware circuit logic with software-based dictionary compression on the processor. This substitution allows for more flexible and powerful compression algorithms without being constrained by hardware logic scale, while the hardware circuit handles the speed-critical operations.
3Loss of substance
If post-compression is performed using a processor with enhanced dictionary compression, then compression rate is improved, but the processing time increases
Solution Approach 1:
The post-compression is performed periodically on data that has already been compressed and stored using in-line compression. This periodic processing allows the system to maintain high IO performance for new data while gradually improving the compression rate of stored data over time.
Solution Approach 2:
The patent uses the same LZMA compression/decompression circuit for both in-line compression and for decompression during post-compression. This copying of the compression engine allows for efficient processing while maintaining consistency in the compression algorithm implementation.
Data Source
AI summary
The controller performs in-line compression of plaintext data from a host and post-process compression of in-line compressed data stored in one or more storage drives. In the in-line compression, in-line compressed data is generated by executing a compression process including first dictionary compression on the plaintext data from the host, and the in-line compressed data is stored in the one or more storage drives. In the post-process compression, the plaintext data is generated by decompressing the in-line compressed data read from the one or more storage drives. In the post-process compression, post-process compressed data is generated by executing a compression process including second dictionary compression on the plaintext data using the processor, the second dictionary compression being more excellent in character string search capability than the first dictionary compression, and the post-process compressed data is stored in the one or more storage drives.