Storage Compression Architecture for High Data Reduction and I/O Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices face a trade-off between data reduction effect and Input/Output (I/O) performance due to the varying efficiency of deduplication and compression methods, which depend on the size of the processing unit, leading to decreased I/O performance when combining these techniques.
Innovation Solution
A storage device architecture that includes a processor, an accelerator, and a memory, where the processor identifies data for compression and transmits instructions to the accelerator, which compresses continuous data items excluding those not to be compressed, optimizing both deduplication and compression processes without compromising I/O performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication and compression are combined to reduce data storage cost, then data reduction effect is improved, but I/O performance decreases due to large calculation load
Solution Approach 1:
The patent divides data processing into two independent segments: deduplication processing and compression processing. Each segment operates with its own optimized processing unit size (small for deduplication, large for compression) and can be executed independently or in parallel, reducing the overall calculation load on the CPU while maintaining high data reduction效果
Solution Approach 2:
The patent introduces an intermediary mechanism (processing unit information) that carries metadata about data segments requiring compression. This intermediary allows the system to identify and process only the necessary data portions without redundant operations, reducing calculation load while maintaining compression efficiency
2Loss of substance
If small processing unit is used for deduplication, then data reduction effect is improved, but operation complexity increases when combining with compression
Solution Approach 1:
The patent segments the data processing workflow into distinct phases with different processing unit sizes. Deduplication uses small processing units for fine-grained comparison, while compression uses large processing units for bulk encoding. This segmentation allows each phase to operate independently with optimized parameters, reducing overall operational complexity
Solution Approach 2:
The patent performs deduplication as a preliminary action before compression. By first identifying and removing duplicate data segments with small processing units, the system reduces the data volume that needs to be compressed subsequently. This preliminary action simplifies the overall operation by reducing the workload for the compression phase
Data Source
AI summary
The present invention realizes a storage device that has a high data reduction effect without decreasing I/O performances. The storage device includes a processor, an accelerator, a memory, and a storage medium, the processor specifies data to be compressed that is data stored in the storage medium from data stored in the memory and transmits a compression instruction including information relating to the data to be compressed to the accelerator, and the accelerator reads the plurality of continuous items of data from the memory and compresses the plurality of items of data to be compressed obtained by excluding data that is not to be compressed from the plurality of items of data, based on the information relating to the data to be compressed received from the processor, to generate compressed data stored in the storage device.