Storage Data Reduction Using Accelerator-Based Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices face challenges in achieving high data reduction effects while maintaining high Input/Output (I/O) performances, as deduplication and compression techniques require different processing unit sizes, leading to decreased I/O performance due to processor operations needed for data removal during deduplication.
Innovation Solution
A storage device configuration that includes a processor and an accelerator, where the processor identifies data for compression and transmits instructions to the accelerator, which compresses and decompresses data based on deduplication and compression algorithms, optimizing processing units for each operation to enhance data reduction without impacting I/O performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication and compression are combined to reduce data storage cost, then data reduction effect is improved, but I/O performance decreases due to large calculation load
Solution Approach 1:
The storage device divides data into multiple data strings and processes them in parallel using multiple threads. The processor allocates different data strings to different threads, enabling simultaneous deduplication and compression operations without overwhelming the system's calculation capacity, thus maintaining I/O performance while achieving data reduction
Solution Approach 2:
The system combines deduplication and compression techniques into a composite data reduction approach. Deduplication removes duplicate data blocks while compression encodes remaining unique data, creating a multi-layered reduction strategy that achieves superior data storage cost reduction compared to either technique alone
2Loss of substance
If small processing unit is used for deduplication, then data reduction effect is improved, but processor operation load increases
Solution Approach 1:
The system segments the deduplication process into fine-grained operations on small data blocks while distributing these operations across multiple threads. Each thread handles small processing units independently, achieving high data reduction effect through thorough duplicate detection without concentrating excessive processing load on a single processor core
Solution Approach 2:
The system implements self-service mechanisms where processed data is immediately stored in a data cache, and duplicate detection leverages previously processed data without requiring repeated full-scans. This reduces redundant processor operations while maintaining comprehensive deduplication effectiveness
Data Source
AI summary
The present invention realizes a storage device that has a high data reduction effect without decreasing I/O performances. The storage device includes a processor, an accelerator, a memory, and a storage medium, the processor specifies data to be compressed that is data stored in the storage medium from data stored in the memory and transmits a compression instruction including information relating to the data to be compressed to the accelerator, and the accelerator reads the plurality of continuous items of data from the memory and compresses the plurality of items of data to be compressed obtained by excluding data that is not to be compressed from the plurality of items of data, based on the information relating to the data to be compressed received from the processor, to generate compressed data stored in the storage device.