Data Deduplication Device Using Fingerprint Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data deduplication techniques in block chain systems require high processing loads due to the need for hash calculations, which can decrease the performance of tiering and backup operations.
Innovation Solution
A data deduplication device that acquires hash values of data groups, generates fingerprints, and compares them to stored fingerprints to determine duplication, thereby reducing the need for hash calculations and processing load by selectively using message digests for deduplication determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hash calculation is performed on the server to eliminate duplication, then deduplication accuracy is improved, but processing load increases and performance decreases
Solution Approach 1:
The patent introduces a storage system as an intermediary that performs hash calculations and deduplication processing instead of the server. The storage system receives data from the server, calculates hashes, compares them with stored fingerprints, and determines duplication. This mediator approach transfers the computational burden from the server to the storage system, reducing server processing load while maintaining deduplication accuracy.
Solution Approach 2:
The patent replaces the server-based hash calculation mechanism with a storage system-based hash calculation mechanism. Instead of the server performing computationally intensive hash operations, the storage system executes these operations locally. This substitution changes which component performs the mechanical hash calculation work, thereby improving overall system performance while preserving deduplication functionality.
2Device complexity
If the size of the chunk is increased to several kilobytes or more, then the number of chunks is reduced, but the time and cost for comparing mutual chunks for duplication determination increase
Solution Approach 1:
The patent extracts the hash calculation and comparison operation from the chunk data itself, using only the hash value (fingerprint) for duplication determination. Instead of comparing entire chunks of several kilobytes, the system calculates hashes and compares these compact representations. This extraction of the essential identifying feature (hash) from the full data enables efficient comparison while maintaining accurate duplication detection.
Solution Approach 2:
The patent changes the parameter used for comparison from the full chunk data (several kilobytes) to a hash value (fixed-length fingerprint). This parameter transformation converts a large, variable-size data structure into a compact, fixed-size representation that is much faster and cheaper to compare, while still providing accurate duplication determination through the properties of cryptographic hash functions.
Data Source
AI summary
A data deduplication device reduces a processing load in deduplication. Storage target data includes a content including a plurality of blocks having a structure in which transaction data and a hash value of a preceding block are associated with each other. A storage includes a storage device and a processor, which (1) acquires a hash value associated with one or more blocks of a chunk including the block in the content, and specifies a fingerprint corresponding to the chunk based on the acquired one or more hash values of the block, (2) determines whether the fingerprint corresponding to the chunk is the same as a fingerprint of a chunk stored in the storage device, and (3) does not store the chunk in the storage device when it is determined to be the same, and stores the chunk in the storage device when it is determined to not be the same.