Data Fingerprint Strengthening via Composite Hash Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in deduplication processes, particularly in identifying and managing duplicate data chunks across network-accessible storage systems, leading to suboptimal storage utilization and resource allocation.
Innovation Solution
The implementation of a fingerprinting mechanism that uses a combination of hash values and intermediate values to create stronger data chunk fingerprints, allowing for efficient deduplication by parsing data streams into chunks, computing fingerprints, and comparing them with existing data on remote servers to identify and eliminate duplicates, thereby reducing storage needs and network bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard hash functions are used for data chunk fingerprinting, then the deduplication process is simple and fast, but fingerprint collisions occur more frequently reducing deduplication accuracy
Solution Approach 1:
The patent segments the fingerprinting process into multiple components: a primary hash value and multiple intermediate hash values. This segmentation allows the system to maintain simplicity while improving accuracy by combining multiple hash components into a composite fingerprint that reduces collisions.
Solution Approach 2:
The patent creates a composite fingerprint structure by combining a primary hash value with multiple intermediate hash values. This composite approach strengthens the overall fingerprint accuracy without requiring a completely new hashing algorithm, thus improving measurement precision while controlling complexity.
2Reliability
If more intermediate hash values are included in fingerprints, then fingerprint collision rate decreases, but computing resources consumed increases
Solution Approach 1:
The patent applies partial action by selectively including only necessary intermediate hash values in the fingerprint rather than computing all possible intermediate values. This approach achieves sufficient deduplication reliability while avoiding excessive computing resource consumption.
Solution Approach 2:
The patent performs preliminary computation of intermediate hash values during the initial hashing process, so these values are readily available when needed for fingerprinting. This preliminary action reduces redundant computations and optimizes resource usage during the actual deduplication operation.
3Measurement precision
If data is parsed into smaller chunks, then deduplication precision improves, but the number of fingerprints to process increases
Solution Approach 1:
The patent segments data into chunks for precise deduplication while using an efficient fingerprinting mechanism that processes multiple chunks in parallel. The segmentation of the fingerprint into primary and intermediate hash values enables parallel computation, maintaining productivity despite increased chunk granularity.
Solution Approach 2:
The patent adds dimensional efficiency by organizing the fingerprinting process into hierarchical levels (primary hash, intermediate hashes, final fingerprint). This dimensional structure allows for optimized processing pipelines that maintain throughput while handling increased numbers of smaller chunks.
Data Source
AI summary
Techniques and mechanisms described herein facilitate the strengthening of a data fingerprint. According to various embodiments, a data stream may be parsed into a plurality of data chunks. A hash function may be applied to each of the data chunks to determine a respective data chunk hash value. A respective data chunk fingerprint for each of the plurality of data chunks may be transmitted to a remote server. Each data chunk fingerprint may include the respective data chunk hash value and one or more respective intermediate hash values.