Collision-Free Hash Vector for Object Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems are inefficient in managing duplicate data, leading to excessive storage requirements as they fail to effectively deduplicate data segments across multiple versions of files, especially in scenarios where versions have minimal differences.
Innovation Solution
A data management device employing a collision-free hash function generates a hash vector to deduplicate segments stored in object storage, identifying and removing duplicates while updating file recipes to reduce storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional storage systems store multiple versions of files with minimal differences, then storage capacity is consumed for duplicate data, but data versioning and retrieval capabilities are maintained
Solution Approach 1:
The patent divides files into fixed-size segments (e.g., 4KB) and creates a segment tree structure where each node represents a segment. This segmentation enables independent deduplication of individual segments rather than storing entire file versions, significantly reducing storage space while maintaining versioning capabilities through segment-level references.
Solution Approach 2:
The patent uses hash-based copying where segment hashes are computed and stored in a hash vector. Instead of storing duplicate segment data, the system creates references (copies of hash values) to unique segments. When a file version is retrieved, the system reconstructs it by assembling references to existing segments, eliminating redundant data storage while preserving version information.
2Quantity of substance
If data deduplication is implemented using traditional hash functions, then storage space is reduced, but hash collisions occur leading to imperfect deduplication
Solution Approach 1:
The patent transitions from traditional single-dimension hash comparison to a multi-dimensional approach using segment trees. Each segment is represented as a node in a tree structure with hierarchical relationships. This dimensional transformation allows the system to uniquely identify segments through their tree path and position, eliminating hash collisions while maintaining efficient storage through the hierarchical structure.
Solution Approach 2:
The patent pre-computes and stores hash values for all segments in a hash vector before actual deduplication operations. This preliminary hashing action creates a reference map that enables O(1) lookup time during deduplication. By performing the hashing action in advance and storing the results in a structured hash vector, the system achieves both speed and accuracy without collisions during the deduplication process.
3Reliability
If all data segments are retained for perfect deduplication, then deduplication accuracy is improved, but storage overhead and processing complexity increase
Solution Approach 1:
The patent segments the data space into a tree structure where each node represents a manageable unit. This segmentation transforms the complex problem of managing all possible data segments into a hierarchical structure with clear parent-child relationships. The segment tree allows the system to track and manage segments efficiently through their hierarchical paths, reducing the complexity of maintaining deduplication accuracy across the entire data set.
Solution Approach 2:
The system pre-establishes the segment tree structure and pre-computes hash values for all segments before deduplication operations begin. This preliminary structuring action creates a ready-made framework that simplifies subsequent deduplication operations. The pre-computed hash vector serves as a lookup table that eliminates the need for complex real-time comparisons, reducing processing complexity while maintaining perfect deduplication accuracy.
Data Source
AI summary
A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor generates a collision free hash function based on segments stored in the object storage. The processor generates a hash vector using the collision free hash function. The processor deduplicates the segments using the hash vector. The processor stores the deduplicated segments in the object storage.