Multi-Scale Macro-Fingerprint Navigation for Data De-Duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional de-duplication techniques, such as hash fingerprinting and pattern recognition, face limitations due to memory constraints and random I/O workloads, making them inefficient for large-scale data storage systems.
Innovation Solution
The method generates macro-fingerprints from micro-fingerprints without re-reading underlying data, using a prioritization scheme to select and store summary values, and navigates data hierarchies to identify and remove redundant data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hash fingerprinting is used for de-duplication, then data redundancy can be identified and removed, but memory constraints limit the scalability of the system
Solution Approach 1:
The patent segments the data into fixed-size chunks and processes them independently through hash fingerprinting. Each chunk is hashed and compared against the fingerprint table, allowing the system to handle large data volumes by processing them in manageable segments rather than requiring all data to fit in memory simultaneously.
Solution Approach 2:
The patent introduces a hierarchical dimension to the de-duplication process by implementing multi-scale navigation. It creates fingerprints at multiple granularities (fine-grained chunk level and coarse-grained block level), allowing the system to navigate through data at different scales and reduce the memory footprint required for fingerprint storage while maintaining de-duplication effectiveness.
2Quantity of substance
If pattern recognition is used for de-duplication, then large data blocks can be processed, but random I/O workload increases substantially
Solution Approach 1:
The patent performs preliminary hashing of data chunks into fingerprints before the actual de-duplication comparison. By pre-computing and storing these fingerprints in a table, the system avoids repeated I/O operations during the comparison phase, as the hash lookup is a fast in-memory operation that eliminates the need for repeated data retrieval.
Solution Approach 2:
The patent introduces fingerprints as an intermediary representation between the original data and the de-duplication comparison process. Instead of directly comparing large data blocks (which would require substantial I/O), the system compares compact hash fingerprints that serve as proxies for the actual data, dramatically reducing I/O workload while maintaining accurate redundancy detection.
3Reliability
If conventional de-duplication methods are used, then redundant data can be identified, but the process is complex and requires substantial computational resources
Solution Approach 1:
The patent replaces complex mechanical comparison processes with hash-based fingerprinting. Instead of performing byte-by-byte comparison of data blocks (a mechanically intensive process), the system uses cryptographic hash functions to generate compact fingerprints that can be compared efficiently. This substitution dramatically reduces computational complexity while maintaining high accuracy in redundancy detection.
Solution Approach 2:
The patent changes the parameter representation from raw data bytes to hash digest values. By transforming the data into a different parameter space (hash space), the system enables efficient comparison operations. The multi-scale navigation further changes the granularity parameter, allowing comparisons at both fine-grained (chunk) and coarse-grained (block) levels, optimizing the balance between accuracy and computational efficiency.
Data Source
AI summary
A system configured to generate a macro-fingerprint from at least one predefined set of summaries is provided. The system includes data storage storing a first predefined set of summaries associated with a first region of data, each member of the first predefined set of summaries characterizing data within the first region of data; and at least one processor coupled to the data storage and configured to: read the first predefined set of summaries; select at least one first member from the first predefined set of summaries based on a value of the at least one first member; and store the at least one first member within a first macro-fingerprint. The first region of data may have a first size indicative of a quantity of data included in the first region of data. The macro fingerprints are created from previously created smaller (micro) fingerprints without having to reread the data.