Object Fingerprinting via Block Hash Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hashing algorithms for generating object fingerprints in data storage systems either produce less unique hash values, impacting data deduplication efficiency, or require significant processing resources, leading to performance issues.
Innovation Solution
A data storage system architecture that divides objects into blocks, generates hash values for each block in a storage subsystem, and combines these hash values to create a robust object hash, reducing the need for repeated hashing of entire objects and optimizing processing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If faster hash algorithms are used to generate object hashes, then processing speed is improved, but hash uniqueness deteriorates
Solution Approach 1:
The object is divided into multiple blocks, and each block is hashed independently using fast hash algorithms. The individual block hashes are then combined to form the final object hash. This segmentation allows the use of faster algorithms on smaller data units while maintaining overall hash uniqueness through the combination process.
Solution Approach 2:
The final object hash is constructed as a composite of multiple individual block hash values. By combining multiple hash results from different blocks through concatenation or other combination methods, the system achieves both speed (using fast algorithms on blocks) and uniqueness (through the composite structure of multiple hashes).
2Reliability
If slower hash algorithms are used to generate object hashes, then hash uniqueness is improved, but processing resources increase
Solution Approach 1:
The object is divided into multiple blocks, and each block is hashed independently using fast hash algorithms. The individual block hashes are then combined to form the final object hash. This segmentation allows the use of faster algorithms on smaller data units while maintaining overall hash uniqueness through the combination process.
Solution Approach 2:
Instead of applying a slow, resource-intensive hash algorithm to the entire object, the system applies fast hash algorithms to only partial portions (blocks) of the object. This partial action approach significantly reduces processing resources while achieving sufficient uniqueness through the combination of multiple block hashes.
3Reliability
If entire object content is hashed by a lone device, then hash completeness is ensured, but processing time increases
Solution Approach 1:
The object is divided into multiple blocks that can be hashed in parallel by different processing units or devices. This segmentation enables concurrent processing of block hashes, significantly reducing total processing time while maintaining hash completeness through the combination of all block hashes.
Solution Approach 2:
Multiple individual block hash results are merged or combined to form the final object hash. This combining process ensures that the complete object content is represented in the final hash while the parallel processing of individual blocks reduces overall processing time.
Data Source
AI summary
An embodiment computing device operating in a data storage system includes an object storage controller operable to divide an object into blocks and to create an object hash from hash values, and a network interface in communication with the object storage controller, the network interface operable to transmit the blocks to a storage subsystem that generates one of the hash values from each of the blocks, to receive the hash values from the storage subsystem, and to provide the hash values to the object storage controller for creation of the object hash from the hash values. In an embodiment, the object storage controller is operably coupled to a processor and a memory or stored on a computer readable medium.