Data Chunk Similarity Search Using Fingerprint Trees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in finding and compressing similar data chunks due to resource-intensive one-to-one similarity checks, especially when dealing with vast numbers of data chunks.
Innovation Solution
A method involving the use of similarity hashes to generate fingerprints for data chunks, which are then catalogued in trees ordered by predefined symbol sets, allowing for efficient comparison and compression by scanning tree paths and calculating deltas between similar chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If one-to-one similarity checks are performed between received data chunk and all stored data chunks, then compression accuracy is improved, but computational complexity and resource consumption increase significantly
Solution Approach 1:
The patent segments the large-scale similarity search problem into two stages: first, it divides stored data chunks into multiple groups using clustering algorithms based on their fingerprints; second, it performs similarity checks only within relevant groups rather than against all chunks. This segmentation reduces the search space from O(N) to O(k) where k << N, thereby resolving the contradiction between comprehensive similarity checking and search complexity.
Solution Approach 2:
The patent introduces clustering results as an intermediary structure between data storage and similarity search. By pre-organizing chunks into clusters based on fingerprint similarity, the system creates an intermediate index that enables efficient navigation to potentially similar chunks without exhaustive comparison. This intermediary clustering structure allows accurate similarity detection while avoiding the computational burden of one-to-one checks across all chunks.
2Quantity of substance
If vast numbers of data chunks are stored in the storage system, then storage capacity is improved, but memory consumption and computational resources for similarity search increase
Solution Approach 1:
The patent applies preliminary action by computing fingerprints and performing clustering on stored data chunks in advance, before similarity search is needed. These preprocessing steps create a organized structure (clusters with representative fingerprints) that remains static or updates incrementally. When a new chunk arrives, the system only needs to compare its fingerprint against cluster representatives rather than all individual chunks, dramatically reducing real-time computational resource consumption while maintaining the ability to handle vast numbers of stored chunks.
3Productivity
If compression is performed based on similar stored data chunks, then storage efficiency is improved, but the complexity of finding similar chunks increases
Solution Approach 1:
The patent segments the similarity search process into two phases: a preprocessing phase that creates clustering structures from stored chunks, and a query phase that uses these structures to efficiently find similar chunks for compression. By pre-organizing data into clusters with representative fingerprints, the system enables compression-based storage efficiency without requiring complex real-time searches across all stored data, thus resolving the contradiction between storage efficiency and search complexity.
Data Source
AI summary
A method for storing a received data chunk (DC) in a storage system, the method includes (a) obtaining a received fingerprint of the received DC, the received fingerprint may include received fingerprint elements that are indicative of occurrences, within the received DC, of content elements, the received fingerprint elements are ordered according to a given order; (b) searching, within a tree, for a similar stored fingerprint; the tree may include tree nodes that represent multiple stored fingerprints of stored data chunks that are stored in the storage system; different levels of the tree are allocated to different content elements; (c) compressing, when finding the similar stored fingerprint, the received DC based on a similar DC associated with the similar stored fingerprint, and updating storage system metadata to indicate that the received DC is stored in the storage system in a compressed form, and based on the similar stored DC.