Hierarchical Index File for Data De-duplication Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data quantity increases, the length of Hash lists in data de-duplication systems grows, leading to increased memory-loading times and search times, reducing the efficiency of data retrieval.
Innovation Solution
A method for creating a hierarchical index file that includes location blocks with storage fields for primary Hash values, using a first and second Hash procedure to calculate block and field numbers, and a location conflict list to optimize data block location recording, thereby enhancing search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a Hash list is used to record all data blocks, then data de-duplication can be achieved, but memory-loading time and search time increase as data quantity increases
Solution Approach 1:
The patent divides the Hash list into multiple Hash tables, each containing multiple buckets. This segmentation reduces the search space from a single large list to smaller, manageable tables and buckets, thereby decreasing memory-loading time and search time while maintaining data de-duplication capability.
Solution Approach 2:
The patent introduces a hierarchical structure with Hash tables as the first dimension and buckets as the second dimension. This two-dimensional organization allows for more efficient data distribution and retrieval compared to a one-dimensional Hash list, reducing both loading time and search time.
2Quantity of substance
If the Hash list length increases to accommodate more data, then more data can be stored, but access efficiency decreases
Solution Approach 1:
By segmenting the large Hash list into multiple smaller Hash tables with multiple buckets each, the system can store larger quantities of data while maintaining efficient access. The segmentation allows parallel processing and reduces the average search path length.
Solution Approach 2:
Each Hash table and bucket is optimized to handle a specific portion of the data, allowing for localized access patterns. This local optimization ensures that access efficiency is maintained even as overall data capacity increases.
Data Source
AI summary
An method for creating a index of the data blocks is applicable in data de-duplication procedure, includes loading an index file, the index file includes a plurality of location blocks, each location block includes a plurality of storage fields, and each storage field records a primary Hash value corresponding to the data block; performing a first Hash procedure on a primary Hash value of the data block and calculating a block number; performing a second Hash procedure on the primary Hash value in the same data block and calculating a field number; loading a location conflict list; comparing the field number with the field number in the location conflict list to search whether the same field number is stored in the location conflict list; writing the primary Hash value into the corresponding block number and the field number if the field number does not exist in the location conflict list.