Deduplication Database Management Using Tree-Structured Hash Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database deduplication systems face performance issues due to large hash databases that exceed system memory limits, leading to slow access times and reduced performance, as well as redundant storage of hash value data.
Innovation Solution
Implementing a tree-structured hash database with age indicators for entries, trimming unique data, and selectively maintaining portions in system memory and persistent storage, along with a deduplication engine that uses Bloom Filters and fine-grain mapping to reduce database size and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large hash database is used to store hash values of previously stored data, then deduplication accuracy is improved, but system memory usage increases and access time increases
Solution Approach 1:
The hash database is segmented into multiple hash pages, each storing a specific range of hash values. Each hash page contains a base hash value and multiple entries with offset values relative to the base. This segmentation allows the system to search only relevant portions of the database rather than scanning the entire database, significantly reducing access time while maintaining complete deduplication coverage.
Solution Approach 2:
The patent introduces a hierarchical dimension to the hash database structure by organizing hash pages in a tree structure with root nodes and leaf nodes. This multi-dimensional organization enables efficient navigation and search operations, reducing access time from linear scanning to logarithmic or constant-time operations depending on the search implementation.
2Reliability
If a large hash database is used to store hash values of previously stored data, then deduplication accuracy is improved, but system memory usage increases
Solution Approach 1:
The patent extracts only the essential components needed for deduplication from the full hash value. Instead of storing complete hash values, it stores base hash values in hash pages and uses offset values to represent individual hash entries. This extraction reduces the memory footprint of the hash database while preserving the ability to accurately identify duplicate data through hash comparison.
3Quantity of substance
If hash pages include base hash values and offset values, then storage efficiency is improved, but computational complexity increases
Solution Approach 1:
The hash database is pre-organized with base hash values and structured entry offsets during database construction. This preliminary structuring allows subsequent search operations to simply compare offsets against the base value rather than performing complex hash calculations or comparisons, reducing the computational complexity of duplicate detection operations.
Data Source
AI summary
An apparatus may include a controller configured search a hash database storing entries corresponding to hash values of previously stored data to find a hash page of the hash database corresponding to a range of hash values including the first hash value. When the hash page is found, the controller may be configured to determine whether the hash page does not include an entry for the first hash value, the first hash page further including a base hash value for the range of hash values. When the hash page does not include an entry for the first hash value, the controller may be configured to generate a first entry of the first hash page for the first hash value, the first entry including an offset value from the base hash value corresponding to the first hash value and a data location at which the received data is to be stored.