Hierarchical Data Structure with Probabilistic Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data backup systems face inefficiencies and resource constraints due to full copy strategies, particularly in deduplication processes, which lead to performance degradation and increased costs, especially in large databases with concurrent access requirements.
Innovation Solution
A hierarchical data structure utilizing probabilistic membership query filters and truncated key indexes across multiple layers, allowing for efficient data storage and retrieval while minimizing false positives and resource usage, enabling scalable and concurrent access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full copy strategies are used to store backup data, then data completeness is ensured, but system resource consumption and storage costs increase significantly
Solution Approach 1:
The patent extracts only the changed data blocks from the full backup image, storing only the differences between backup versions. This is achieved through block-level deduplication where the system identifies and stores only modified blocks rather than copying entire datasets, thereby reducing resource consumption while maintaining data completeness for recovery purposes
Solution Approach 2:
The system changes the storage parameter from storing complete data copies to storing only incremental changes. By using hashing algorithms to identify duplicate blocks and storing only unique changed blocks, the system transforms the backup approach from full replication to selective storage, reducing resource usage while preserving data integrity
2Quantity of substance
If traditional deduplication processes are implemented in large databases, then storage efficiency improves, but performance degradation occurs due to computational overhead
Solution Approach 1:
The patent segments the large database into smaller data blocks that can be processed independently. By dividing the backup operation into block-level operations rather than processing entire datasets as single units, the system improves performance through parallel processing while maintaining deduplication efficiency across the complete dataset
Solution Approach 2:
The system performs partial deduplication operations by focusing computational effort only on changed blocks rather than re-processing entire datasets. This partial action approach reduces computational overhead significantly while still achieving the necessary storage efficiency through selective hashing and comparison of only modified data portions
3Measurement precision
If probabilistic membership query filters are used, then false positive rates decrease, but memory usage increases due to filter data structures
Solution Approach 1:
The patent implements a hierarchical filter structure where multiple levels of probabilistic membership query filters are nested within each other. Each level of the hierarchy stores information about progressively smaller subsets of data blocks, allowing the system to achieve high precision in determining block existence while distributing memory requirements across multiple hierarchical levels rather than requiring a single large filter
Data Source
AI summary
In an embodiment, a hierarchical data structure includes data blocks separated into a plurality of data segments partitioned into a plurality of supersegments. Each data segment includes a predetermined number of data blocks. Each supersegment includes a predetermined number of data segments. Responsive to receiving data to store in the hierarchical data structure, a first subset of data segments are grouped into a first supersegment to allow a first portion of the data to be sequentially stored to a first set of data blocks in a first data segment of the first supersegment. Also, a second portion of the data is sequentially stored to a second set of data blocks in a second data segment of the first supersegment. Probabilistic membership query filters are generated at each different level for each data segment and the first supersegment of the hierarchical data structure to allow for efficient search and data retrieval.