Data Reduction Estimate Generation via Signature Prefix Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data reduction methods in information processing systems require substantial computational and memory resources, leading to inefficiencies and performance issues when generating deduplication or compression estimates for storage volumes.
Innovation Solution
The technique involves scanning pages of datasets using designated scan criteria, such as content-based signature prefixes or polynomial-based signature subspace, to efficiently generate data reduction estimates, reducing the computational and memory resources needed and improving storage system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data reduction methods are used to generate deduplication estimates for storage volumes, then data reduction decisions can be made, but substantial computational and memory resources are consumed leading to performance issues
Solution Approach 1:
The patent segments the data reduction estimation process into distinct phases: a fast filtering phase that scans only signature prefixes to identify candidate duplicate pages, and a more accurate but resource-intensive verification phase that processes only the candidates identified by the filter. This segmentation allows the system to achieve accurate data reduction estimates while minimizing computational and memory resource consumption during the estimation process.
Solution Approach 2:
The patent applies partial action by performing data reduction estimation on a sampled subset of pages rather than processing every page in full detail. The signature prefix filtering mechanism examines only a portion of each page's signature (the prefix) to identify candidates, and then performs more complete analysis only on those candidates that match. This partial processing approach significantly reduces resource requirements while maintaining estimate accuracy.
2Measurement precision
If conventional data reduction methods scan all pages to generate accurate estimates, then measurement precision is improved, but computational resources and memory resources are significantly increased
Solution Approach 1:
The patent divides the page scanning process into two segments: first, a lightweight scan of signature prefixes that requires minimal memory resources, and second, a more resource-intensive scan of only those pages that match the prefix criteria. This segmentation enables accurate data reduction estimation while keeping the quantity of computational and memory resources required at acceptable levels.
Solution Approach 2:
The patent introduces signature prefixes as an intermediary mechanism that mediates between the need for accurate data reduction estimates and the limitation of available computational and memory resources. By using prefixes as a filtering intermediary, the system can quickly identify candidate duplicate pages without performing full-page comparisons on all pages, thus reducing resource consumption while maintaining estimate precision.
3Measurement precision
If full page scanning is performed to generate data reduction estimates for multiple datasets, then accurate comparison is achieved, but the time required for processing increases substantially
Solution Approach 1:
The patent performs preliminary action by scanning and comparing signature prefixes of pages before performing full page content comparison. This preliminary filtering step quickly identifies candidate duplicate pages across multiple datasets based on their prefix signatures, and only these candidates undergo the more time-consuming full comparison process. This preliminary action significantly reduces the overall processing time while maintaining accurate data reduction estimates.
Solution Approach 2:
The patent segments the time-consuming page comparison process into a fast prefix-matching phase and a slower full-content verification phase. By processing pages in these segments rather than performing full comparisons on all pages simultaneously, the system reduces total processing time while preserving measurement precision through the verification phase.
Data Source
AI summary
An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to identify at least first and second datasets to be scanned to generate a data reduction estimate for a prospective combination of the first and second datasets, to designate a scan criterion to be utilized in the scan of each of the datasets, and for each of a plurality of pages of each of the datasets, to scan the page, where scanning the page comprises performing a computation on the page to obtain a page result, determining whether or not the page result satisfies the designated scan criterion, and responsive to the page result satisfying the designated scan criterion, updating a corresponding entry of a data reduction estimate table for the dataset. The processing device merges contents of the data reduction estimate tables, and generates the data reduction estimate based at least in part on the merged contents.