Data Block Boundary Detection for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data block boundary detection methods for deduplication in backup streams, particularly for Oracle data blocks, suffer from false positives due to weak pattern detection, leading to inefficient deduplication and potential loss of deduplication opportunities.
Innovation Solution
The method involves determining block boundaries by matching content between a data block's header and footer, and further confirming the boundary by locating a second adjacent header to reduce false positives, allowing for iterative searches across different block size candidates and patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a weak byte pattern is used to detect markers in Oracle data streams, then marker detection is simplified, but false positives increase leading to degraded deduplication
Solution Approach 1:
The patent segments the marker detection process into two independent stages: first detecting potential markers using a simple weak byte pattern, then validating each potential marker by checking for the presence of a footer at the expected offset. This segmentation allows the system to maintain operational simplicity while eliminating false positives through validation.
Solution Approach 2:
The patent introduces an intermediary validation step (footer checking) between the weak pattern detection and the final marker confirmation. This intermediary mechanism acts as a filter that eliminates false positives without requiring changes to the initial simple detection pattern, thus maintaining ease of operation while improving reliability.
2Measurement precision
If false positive markers are detected, then block boundaries are incorrectly identified, but this forces unnecessary anchoring and reduces deduplication efficiency
Solution Approach 1:
The patent performs preliminary validation by checking for the presence of a footer at the expected offset before confirming a marker detection. This preliminary action prevents false positive markers from being processed further, thereby avoiding incorrect block boundary identification and unnecessary anchoring operations that would reduce deduplication efficiency.
3Productivity
If metadata portion is removed before anchoring, then deduplication of data portion is improved, but block boundary detection becomes more complex
Solution Approach 1:
The patent segments the data block into distinct portions (metadata/header and data portion) with a clear boundary marker. By detecting the footer pattern that marks the boundary, the system can cleanly separate metadata removal from anchoring operations, maintaining simplicity in the overall process while improving deduplication efficiency.
Data Source
AI summary
Techniques for deduplicating data streams are described herein. According to one embodiment, a first data stream is received to be stored in a storage system, where the first data stream includes data blocks and each data block includes a header and a footer. A boundary detector is to detect boundaries of the data blocks by matching at least a portion of a header with a footer of each data block and a header of an adjacent data block. An anchoring unit is to anchor the first data stream based on the determined boundaries of the data blocks using a plurality of anchors. A deduplication engine is to deduplicate the first data stream into a plurality of deduplicated data segments based on the plurality of anchors. The deduplicated data segments are then stored in one or more persistent storage devices of the storage system.