Duplicated Data Detection Using Segmented Byte Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for determining duplicated data are inefficient, consuming significant CPU and storage resources due to direct comparison of input data with multiple reference data, making them unsuitable for processing massive information.
Innovation Solution
A method that filters out obviously non-duplicated data by comparing initial positions and feature values, reducing the need for extensive comparisons by associating input data with reference data only when a match is found, thereby minimizing resource consumption and improving recognition efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional method compares input data with multiple reference data cache line by cache line, then data recognition accuracy is maintained, but CPU resource consumption increases and recognition efficiency decreases
Solution Approach 1:
The patent divides the data comparison process into two segments: first comparing a predetermined number of bytes (e.g., 8 bytes) at the beginning of data structures, and only if they match, then comparing the remaining bytes. This segmentation reduces the number of full data comparisons needed, thereby reducing CPU resource consumption while maintaining recognition accuracy for duplicated data.
Solution Approach 2:
The patent performs a preliminary comparison of the first predetermined number of bytes before proceeding to compare the rest of the data. This preliminary action acts as a filter to quickly eliminate non-duplicated data, reducing the overall computational burden and improving recognition efficiency without sacrificing the ability to identify duplicated data accurately.
2Reliability
If direct comparison of input data with reference data is performed, then data duplication detection is achieved, but storage and calculation resources are wasted
Solution Approach 1:
The patent segments the data comparison into a first comparison phase (predetermined number of bytes) and a second comparison phase (remaining bytes). This segmentation allows the system to quickly identify and discard non-duplicated data after the first phase, reducing unnecessary storage and calculation resource waste while maintaining detection accuracy through the second phase for candidates that pass the first filter.
Solution Approach 2:
The preliminary comparison of the first predetermined number of bytes serves as a filtering mechanism that eliminates obviously non-duplicated data before full comparison. This preliminary action prevents wasteful allocation of storage and calculation resources to data that can be ruled out without complete comparison, thereby reducing resource waste while maintaining reliable detection through the subsequent complete comparison of candidates.
Data Source
AI summary
Technique determine (or detect) duplicated data. The techniques involve: in response to determining that data at a first position in input data is the same as predetermined data, determining a feature value of a selected portion of input data; determining whether the feature value matches with a pre-stored duplicated data pattern in a duplicated data pattern list; and in response to determining that the feature value matches with the duplicated data pattern, determining an association of the input data with reference data which is associated with the matched pattern.