Distributed Database Deduplication via Proxy Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed databases face challenges in efficiently improving deduplication processing due to the lack of effective task decomposition, which hinders the enhancement of processing efficiency.
Innovation Solution
A method for data deduplication in a distributed database where a proxy node acquires deduplication operation request information, sends it to serving nodes for preprocessing, and summarizes preprocessed record information with Hash values, allowing for immediate deduplication processing, thereby leveraging parallel computing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed database task decomposition is applied to deduplication processing, then parallel computing capability is improved, but task decomposition effectiveness deteriorates due to lack of inherent distributed characteristics
Solution Approach 1:
The patent divides the deduplication processing into two distinct phases: a preprocessing phase where each serving node independently generates Hash values for its local records, and a deduplication phase where the proxy node performs the actual deduplication on collected preprocessed data. This segmentation enables parallel execution at the preprocessing stage while keeping the deduplication logic centralized, thus improving productivity without excessive complexity.
Solution Approach 2:
The patent implements preliminary action by having each serving node precompute Hash values for its records before the actual deduplication operation. This preprocessing step transforms raw record data into a standardized format with Hash values already calculated, enabling the proxy node to perform deduplication efficiently without repeating Hash calculations. This preliminary action resolves the contradiction by preparing data in advance, improving overall processing efficiency while maintaining clear task boundaries.
2Device complexity
If centralized deduplication processing is used, then task decomposition is avoided, but processing speed deteriorates due to single-node bottleneck
Solution Approach 1:
The patent segments the deduplication workload by separating data preparation (Hash generation) from data processing (deduplication). Serving nodes perform the segmentation of Hash generation in parallel, while the proxy node handles the centralized deduplication. This segmentation allows the system to maintain structural simplicity at the deduplication stage while achieving speed improvement through parallel preprocessing at the serving nodes.
Solution Approach 2:
The patent applies preliminary action by having serving nodes precompute Hash values and prepare preprocessed record data before submitting to the proxy node. This preliminary preparation eliminates the need for the proxy node to perform repeated Hash calculations on raw data, significantly improving processing speed while keeping the overall system structure relatively simple with clear division of labor between serving and proxy nodes.
3Reliability
If redundant Hash calculations are performed at the proxy node, then data processing completeness is improved, but energy consumption increases
Solution Approach 1:
The patent implements preliminary action by having each serving node calculate Hash values for its records before submitting them to the proxy node. This preliminary Hash calculation ensures that the proxy node receives preprocessed data with reliable Hash values already computed, maintaining deduplication accuracy while avoiding redundant Hash calculations at the proxy node, thus reducing overall energy consumption.
Solution Approach 2:
The patent uses copying by having serving nodes generate and attach Hash values (copies of the original data's fingerprint) to their record data before transmission. The proxy node receives these copied Hash values and uses them directly for deduplication comparison without needing to recalculate Hash values from the original data, ensuring accuracy through verified Hash copies while significantly reducing computational energy consumption at the proxy node.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly improves the efficiency of deduplication processing by utilizing the distributed database's parallel computing capability, reducing the need for redundant Hash calculations and enhancing the overall processing speed.
Implementation Method 1
a first Hash value generated from a Hash calculation on the record information
Data Source
Figure 1
Figure 2~4
Figure 5~9
AI summary
A data deduplication method, a node, and a computer readable storage medium. The data deduplication method comprises: obtaining deduplication operation request information; according to the deduplication operation request information, sending to a service node request operation information carrying distributed deduplication operation information, such that the service node can preprocess, according to the distributed deduplication operation information, record information satisfying a condition to obtain preprocessed record information comprising the record information and a first hash value; receiving the preprocessed record information sent by the service node, and aggregating the preprocessed record information to a predetermined processing storage area; and traversing the preprocessed record information in the processing storage area to perform deduplication processing on the record information according to the first hash value.