Data Segment Processing Mode Selection for Deduplication and Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies due to the use of a single processing mode for all types of data, leading to higher processing times and costs, particularly for data with varying de-duplication rates and sizes, which limits system performance.
Innovation Solution
A method that determines the appropriate processing mode based on the sizes of data segments, compression levels, and de-duplication rates, allowing for either a mode where compression is performed only on non-duplicate segments or on all segments, optimizing processing time and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single processing mode is used for all data, then system simplicity is maintained, but processing efficiency deteriorates due to suboptimal performance across varying data types
Solution Approach 1:
The patent implements dynamic mode selection where the processing mode is not fixed but adapts based on real-time data characteristics. The system calculates de-duplication rates and compares processing times to dynamically determine whether to use Mode 1 or Mode 2, making the processing approach flexible and responsive to varying data types rather than static
Solution Approach 2:
The patent changes the parameter of processing mode selection based on data parameters such as de-duplication rate and segment size. By using thresholds (e.g., de-duplication rate threshold, time ratio threshold) to adjust the processing mode, the system optimizes performance for different data characteristics without requiring complex manual configuration
2Quantity of substance
If de-duplication processing is performed before compression, then storage space is optimized, but processing time increases significantly for data with low de-duplication rates
Solution Approach 1:
The patent changes the processing parameter (mode selection) based on the de-duplication rate parameter. When the de-duplication rate is high, Mode 1 is used to maximize storage optimization. When the de-duplication rate is low, Mode 2 is selected to minimize processing time, effectively adapting the approach to the actual data characteristics
Solution Approach 2:
The system dynamically switches between performing de-duplication before compression (Mode 1) and performing compression on all segments (Mode 2) based on real-time calculation of de-duplication rates and processing time comparisons, making the processing workflow flexible rather than fixed
3Manufacturing precision
If compression is performed on all data segments, then compression effectiveness is maximized, but processing time increases for data with high de-duplication rates
Solution Approach 1:
The patent adjusts the compression processing parameter based on the de-duplication rate. For high de-duplication rate data, it changes the mode to compress only non-duplicate segments, reducing processing time. For low de-duplication rate data, it maintains compression on all segments to maximize compression effectiveness
Solution Approach 2:
The patent applies partial compression action by compressing only the non-duplicate segments when appropriate, rather than compressing all segments. This partial action is sufficient to achieve storage optimization while significantly reducing processing time for high de-duplication rate data
Data Source
AI summary
Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for data processing. The method includes: determining, based on sizes of multiple data segments included in data to be processed, a first time required to perform a matching operation for each data segment, wherein the matching operation is used to determine non-duplicate data segments; determining, based on the size of each data segment and a compression level for the data to be processed, a second time required to perform a compression operation for each data segment; and determining, based on the first time, the second time, and a de-duplication rate for the data to be processed, a target mode for processing the multiple data segments from a first mode and a second mode, wherein in the first mode, a compression operation is performed only on the non-duplicate data segments in the multiple data segments, and in the second mode, a compression operation is performed on each of the multiple data segments. In this way, the data processing mode can be dynamically selected according to features of the data to be processed, thereby improving the efficiency of data processing.