Adaptive Data Segmentation for Better Compression Dictionary Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data segmentation methods in data compression often result in excessively large segment lengths, reducing the matching probability and utilization efficiency of dictionaries.
Innovation Solution
A method and device that dynamically acquire a segmentation rule based on the length of the unsegmented data stream, where the probability of satisfying the segmentation rule increases monotonically with the length, allowing for effective control of segment length and improved dictionary utilization by ensuring that a segmentation rule corresponding to a smaller length is also satisfied for a larger length.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a segmentation method with fixed number of bytes is used, then the segmentation process is simple, but the dictionary utilization efficiency deteriorates when data changes occur
Solution Approach 1:
The patent applies dynamics by making the segmentation rule adaptive rather than fixed. The segmentation rule is dynamically adjusted based on the content fingerprint of the data stream, allowing the segmentation boundaries to adapt to data changes. This resolves the contradiction by maintaining operational simplicity through automated adaptation rather than manual reconfiguration.
Solution Approach 2:
The patent changes the parameter of segmentation rule from a static fixed value to a dynamic value determined by content fingerprint analysis. By calculating content fingerprints and using them to determine segmentation boundaries, the system transforms the segmentation parameter into something that automatically adjusts to data characteristics, improving dictionary utilization while keeping the process simple.
2Productivity
If the length of data segment is increased, then the compression ratio improves, but the matching probability deteriorates
Solution Approach 1:
The patent uses content fingerprint calculation to dynamically determine optimal segmentation lengths. By analyzing the fingerprint characteristics of data blocks, the system adjusts segment lengths to balance compression ratio and matching probability, rather than using a purely fixed or excessively long segment length.
Solution Approach 2:
The system implements feedback by using content fingerprint analysis to inform segmentation decisions. The fingerprint calculation provides information about data characteristics that feeds back into the segmentation rule, allowing the system to adjust segment lengths based on actual data patterns rather than predetermined fixed lengths.
3Reliability
If a sliding window method with content fingerprint is used, then the dictionary utilization efficiency improves, but the segmentation process complexity increases
Solution Approach 1:
The patent applies self-service by enabling the data stream itself to provide the information needed for segmentation through content fingerprint calculation. The data's own characteristics (captured in the fingerprint) are used to determine its optimal segmentation, eliminating the need for external complex analysis or manual intervention.
Solution Approach 2:
The patent replaces complex mechanical segmentation processes with a computational approach based on content fingerprinting. Instead of using complex rule-based or manual segmentation methods, the system uses fingerprint calculation and pattern recognition to automatically determine segmentation boundaries, reducing overall process complexity.
Data Source
AI summary
A method and device for data segmentation in data compression are disclosed. The method includes: acquiring a segmentation rule corresponding to the length of a current unsegmented data stream according to the length of the current unsegmented data stream in a data stream to be compressed, whereas the length of the unsegmented data stream increases, the probability that the corresponding segmentation rule is satisfied increases monotonically, and when the segmentation rule corresponding to a first length is satisfied, the segmentation rule corresponding to a second length is definitely satisfied too; where the first length and the second length are the length of the unsegmented data stream, and the first length is smaller than the second length; and segmenting, by using the segmentation rule, the data stream to be compressed.