Adaptive Compression Dictionaries for Data Replication Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods in data replication processes often result in suboptimal compression performance due to limitations in the initial compression dictionaries, leading to inefficient use of storage space and increased latency in data synchronization between data sources and targets.
Innovation Solution
A method that evaluates the compression performance of a first compression dictionary and generates a second dictionary based on existing data on the data target, using a larger sample size to improve compression efficiency, and updates the data target using the new dictionary if the initial performance is below a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a first compression dictionary is used to compress existing data on the data target, then data compression is performed, but the compression performance is lower than a preset performance threshold
Solution Approach 1:
The system performs preliminary evaluation of compression performance before finalizing the compression dictionary. By evaluating the compression performance of the first compression dictionary against existing data and comparing it with a preset threshold, the system determines whether to generate a second compression dictionary, thus preventing suboptimal compression from being applied
Solution Approach 2:
The system implements a feedback mechanism where compression performance is evaluated and used to trigger dictionary regeneration. When the compression performance falls below the preset threshold, the system generates a new compression dictionary based on the existing data, creating a closed-loop control that continuously optimizes compression performance
2Manufacturing precision
If a compression dictionary is generated based on a larger sample size of existing data, then compression performance is improved, but latency of data synchronization increases
Solution Approach 1:
The system applies partial action by only generating a new compression dictionary when necessary (when compression performance falls below the preset threshold). This avoids the excessive action of continuously regenerating dictionaries, thus reducing unnecessary processing time and data synchronization latency while still achieving improved compression performance when needed
Solution Approach 2:
The system changes the parameter of the compression dictionary by generating a second dictionary with different characteristics based on the existing data when the first dictionary's performance is insufficient. This parameter change (switching dictionaries) improves compression performance without requiring continuous regeneration, thereby limiting time loss
3Manufacturing precision
If the data target is updated using a second compression dictionary generated from existing data, then compression performance is improved, but device complexity increases
Solution Approach 1:
The system implements dynamic compression dictionary management where the compression dictionary is not static but can be updated from the first dictionary to a second dictionary based on performance evaluation. This dynamic approach allows the system to adapt to data patterns while maintaining a clear decision-making process that limits complexity through threshold-based triggers
Data Source
AI summary
The present invention provides a computer-implemented method, computer system and computer program product for data compression. According to the computer-implemented method, one or more data blocks on a data source to be replicated to a data target may be detected. Then, compression performance of a first compression dictionary may be evaluated. The first compression dictionary may be previously used to compress existing data on the data target. If the compression performance is lower than a preset performance threshold, a second compression dictionary may be generated based on the existing data on the data target. The data target may be updated based on the existing data and the one or more data blocks using the second compression dictionary.