Data Replication Compression Markers for Lower IO Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication technologies waste computational resources and extend IO response times by blindly applying compression to data blocks that are already compressed or encrypted, leading to ineffective compression attempts.
Innovation Solution
Attach a compressibility marker to data blocks at the source end indicating their compressibility, allowing the target end to skip unnecessary compression attempts and optimize resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to all data blocks during replication, then storage space is reduced, but computational resources are wasted and IO response time is extended for incompressible data
Solution Approach 1:
The source end determines compressibility of data blocks before replication and attaches compressibility markers in advance. This preliminary action allows the target end to skip compression for incompressible data, avoiding wasted computational resources and extended IO response times while still achieving space savings for compressible data.
Solution Approach 2:
Compressibility markers are attached to data blocks and transmitted to the target end, providing feedback information about which data blocks are compressible and which are not. This feedback mechanism enables the target end to make informed decisions about whether to apply compression, optimizing both storage space utilization and IO performance.
2Quantity of substance
If data compression is applied to all data blocks during replication, then storage space is reduced, but computational resources are wasted
Solution Approach 1:
The source end determines compressibility of data blocks before replication and attaches compressibility markers in advance. This preliminary action allows the target end to skip compression for incompressible data, avoiding wasted computational resources while still achieving space savings for compressible data.
Solution Approach 2:
Compressibility markers are attached to data blocks and transmitted to the target end, providing feedback information about which data blocks are compressible and which are not. This feedback mechanism enables the target end to make informed decisions about whether to apply compression, optimizing both storage space utilization and computational resource consumption.
3Productivity
If compressibility markers are attached to all data blocks, then compression efficiency is improved, but data transmission overhead increases
Solution Approach 1:
The compressibility information is segmented into separate markers that are attached to data blocks. This segmentation allows the compression decision logic to be separated from the data transmission, enabling efficient compression for compressible data while minimizing the overhead impact through targeted marking rather than encoding compressibility into every data byte.
Data Source
AI summary
Techniques for data replication involve acquiring, in response to receiving a data replication request, compressibility information of a data block at a source end. Such techniques further involve determining a compressibility marker for the data block based on the compressibility information. Such techniques further involve replicating the data block and the compressibility marker to a target end via a transmission path, wherein the compressibility marker indicates whether the data block is compressible at the target end. Accordingly, the transmission delay during data replication can be reduced, and the data throughput can be increased. At the same time, the risk of failing to recover data due to long transmission delay can be reduced. In addition, compression attempts are no longer made on incompressible data blocks thus reducing unnecessary computation during the replication process, thereby achieving efficient utilization of system resources and improving the user experience.