Opportunistic Compression in Content Addressable Storage Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication methods consume excessive bandwidth and computational resources, making it difficult to achieve desired recover point objectives in information processing systems.
Innovation Solution
Implementing opportunistic compression in data replication processes by transmitting data in compressed form when available and performing end-to-end tests only when necessary, thereby reducing network bandwidth and computational resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transmitted in uncompressed form during replication, then data integrity and reliability are maintained, but network bandwidth consumption increases
Solution Approach 1:
The system dynamically changes the compression parameter based on data characteristics. When data is compressible, compression is applied to reduce bandwidth consumption; when data is already compressed or random, no compression is applied to avoid CPU overhead. This parameter change resolves the contradiction by adapting the transmission format to the actual data properties.
Solution Approach 2:
The patent uses content-based signatures (hashes) to identify duplicate data blocks across replicas. Instead of transmitting actual data blocks, the system transmits only the signature and relies on existing copies at target sites. This copying principle dramatically reduces network bandwidth consumption while maintaining data integrity through signature verification.
2Loss of energy
If compression is applied to all data during replication, then network bandwidth consumption is reduced, but computational resource consumption increases
Solution Approach 1:
The system changes the compression parameter dynamically based on data compressibility assessment. Compressibility is determined by analyzing data characteristics (e.g., entropy, repetition patterns). Only when data exhibits compressible patterns does the system apply compression, avoiding unnecessary CPU cycles on already-compressed or random data while still achieving bandwidth savings where applicable.
Solution Approach 2:
Instead of applying compression uniformly to all data, the system applies compression selectively only to portions of data that benefit from it. This partial action principle avoids the excessive computational overhead of compressing data that wouldn't compress well, while still achieving bandwidth reduction on compressible portions.
3Reliability
If end-to-end tests are performed on all replicated data, then data reliability is ensured, but processing time increases
Solution Approach 1:
The system uses content-based signatures to identify and skip duplicate data blocks during replication. When a target site already contains a copy of the data (verified by signature match), the system skips the end-to-end test for that block. This copying principle reduces processing time by avoiding redundant verification while maintaining reliability through signature-based identification.
Solution Approach 2:
The system performs end-to-end tests selectively only on new or modified data blocks, not on all replicated data. By identifying duplicate blocks through signature comparison, the system applies verification only where necessary, reducing overall processing time while maintaining data reliability for the portions that require verification.
Data Source
AI summary
An apparatus in one embodiment comprises a first storage system having a plurality of storage devices. The first storage system is configured to participate in a replication process with a second storage system. The first storage system comprises a replication engine configured to request from a data module of the first storage system at least one data page to be replicated to the second storage system as part of the replication process. If the requested data page is already available in compressed form in the data module, the replication engine receives the requested data page in compressed form and transmits it to the second storage system along with at least a portion of a content-based signature of the requested data page. If the requested data page is not already available in compressed form in the data module, the replication engine receives the requested data page in uncompressed form and transmits it to the second storage system.