Stream Locality Delta Compression for Index Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems, despite achieving high data compression, still face inefficiencies due to the large size of the sketch index required for delta compression, which can lead to cumbersome storage and increased latency in large enterprise systems.
Innovation Solution
Stream locality delta compression replaces the master sketch index with a limited index of sketches, using stream locality to identify likely similar segments and reducing the need for frequent access to high-latency memory by maintaining a limited sketch index and using it to quickly determine similarity between data segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If a master sketch index is used for delta compression, then data compression effectiveness is improved, but storage space consumption and access latency increase
Solution Approach 1:
The patent divides the master sketch index into multiple worker sketch indexes distributed across different nodes. Each worker node maintains its own sketch index and processes data segments locally, eliminating the need for a single large centralized index while maintaining compression effectiveness through distributed processing.
Solution Approach 2:
The patent transitions from a single-dimension centralized index structure to a multi-dimensional distributed index structure across multiple nodes. This dimensional change allows the system to maintain compression ratios while reducing the storage burden on any single node and improving parallel access capabilities.
2Loss of substance
If a master sketch index is used for delta compression, then data compression effectiveness is improved, but access speed decreases due to high-latency memory requirements
Solution Approach 1:
The patent segments the centralized index access into distributed local index accesses at worker nodes. Each worker node maintains its own sketch index in local memory, eliminating the need to access a large centralized index in high-latency memory, thus improving access speed while maintaining compression effectiveness.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and distributing sketch indexes to worker nodes before data processing begins. This allows worker nodes to perform delta compression operations using local indexes without needing to access the master sketch index during runtime, significantly reducing access latency.
3Quantity of substance
If deduplication is applied to large enterprise systems, then data redundancy is reduced, but the resulting data volume remains too large for efficient storage
Solution Approach 1:
The patent introduces sketch indexes as intermediary structures that enable efficient delta compression. These sketches serve as mediators between deduplication and final compression, allowing the system to identify similar data segments and apply delta compression to reduce the final backup data size while maintaining efficient storage.
Solution Approach 2:
The patent changes the compression parameter from simple deduplication (exact matches only) to delta compression (similar segments with differences). This parameter change allows the system to compress data that is not exactly identical but sufficiently similar, significantly reducing the final backup data volume for large enterprise systems.
Data Source
AI summary
Stream locality delta compression is disclosed. A previous stream indicated locale of data segments is selected. A first data segment is then determined to be similar to a data segment in the stream indicated locale.