WAN Traffic Optimization via Chunk Index Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for optimizing WAN traffic are inefficient in eliminating duplicate data transfers, particularly due to issues with hash collisions and the need for extensive indexing, which hampers bandwidth conservation across wide area networks.
Innovation Solution
A network protocol is implemented between local and remote proxy devices to communicate deduplication information, using chunking and selective indexing to identify and eliminate duplicate data transfers by maintaining a chunk index and stream store, and employing Rabin fingerprinting to efficiently manage data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If extensive indexing is used to identify duplicate data, then duplicate detection accuracy is improved, but system complexity and processing overhead increase
Solution Approach 1:
The patent segments data into fixed-size chunks and creates indexes at the chunk level rather than indexing entire files or data streams. This segmentation allows for efficient duplicate detection by comparing individual chunks independently, reducing the overall complexity while maintaining accurate duplicate identification across large data transfers
Solution Approach 2:
The patent introduces chunk hashes as an intermediary representation between the actual data chunks and the indexing mechanism. By computing and comparing hashes of chunks rather than directly comparing chunk contents, the system achieves accurate duplicate detection with significantly reduced processing overhead and complexity
2Productivity
If Rabin fingerprinting is used to identify similar substrings, then deduplication effectiveness is improved, but computational overhead increases
Solution Approach 1:
The patent changes the parameter of hash computation by using simple checksums or hash functions on fixed-size chunks rather than computationally intensive Rabin fingerprinting algorithms. This parameter change maintains deduplication effectiveness by accurately identifying duplicate chunks while significantly reducing computational overhead and energy consumption
3Loss of energy
If all data chunks are indexed for deduplication, then bandwidth conservation is improved, but memory usage increases
Solution Approach 1:
The patent applies partial indexing by selectively indexing only those chunks that are likely to be duplicated based on data patterns and access frequencies. Rather than indexing every single chunk, the system identifies and indexes representative chunks from duplicate sets, achieving effective bandwidth conservation through differential transfers while limiting memory consumption to manageable levels
Data Source
AI summary
A local proxy caches, in one or more transmitted data files (TDFs) in a deduplicated manner, chunks of one or more streams that have been transmitted to a remote proxy, each of the streams being identified by a stream identifier (ID). For each of the streams, the local proxy maintains a stream object having one or more TDF references, each TDF reference corresponding to at least a segment of the stream, wherein each TDF reference includes information identifying a file location within one of the TDFs at which the segment of the stream is located. The local proxy further maintains a chunk index having chunk index entries corresponding to deduplicated chunks of the streams, where each chunk index entry includes information identifying a stream location within one of the streams at which a corresponding chunk is located.