WAN Traffic Optimization via Selective Chunk Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for optimizing WAN traffic have not been effective in efficiently eliminating duplicate data transfers across wide area networks, leading to suboptimal bandwidth usage.
Innovation Solution
A network protocol is implemented between local and remote proxy devices to communicate deduplication information, using chunking and hashing to identify and eliminate duplicate data transfers by maintaining a chunk index and stream store, and selectively indexing chunks based on their likelihood of change.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all chunks are indexed in the chunk index, then deduplication accuracy is improved, but memory consumption increases significantly
Solution Approach 1:
The patent applies partial indexing by selectively indexing only certain chunks (e.g., first and last chunks, or chunks at specific intervals) rather than all chunks. This partial action approach maintains adequate deduplication functionality while significantly reducing the memory burden of maintaining a complete chunk index for all data blocks.
Solution Approach 2:
The patent segments the chunk index into multiple zones or regions (e.g., frequently accessed chunks, recently modified chunks, and less important chunks). By segmenting the indexing strategy, the system can allocate memory resources preferentially to high-value chunks while using less memory for less critical chunks, thus balancing deduplication accuracy with memory constraints.
2Device complexity
If chunk size is increased, then number of index entries is reduced, but granularity of deduplication is decreased
Solution Approach 1:
The patent implements dynamic chunk sizing where the chunk size is not fixed but adapts based on characteristics of the data being processed. The system can adjust chunk boundaries dynamically to optimize between the number of index entries and deduplication granularity, allowing larger chunks for repetitive data and smaller chunks for unique data segments.
Solution Approach 2:
The patent applies different chunk sizes to different regions or types of data within the same data stream. For example, frequently changing data regions may use smaller chunk sizes for finer granularity, while stable, repetitive regions use larger chunk sizes to reduce index complexity. This local quality approach allows optimized deduplication for each data region's specific characteristics.
Data Source
AI summary
According to one embodiment, a local proxy caches in a local stream store one or more streams of data transmitted over the WAN to a remote proxy. In response to a flow of data received from one of the clients of the local LAN, the local proxy chunks using a predetermined chunk algorithm the flow into chunks in sequence, and selectively indexes the chunks in a chunk index maintained by the local proxy based on locations of the chunks in the flow, where a number of chunks in a first region of the flow indexed is different than a number of chunks in a second region of the flow indexed. The chunk index includes multiple chunk index entries referenced to the streams stored in the stream store.