Data Chunk Signature Indexing for WAN Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network communication systems face significant latency and performance degradation due to insufficient bandwidth in Wide Area Networks (WAN) and other low-bandwidth links, particularly when accessing data across geographically dispersed enterprises, and existing solutions like Low-Bandwidth Network File Systems (LBFS) are not scalable, compatible with standard file systems, or support interactive access.
Innovation Solution
A method that reduces network traffic by identifying data chunks, computing signatures, and using index tables to efficiently transmit only new data chunks over WAN links, while maintaining a synchronized data transmission context between senders and receivers, supporting existing file system protocols and enabling scalable deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is transmitted over WAN links to enable geographically dispersed enterprise information sharing, then information sharing capability is improved, but network latency and response time increase significantly
Solution Approach 1:
The patent segments data into data blocks and further divides them into data chunks with unique identifiers. This segmentation allows the system to transmit only the specific chunks that have changed or are needed, rather than transmitting entire files or data sets, thereby reducing network latency while maintaining information sharing capability.
Solution Approach 2:
The patent establishes data transmission contexts in advance between senders and receivers, pre-defining the structure and identifiers for data blocks and chunks. This preliminary action enables efficient data transmission by allowing receivers to efficiently process and reassemble data without requiring complex runtime negotiations, thus reducing latency.
2Ease of operation
If client applications read and write files over WAN links, then file access functionality is maintained, but network bandwidth is consumed twice (read and write operations)
Solution Approach 1:
The patent extracts only the changed portions of files (data chunks) for transmission over the WAN link, rather than transmitting entire files. By identifying and transmitting only the minimal necessary data changes, the system maintains file access functionality while significantly reducing bandwidth consumption for both read and write operations.
Solution Approach 2:
The patent uses data chunk identifiers and transmission contexts to create efficient references to data rather than always transmitting complete copies. The system can reference previously transmitted data chunks through their unique identifiers, reducing redundant bandwidth consumption while maintaining full file access capability.
3Quantity of substance
If Low-Bandwidth Network File System (LBFS) is deployed to reduce network traffic, then bandwidth efficiency is improved, but compatibility with standard file systems and scalability are reduced
Solution Approach 1:
The patent implements a universal data transmission mechanism that works across different file systems and network configurations. By defining abstract data blocks and chunks with standardized identifiers and transmission contexts, the system achieves broad file system compatibility while maintaining bandwidth efficiency, unlike LBFS which is tied to specific implementations.
Solution Approach 2:
The patent introduces data transmission contexts as an intermediary layer between the file system and the network transmission mechanism. This intermediary abstraction allows the system to reduce network traffic through chunk-based transmission while remaining compatible with standard file systems, as the transmission context handles the complexity of data segmentation and reassembly transparently.
4Productivity
If interactive file access is enabled over low bandwidth links, then user productivity is improved, but network latency increases due to limited bandwidth
Solution Approach 1:
The patent segments files into smaller data chunks that can be transmitted independently and efficiently over low bandwidth links. This segmentation allows interactive file access to proceed with minimal latency, as only the specific chunks needed for current operations are transmitted, rather than waiting for entire files to be transferred.
Solution Approach 2:
The patent maintains continuous data transmission contexts that persist across multiple file operations. This continuity allows the system to efficiently handle interactive access patterns by reusing existing transmission contexts and avoiding repeated setup overhead, thereby maintaining user productivity while working within bandwidth constraints.
Data Source
AI summary
A method is disclosed for reducing network traffic. At a sender, a data chunk is identified for transmission to a receiver, which is connected to the sender over a communication link. The sender computes a signature of the data chunk and determines whether the data chunk has been previously transmitted by looking up the signature in a sender index table. The sender index table associates the signatures of previously transmitted data chunks with unique index values. A message is transmitted to the receiver, where if the data chunk has previously been transmitted then the message includes an index value from the sender index table that is associated with the signature of the data chunk. At the receiver, the data chunk is located in a receiver cache that stores the previously transmitted data chunks by looking up the index value included in the message in a receiver index table. The receiver index table associates the unique index values with the locations in the receiver cache of the previously transmitted data chunks.