Overlay Network Data Differencing via Edge Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data differencing techniques, such as RFC 3229 and Google SDCH, are inefficient in overlay networks as they rely on the origin server for differencing and do not allow for simultaneous reading and sending of data chunks, and lack easy computation of shared dictionaries, leading to increased latency and bandwidth usage.
Innovation Solution
Implementing data differencing-aware processes at both the origin server and edge server, which maintain object dictionaries and use fingerprints to synchronize versions, allowing for efficient computation and transmission of data differences between client and server, reducing redundant data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data differencing is performed at the origin server using RFC 3229, then bandwidth usage is reduced, but latency increases due to sequential processing and inability to read/send data simultaneously
Solution Approach 1:
The patent segments the data differencing process into independent chunks that can be processed in parallel. Multiple data chunks are read from storage and processed simultaneously through the differencing engine, rather than sequentially processing one chunk at a time. This segmentation enables concurrent execution of differencing operations on different portions of data, reducing overall processing time and latency while maintaining bandwidth efficiency.
Solution Approach 2:
The patent implements preliminary action by pre-loading multiple data chunks into memory before processing begins. The system reads and prepares multiple chunks in advance, so that when differencing is initiated, the processing can proceed without waiting for sequential disk reads. This preliminary preparation of data enables faster, parallel differencing operations that reduce latency while still achieving bandwidth reduction through efficient delta encoding.
2Loss of energy
If data differencing is performed at the origin server, then redundant data transfer is reduced, but device complexity increases due to software installation requirements
Solution Approach 1:
The patent implements self-service by enabling edge servers to perform differencing operations locally using their own cached copies of previous versions. Instead of requiring the origin server to have special differencing software, the edge servers independently compute data differences by comparing incoming data with their local caches. This distributes the computational burden and eliminates the need for complex software installation at the origin server, while still achieving bandwidth reduction through local differencing.
Solution Approach 2:
The patent inverts the traditional differencing architecture by moving the differencing function from the origin server to the edge servers. Instead of the origin server computing deltas and sending them to clients, edge servers perform the differencing locally by comparing received data with their cached versions. This inversion simplifies the origin server's role to mere data forwarding while enabling edge servers to intelligently process and reconstruct data, reducing both software complexity at the origin and bandwidth usage.
3Loss of energy
If shared dictionaries are computed and maintained, then data compression efficiency is improved, but computation time and processing overhead increase
Solution Approach 1:
The patent segments the dictionary maintenance process by creating multiple independent dictionary instances at different edge servers rather than maintaining a single centralized dictionary. Each edge server maintains its own dictionary based on local cache contents, allowing parallel dictionary operations without centralized coordination overhead. This segmentation enables efficient local compression while avoiding the computational burden of centralized dictionary management and synchronization.
Solution Approach 2:
The patent uses copying by having edge servers replicate dictionary data from their local caches rather than computing dictionaries from scratch or fetching them centrally. When an edge server needs a dictionary for differencing, it copies the necessary data from its existing cache, which already contains previous versions of the content. This copying approach eliminates redundant computation and reduces processing overhead while maintaining compression efficiency through accurate reference data.
Data Source
AI summary
A data differencing technique enables a response from a server to the request of a client to be composed of data differences from previous versions of the requested resource. To this end, data differencing-aware processes are positioned, one at or near the origin server (on the sending side) and the other at the edge closest to the end user (on the receiving side), and these processes maintain object dictionaries. The data differencing-aware processes each execute a compression and differencing engine. Whenever requested objects flow through the sending end, the engine replaces the object data with pointers into the object dictionary. On the receiving end of the connection, when the data arrives, the engine reassembles the data using the same object dictionary. The approach is used for version changes within a same host/path, using the data differencing-aware processes to compress data being sent from the sending peer to the receiving peer.