Remote Differential Compression for Low-Metadata Object Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer methods over limited bandwidth networks are inefficient, particularly when transferring large objects, as they incur significant overhead in transmitting metadata, leading to high bandwidth utilization even when the differences between object versions are small.
Innovation Solution
The implementation of remote differential compression (RDC) with recursive application to metadata and a local maximum-based chunking method to minimize metadata transfer, allowing for efficient updating of objects by identifying and transferring only the differences between object versions, thereby reducing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data transfer methods are used to transfer large objects over networks, then complete object data is transmitted, but bandwidth consumption increases significantly due to metadata overhead
Solution Approach 1:
The patent extracts and transfers only the differential portions (changes) between old and new object versions rather than transmitting complete objects. By identifying and isolating only the modified chunks and their metadata, the system removes unnecessary data transmission while maintaining data transfer completeness.
Solution Approach 2:
The patent divides large objects into smaller chunks and further segments metadata into signature lists and chunk lists. This segmentation allows selective transmission of only those segments that have changed, reducing overall bandwidth consumption while ensuring complete reconstruction of updated objects at the receiving end.
2Loss of energy
If RDC is applied to large objects frequently, then bandwidth usage is reduced, but the complexity of the transfer protocol increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing signature lists and chunk lists locally before transfer. Both sending and receiving endpoints independently generate these metadata structures in advance, allowing for efficient differential comparison without requiring complex real-time computation during the actual transfer process.
Solution Approach 2:
The patent uses signature lists as simplified copies or representations of the actual object data. Instead of transmitting complex object structures for comparison, the system uses compact signature hashes that uniquely represent each chunk, reducing protocol complexity while maintaining the ability to identify differences accurately.
3Measurement precision
If metadata is transmitted in full for each RDC operation, then chunk identification is accurate, but the amount of metadata sent over the network increases
Solution Approach 1:
The patent extracts only the essential metadata elements needed for differential compression - specifically signature lists and chunk lists - and transmits only the differential portions of these metadata structures. By removing redundant metadata and transmitting only the changes in signature and chunk information, the system maintains accurate chunk identification while minimizing metadata volume.
Data Source
AI summary
A method and system are related to updating objects over limited bandwidth networks. Objects are updated between two or more computing devices using remote differential compression (RDC) techniques such that required data transfers are minimized. In one aspect, efficient large object transfers are achieved by recursively applying the RDC algorithm to its own metadata; a single or multiple recursion step(s) may be used in this case to reduce the amount of metadata sent over the network by the RDC algorithm. Objects and/or signature and chunk length lists can be chunked by locating boundaries at dynamically determined locations. A mathematical function evaluates hash values associated within a horizon window relative to potential chunk boundary. The described method and system is useful in a variety of networked applications, such as peer-to-peer replicators, email clients and servers, client-side caching systems, general-purpose copy utilities, database replicators, portals, software update services, file/data synchronization, and others.