Stream Deduplication via Peer Node Prediction and On-Demand Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream-based data deduplication systems face challenges in overlay networks, particularly in maintaining synchronized dictionaries across numerous peer nodes, leading to inefficiencies and increased latency due to the 'shift' problem and scalability issues.
Innovation Solution
The implementation of a deduplication system using directed cyclic graphs (DCGs) and on-demand cache synchronization, where each peer maintains a unique DCG for data compression and decompression, allowing for asynchronous dictionary updates and predictive pre-warming through annotated meta-information, enabling efficient data transmission and caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional stream-based data deduplication is used with synchronized dictionaries, then data compression efficiency is improved, but system complexity and latency increase due to the shift problem and the need for peer pairing
Solution Approach 1:
The patent segments the centralized dictionary into distributed peer-specific dictionaries across multiple peers in the overlay network. Each peer maintains its own dictionary independently, eliminating the need for complex synchronization mechanisms while preserving deduplication functionality through local fingerprint matching and on-demand retrieval.
Solution Approach 2:
The patent extracts the dictionary synchronization requirement from the deduplication process by implementing on-demand cache synchronization. Instead of maintaining synchronized dictionaries proactively, the system only retrieves specific fingerprint entries from remote peers when absolutely necessary, minimizing synchronization overhead and complexity.
2Measurement precision
If peer pairing is implemented for dictionary discovery, then deduplication accuracy is improved, but scalability deteriorates in overlay networks with numerous peers
Solution Approach 1:
The patent makes each peer universal by enabling them to function both as senders and receivers of deduplicated data independently. Peers can autonomously serve multiple functions including local deduplication, remote fingerprint retrieval, and dictionary population without requiring predetermined pairing relationships, thus scaling efficiently to numerous peers in the overlay network.
Solution Approach 2:
The patent implements self-service mechanisms where each peer independently manages its own dictionary and deduplication operations. Peers autonomously retrieve fingerprints from remote peers on-demand based on their specific needs, eliminating the requirement for centralized coordination or pre-established pairing relationships, thereby enabling scalable deployment across large overlay networks.
3Ease of manufacture
If fixed-length blocks are used for dictionary population, then processing simplicity is improved, but the shift problem worsens affecting fingerprint accuracy
Solution Approach 1:
The patent changes the block length parameter from fixed to variable based on semantic boundaries in the data stream. Instead of uniformly dividing data into fixed-length blocks, the system identifies natural segment boundaries (such as protocol delimiters or content structure markers) and creates variable-length blocks that align with these boundaries, thereby maintaining fingerprint accuracy while preserving processing simplicity through rule-based segmentation.
Data Source
AI summary
Stream-based data deduplication is provided in a multi-tenant shared infrastructure but without requiring “paired” endpoints having synchronized data dictionaries. Data objects processed by the dedupe functionality are treated as objects that can be fetched as needed. As such, a decoding peer does not need to maintain a symmetric library for the origin. Rather, if the peer does not have the chunks in cache that it needs, it follows a conventional content delivery network procedure to retrieve them. In this way, if dictionaries between pairs of sending and receiving peers are out-of-sync, relevant sections are then re-synchronized on-demand. The approach does not require that libraries maintained at a particular pair of sender and receiving peers are the same. Rather, the technique enables a peer, in effect, to “backfill” its dictionary on-the-fly. On-the-wire compression techniques are provided to reduce the amount of data transmitted between the peers.