Moving Window Deduplication for High-Throughput Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing key-value storage systems are not designed for deduplication and require high costs and complex processes to manage duplicates, leading to inefficiencies and inconsistencies in high throughput streaming systems.
Innovation Solution
A streaming API that allows for in-line deduplication by maintaining a moving window of insert identifiers in memory, determining duplicates in real-time, and storing deduplication information alongside data, reducing costs and ensuring read-after-write consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If key-value storage systems are used for deduplication, then data can be stored, but storage cost increases and system complexity increases
Solution Approach 1:
The patent extracts the deduplication function from the key-value storage system and implements it separately using a moving window approach in memory. This separates the storage function from the deduplication function, allowing each to be optimized independently and reducing overall system complexity while maintaining deduplication accuracy.
Solution Approach 2:
The patent introduces a moving window data structure as an intermediary between the data writing process and the deduplication check. This intermediary structure efficiently tracks recent insert identifiers without requiring complex key-value storage operations, simplifying the system while ensuring reliable deduplication.
2Reliability
If key-value storage systems are used for deduplication, then data can be stored, but storage cost increases
Solution Approach 1:
The patent uses a moving window approach that only stores recent insert identifiers for a limited time period. This disposable approach to tracking data eliminates the need for expensive long-term storage of all data in key-value systems, reducing storage costs while maintaining sufficient deduplication accuracy for the operational timeframe.
Solution Approach 2:
The patent implements a dynamic moving window that automatically adjusts its scope based on the time window parameter. This dynamic structure optimizes storage usage by only retaining data that needs to be tracked for deduplication purposes, reducing overall storage requirements compared to static key-value storage systems.
3Reliability
If deduplication happens on read in key-value storage, then existing data can be checked, but read latency increases and throughput decreases
Solution Approach 1:
The patent performs deduplication checks in advance during the write operation by checking the moving window before inserting data. This preliminary action ensures that duplicates are identified and handled before they enter the storage system, eliminating the need for expensive read operations to check for duplicates and improving overall throughput.
Solution Approach 2:
The patent makes deduplication a continuous process that occurs automatically during every write operation through the moving window mechanism. This continuous checking during writes eliminates the need for separate read operations to verify duplicates, maintaining continuous productive action throughout the data stream and improving throughput.
4Reliability
If data is kept in temporary buffer for deduplication, then deduplication can be performed, but data management complexity increases
Solution Approach 1:
The patent extracts the data management complexity from the deduplication process by using a simple moving window structure that automatically manages its own contents. This extracted approach handles duplicate detection without requiring complex buffer management services, reducing overall data management complexity while maintaining accuracy.
Solution Approach 2:
The moving window data structure is self-managing, automatically adding and removing insert identifiers based on the time window parameter without requiring external service intervention. This self-service capability eliminates the need for complex buffer management services and reduces data management complexity while ensuring reliable deduplication.
5Productivity
If asynchronous add and extract processes are used, then throughput can be improved, but race conditions occur and consistency is lost
Solution Approach 1:
The patent performs the deduplication check as a preliminary action during the write operation before data is committed. This preliminary checking ensures that consistency is maintained from the start, eliminating race conditions that would occur with asynchronous add and extract processes. The moving window is updated atomically during the write process, ensuring consistent state.
Solution Approach 2:
The patent uses the moving window structure as a cushioning mechanism that prevents race conditions by providing a controlled, atomic way to track insert identifiers. This beforehand cushioning approach ensures that even with asynchronous operations, the deduplication logic maintains consistent state by checking and updating the moving window in a controlled manner.
Data Source
AI summary
The present disclosure describes a service which provides primary in-line deduplication. A streaming application program interface (API) may allow for streaming records into a storage system with high throughput and low latency. As part of this process, the API allows user to add identifiers as a field used for data deduplication. The deduplication service keeps a moving window of the identifiers in memory and does in-line deduplication by quickly determining whether data is a duplicate. Keeping only deduplication keys in memory reduces the cost of running the service. Moreover, the real-time nature of the moving window approach allows for storing deduplication information alongside the data and accessing it immediately on read. In this regard, read after write consistency is supported, and costs are reduced.