Deduplication Storage Data Index for Batch Transfer Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication storage systems incur significant bandwidth and processing inefficiencies, as well as financial costs, due to the need for frequent small transfers when updating data in remote storage services, which are charged based on the number of individual transfers.
Innovation Solution
Implementing a deduplication storage system that buffers data updates in memory according to arrival order and transfers them as a single container entity group when reaching a threshold size, along with generating a data index to manage container indexes, allowing for controlled transfers and efficient housekeeping of stale data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data updates are transferred individually to remote storage, then data consistency is maintained, but the number of transfers increases significantly, leading to higher financial costs and bandwidth inefficiency
Solution Approach 1:
The patent merges multiple individual data update transfers into a single batched transfer operation. The storage controller accumulates data updates in a buffer and transfers them together as one unit to remote storage, reducing the total number of transfer operations while maintaining data consistency through structured batch processing.
Solution Approach 2:
The patent performs preliminary actions by buffering and preparing data updates before transfer. The storage controller collects and organizes multiple data updates in advance within a buffer, so that when a transfer occurs, it contains a consolidated set of updates rather than individual sequential transfers.
2Productivity
If data updates are buffered and transferred as a single container entity group, then transfer efficiency improves and costs reduce, but system complexity increases due to buffer management and container index tracking
Solution Approach 1:
The patent segments the buffer management into distinct components: a data buffer for storing update content, a container index structure for tracking updates, and a transfer controller for managing batch operations. This segmentation organizes the complexity into manageable, independent modules with clear responsibilities.
Solution Approach 2:
The patent introduces a container entity group as an intermediary structure between the buffer and remote storage. This container index acts as a mediator that organizes multiple data updates into a coherent transfer unit, simplifying the interface between buffer management and transfer operations while reducing direct complexity.
3Reliability
If frequent small transfers are performed to maintain updated data in remote storage, then data freshness is ensured, but bandwidth is wasted and financial costs increase due to transfer-based pricing
Solution Approach 1:
The patent combines multiple small data updates into a single larger transfer operation. By accumulating updates in a buffer and transferring them together as one container entity group, the system reduces the total number of transfer operations and minimizes bandwidth waste while still ensuring remote storage receives all updates.
Solution Approach 2:
The patent maintains continuous buffer accumulation of data updates without interruption. The buffer continuously receives and stores incoming data updates, and the transfer mechanism continuously monitors buffer contents, ensuring that useful action (data collection and preparation) continues without gaps, improving overall system efficiency.
Data Source
AI summary
Example implementations relate to metadata operations in a storage system. An example includes a machine-readable medium storing instructions that upon execution cause a processor to: receive a data stream to be stored in persistent storage of a deduplication storage system; store data units of the data stream in a container entity group object according to arrival time, where the data units of the container entity group object are referenced by a plurality of container indexes; generate a data index to list each container index that references at least one data unit included in the container entity group object; and in response to a determination that the total size of the container entity group object exceeds the threshold size, transfer the container entity group object from memory to the persistent storage.