Storage System Restore Deduplication via Chunk Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network latency hinders the ability of storage systems to meet restore service level agreements when dealing with large files or numerous files, as it diverts resources and increases storage requirements at the content destination.
Innovation Solution
The storage system calculates chunk identifiers for data chunks and sends them in batches, using a data structure to track received chunks, allowing for deduplication and efficient resource management by referencing previously sent data chunks, thereby reducing the amount of data sent over the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data chunks are sent over the wide area network during restore operation, then complete file restoration is achieved, but network latency prevents service level agreement from being met
Solution Approach 1:
The system performs preliminary actions by sending file metadata and chunk identifiers before sending actual data chunks. The restore agent pre-configures the data structure with expected chunk identifiers, enabling it to anticipate and efficiently process incoming data without waiting for complete data transmission before beginning restoration operations.
Solution Approach 2:
The patent segments the restore operation into distinct phases: metadata transmission, chunk identifier sending, and data chunk transmission. It also segments data into batches with acknowledgments between batches, allowing the system to process and confirm receipt of portions of data independently, reducing the impact of overall transmission latency.
2Productivity
If data chunks are tracked using a data structure at the content destination, then deduplication efficiency is improved, but computing resources at the content destination are consumed
Solution Approach 1:
The patent introduces an intermediary mechanism where chunk identifiers and metadata are transmitted separately from actual data chunks. The restore agent uses these identifiers to build and query the data structure without needing to process complete data chunks, significantly reducing the computing resources required for tracking and deduplication operations.
Solution Approach 2:
Instead of copying and storing complete data chunks in the tracking data structure, the system uses lightweight copies represented by chunk identifiers and metadata. This allows efficient tracking and deduplication decisions to be made using minimal computational resources while maintaining the ability to identify and deduplicate actual data chunks.
3Quantity of substance
If large amounts of data are transmitted over the network, then complete file restoration is achieved, but the impact of network latency increases
Solution Approach 1:
The system implements periodic action by transmitting data in batches with acknowledgments sent after each batch. This allows the restore operation to progress in discrete steps, with the ability to track completion and manage network resources efficiently, reducing the cumulative impact of latency across the entire large data transmission.
Solution Approach 2:
By sending metadata, file information, and chunk identifiers before actual data chunks, the system prepares the restore agent in advance. This preliminary action allows the agent to pre-allocate resources and configure data structures, so that when large volumes of data arrive, the processing overhead is minimized and network latency impact is reduced.
Data Source
AI summary
Data chunks sent to a content destination are tracked. It is determined whether content of a portion of an object to be sent to the content destination matches one of the data chunks previously sent to the content destination. In response to the determination that the portion of the object to be sent to the content destination matches one of the data chunks previously sent to the content destination, a source reference to the matching previously sent data chunk where the content destination can locally obtain the content for the portion of the object is provided to the content destination.