File Handler Disassociates Prefetch and Read Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication storage systems face inefficiencies in restoring small files due to the lack of prefetch advantages and strict in-order processing of Quality of Service (QoS) operations, leading to slow restoration performance and manual prioritization errors.
Innovation Solution
Implementing a queue-based mechanism through extended Data Domain APIs to disassociate prefetch streams from read streams, allowing for prioritization and efficient processing of small file restores by routing read requests through a queue processor and utilizing multiple threads for optimized I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single read request issues a single prefetch request at a fixed offset, then the prefetch mechanism is simple to implement, but small file restoration performance deteriorates because prefetch advantages are not obtained
Solution Approach 1:
The patent segments the prefetch mechanism into multiple independent prefetch requests, each targeting different offsets within the read-ahead cache. Instead of a single prefetch at a fixed offset, the system issues multiple prefetch requests with varying offsets to improve hit probability for small files while maintaining manageable complexity through structured offset calculation
Solution Approach 2:
The patent implements preliminary action by pre-calculating and issuing multiple prefetch requests before the actual read operation. The system determines optimal prefetch offsets in advance and issues these prefetch requests to the storage system, ensuring that data is ready when needed without waiting for sequential reads to complete
2Reliability
If strict in-order processing is used for prefetch operations, then processing is straightforward and reliable, but Quality of Service operations become problematic and cannot prioritize certain backup files
Solution Approach 1:
The patent segments the processing pipeline into separate prefetch processing and read processing streams. The prefetch stream handles data retrieval independently, while the read stream handles actual data requests. This segmentation allows QoS algorithms to operate on the read stream without interfering with prefetch operations, enabling prioritization while maintaining processing reliability
Solution Approach 2:
The patent introduces an intermediary queue structure that mediates between prefetch operations and read operations. This queue allows out-of-order processing by buffering prefetch results and delivering them to read requests in the appropriate order, enabling QoS prioritization while maintaining data integrity and processing reliability
3Adaptability or versatility
If manual prioritization is used to restore specific backup files, then QoS control is achieved, but the process becomes error-prone and time-consuming
Solution Approach 1:
The patent implements self-service by enabling the storage system to automatically apply QoS policies and prioritize backup file restoration without manual intervention. The system reads prioritization rules from configuration data and autonomously determines the restoration order, eliminating manual prioritization operations while maintaining accurate QoS control
Solution Approach 2:
The patent incorporates feedback mechanisms where the system continuously monitors restoration progress and adjusts processing priorities based on QoS policies. This feedback loop allows the system to automatically prioritize critical backup files and optimize restoration operations without requiring manual reconfiguration or intervention
Data Source
AI summary
Embodiments of small file restore process in deduplication file system wherein restoration requires issuing a read request within an I/O request to the file system. The process places the files in a prefetch queue such that a combined size of the files meets or exceeds a size of the prefetch queue as defined by a prefetch horizon. A file handler disassociates prefetch streams from read streams. The handler prefetches the read operations and stores them in memory. The stream corresponding to a read will only open as the read requests hit the queue processor. As a result, the stream usage is very low, since the I/O, worker threads and the read streams are disassociated from each other.