Adaptive Prefetch Queue for Small File Restore Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication file systems face performance issues during small file restoration due to the inefficiencies in prefetching mechanisms, which are not adequately supported by existing APIs, leading to slower restoration times compared to large file workloads.
Innovation Solution
The implementation of an adaptive prefetch mechanism using a queue-based system within the DDBoost API, which allows for prioritization and queuing of small file read requests, enabling efficient prefetching and reducing dependency on disk read latency by utilizing a queue processor to manage and optimize read requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a simple read-ahead prefetch mechanism is used, then the implementation is straightforward, but small file restoration performance deteriorates significantly
Solution Approach 1:
The patent implements preliminary action by issuing prefetch requests before the actual read requests. The queue processor receives read requests, calculates appropriate prefetch offsets based on file size and position, and issues prefetch requests to the storage system in advance. This allows data to be loaded into the read-ahead cache before it is needed, significantly improving small file restoration performance by reducing disk read latency during the actual restore operation.
Solution Approach 2:
The patent applies dynamics by making the prefetch mechanism adaptive rather than static. The queue processor dynamically adjusts prefetch parameters based on the specific characteristics of each read request, including file size, read offset, and current system state. This dynamic adaptation allows the system to optimize prefetch behavior for different workloads, ensuring good performance for both small and large files while avoiding the pitfalls of fixed prefetch strategies.
2Adaptability or versatility
If existing public protocols are used, then compatibility is maintained, but the ability to trigger internal prefetch for small files is lost
Solution Approach 1:
The patent uses an intermediary approach by introducing a queue processor that sits between the client application and the storage system. This queue processor receives standard read requests from clients using existing protocols, translates them into optimized internal operations including prefetch requests, and manages the actual data retrieval. The queue processor acts as a mediator that maintains protocol compatibility at the client interface while enabling advanced prefetch functionality internally, thus resolving the contradiction between compatibility and performance.
3Device complexity
If disk read operations are performed sequentially, then the system is simple to implement, but restoration time increases due to disk read latency
Solution Approach 1:
The patent implements preliminary action by issuing prefetch requests before the actual read operations. The queue processor receives read requests, calculates appropriate prefetch offsets, and issues prefetch requests to the storage system in advance. This allows data to be loaded into the read-ahead cache before it is needed, significantly reducing disk read latency during the actual restore operation and decreasing overall restoration time.
Solution Approach 2:
The patent applies continuity of useful action by maintaining overlapping operations. While the client is waiting for read requests to complete, the queue processor continuously issues prefetch requests and manages read-ahead cache operations. This ensures that disk read operations are continuously productive rather than idle, with prefetch requests being issued in a continuous manner to keep the storage system busy performing useful work and reducing overall restoration time.
Data Source
AI summary
Embodiments of file restores in a Data Domain (DD) file system implementing a DD Bandwidth Optimized Open Storage Technology (DDBoost) library that translates application read and write request to DDBoost application program interfaces (APIs). A prefetch queue processor creates an intent to read the file. The application passes the file handle of the file, and the destination handle where the data must be read into. As the queue is processed, the prefetch for the request (handle/offset/length) is passed to the file server. The filesystem processes the request to open the file to load into memory. As the read request for the same file reaches the filesystem the file data is read from memory for writing to the destination handle. An extended DDBoost API expression is defined to pass the current path and destination path to the application.