Small File Restore Performance in Deduplication File System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication storage systems face performance issues during the restoration of small files, as they do not effectively utilize prefetch mechanisms, leading to slower restoration times compared to large file workloads.
Innovation Solution
The implementation of a queue-based mechanism within the Data Domain application programming interface (API) to prioritize and prefetch small file read requests, allowing them to be processed in batches and cached, thereby improving restoration performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If small files are restored using traditional read-ahead mechanisms, then the restoration process is simple, but the restoration speed is slow due to lack of prefetch advantages
Solution Approach 1:
The system performs preliminary actions by issuing prefetch reads for small files before they are actually requested. The backup application issues prefetch read requests for small files at predetermined intervals, and the file system caches these files in memory before they are needed, eliminating the need to read from disk during actual restoration.
Solution Approach 2:
The restoration process is segmented into different handling paths based on file size. Small files (below threshold) are processed through a specialized prefetch mechanism, while large files use traditional read-ahead. The system also segments the prefetch process into multiple threads that can operate in parallel, with a queue managing the distribution of prefetch tasks.
2Productivity
If multiple prefetch requests are issued simultaneously, then prefetch efficiency is improved, but system resource consumption increases
Solution Approach 1:
The system maintains continuous useful action by keeping multiple prefetch threads running simultaneously, each issuing prefetch requests continuously for their assigned files. The queue ensures that prefetch actions are continuous and uninterrupted, with threads constantly working to prefetch files before they are needed.
Solution Approach 2:
The system dynamically adjusts the number of active prefetch threads and the rate of prefetch requests based on system conditions. The queue manages the dynamic distribution of prefetch tasks, and the system can adapt the level of parallelism based on available resources and workload characteristics.
Data Source
AI summary
Embodiments of small file restore process in deduplication file system wherein restoration requires issuing a read request within an I/O request to the file system. The process places the small files in a prefetch queue such that a combined size of the small files meets or exceeds a size of the prefetch queue as defined by a prefetch horizon. A queue processor issues a read request for the first file in the queue, scans the prefetch queue to find a read request for a file at the prefetch horizon, and prefetches the file at the prefetch horizon. The prefetch queue essentially constitutes a hint from the client that a read I/O is imminent for purposes of filling the read-ahead cache and preventing a need to issue a blocking I/O operation.