Global Read-Ahead Pool for Faster Deduplicated File Restoration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Restoring files from large-scale deduplicated filesystems is time-consuming due to the need to fetch data from storage, which is inefficient and lacks effective techniques to reduce this time.
Innovation Solution
Implementing a read-ahead cache mechanism that prefetches data from the storage layer into memory, allowing subsequent read requests to be serviced from the cache rather than the storage layer, and utilizing a distributed architecture with similarity groups to manage and distribute file restoration tasks across multiple services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is fetched from storage layer for file restoration, then file restoration can be completed, but the process is time-consuming and inefficient
Solution Approach 1:
The system performs preliminary actions by opening multiple internal read-ahead streams to prefetch data from the storage layer into a read-ahead cache before actual restoration requests are processed. This advance data preparation significantly reduces the time required when restoration is actually needed, as data is already available in memory rather than being fetched from storage during the restoration process.
2Speed
If multiple internal read-ahead streams are opened to prefetch data, then restoration speed is improved, but system complexity increases
Solution Approach 1:
The system segments the file restoration process into multiple independent read-ahead streams, where each stream handles a portion of the data prefetching. This segmentation allows parallel processing of data retrieval operations, improving overall prefetching speed while maintaining manageable complexity through modular stream management. The read-ahead cache is also segmented to store data from different streams independently.
3Productivity
If read-ahead cache is used to service restore requests, then restoration efficiency is improved, but memory resources are consumed
Solution Approach 1:
The system applies partial action by opening a limited number of internal read-ahead streams (e.g., 3-5 streams) rather than attempting to prefetch all data simultaneously. This partial approach balances memory resource consumption with restoration efficiency, loading only the necessary portions of data into the read-ahead cache based on the number of streams configured, rather than exhaustively preloading entire files or datasets.
Data Source
AI summary
Access object (AOB) and deduplication object (DOB) services of a deduplication filesystem are provisioned across a cluster. A client-side library receives a request to restore a file, the file being divided into chunks. Read-ahead buffers at the AOBs are grouped to form a global read-ahead pool. Read-ahead IO objects corresponding to the offset ranges in the file are prefetched to populate the global read-ahead pool, the read-ahead IO objects being read by the DOBs from a storage layer of the filesystem. A key-value store is created to track the read-ahead IO objects. The key-value store includes a key and a value, the key being based on a file handle of the file, offset within the file, and size of a buffer to read, and the value identifying a location of the buffer at an AOB. The request is serviced using the global read-ahead pool.