Sparse File Recovery Using Metadata-Guided Parallel Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud environments, reading or restoring files from sparse files stored in discontinuous locations leads to increased latency and decreased throughput due to the need for numerous small, scattered read requests, which cannot be executed in parallel.
Innovation Solution
The data management system employs a two-phase read operation to reduce latency and improve throughput by identifying target address ranges and using metadata to group data blocks within contiguous sparse files, allowing for fewer, larger, and parallelizable read requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in sparse files across multiple discontinuous locations in cloud environment, then storage efficiency is improved, but read latency increases and throughput decreases due to numerous scattered read requests
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing metadata that maps logical file addresses to physical data block locations in the cloud. When a read request arrives, the system uses this pre-prepared metadata to directly identify and retrieve data blocks without performing scattered sequential reads, thus reducing read latency while maintaining storage efficiency
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the logical file system and physical cloud storage. This metadata acts as a mediator that translates logical read requests into optimized physical read operations, enabling efficient data retrieval from discontinuous cloud locations without requiring numerous small read requests
2Quantity of substance
If data is stored in sparse files across multiple discontinuous locations, then storage efficiency is improved, but read operation complexity increases due to scattered access patterns
Solution Approach 1:
The metadata structure serves as an intermediary that simplifies read operations by providing a direct mapping from logical addresses to physical data block locations. This intermediary layer abstracts the complexity of discontinuous cloud storage, allowing read operations to be performed efficiently without directly managing scattered data locations
Solution Approach 2:
The patent extracts the complexity of managing discontinuous data locations by separating it into a dedicated metadata structure. This extracted metadata handles all the complexity of mapping and locating data blocks, leaving the read operation itself simple and straightforward while maintaining storage efficiency
3Loss of information
If numerous small read requests are issued to retrieve data from cloud, then data retrieval completeness is improved, but throughput decreases due to inability to execute requests in parallel
Solution Approach 1:
The system performs preliminary action by pre-identifying all required data block locations through metadata lookup before issuing read requests. This allows the system to batch and parallelize read requests for multiple data blocks simultaneously, improving throughput while ensuring complete data retrieval without missing any blocks
Solution Approach 2:
The patent merges multiple small read requests into fewer, larger parallel read operations by using metadata to identify all required data blocks upfront. This combining approach maintains data retrieval completeness while enabling parallel execution, thus significantly improving throughput compared to sequential small reads
Data Source
AI summary
A method for recovering files from a filesystem stored across sparse files in a cloud environment is described. According to the method, a data management system may receive a request to read the files. The data management system may identify one or more target address ranges corresponding to the files indicated via the request. The data management system may read index information for the sparse files in the cloud environment. The index information may indicate respective address ranges for data blocks within the sparse files. The data management system may identify one or more data blocks within one or more sparse files as corresponding to address ranges that overlap with the one or more target address ranges based on the index information. The data management system may transmit, to the cloud environment, one or more read requests for the identified one or more data blocks.