Multi-Phase Sparse-File Recovery for Lower Cloud Read Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud environments, reading or restoring files from sparse files stored in discontinuous locations leads to increased latency and reduced throughput due to the need for numerous small, scattered read requests, which cannot be executed in parallel.
Innovation Solution
The data management system employs a two-phase read operation to reduce latency and improve throughput by identifying target address ranges and using index information to group data blocks within sparse files, allowing for fewer, larger, and potentially parallel read requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored as sparse files in discontinuous locations in cloud environment, then storage efficiency is improved, but read latency increases and throughput decreases due to numerous small scattered read requests
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing metadata including address ranges and data block locations when snapshots are created. This metadata is used to proactively group data blocks into contiguous ranges before read operations occur, allowing the system to issue optimized read requests without real-time computation delays.
Solution Approach 2:
The patent merges scattered data blocks into contiguous address ranges by using metadata to identify and group data blocks that belong to the same file. Multiple small read requests for discontinuous locations are combined into fewer, larger read requests for contiguous ranges, enabling parallel execution and reducing total read time.
2Measurement precision
If numerous small read requests are issued to retrieve data from sparse files, then data retrieval accuracy is maintained, but system complexity increases and throughput decreases
Solution Approach 1:
The patent introduces metadata as an intermediary layer between the storage system and read operations. This metadata contains pre-processed information about data block locations, address ranges, and file mappings, allowing the system to translate high-level file read requests into optimized low-level read operations without complex real-time computation.
Solution Approach 2:
The system performs preliminary processing during snapshot creation to organize data into structured metadata formats. This pre-computation of address ranges and block groupings eliminates the need for complex runtime decision-making, reducing system complexity while maintaining retrieval accuracy.
3Measurement precision
If small read requests are used to access specific address ranges, then precise data access is achieved, but parallel execution capability is lost and latency increases
Solution Approach 1:
The patent merges multiple small address ranges into larger contiguous ranges that can be read in parallel. By grouping data blocks that belong to the same file into contiguous ranges, the system maintains precise data access while enabling parallel execution of read requests, thereby increasing throughput.
Solution Approach 2:
The system transitions from one-dimensional sequential read requests to multi-dimensional parallel read operations by organizing data into contiguous ranges that can be accessed simultaneously across multiple channels or threads, maintaining precision while boosting productivity.
Data Source
AI summary
A method for recovering files from a filesystem stored across sparse files in a cloud environment is described. According to the method, a data management system may receive a request to read the files. The data management system may identify one or more target address ranges corresponding to the files indicated via the request. The data management system may read index information for the sparse files in the cloud environment. The index information may indicate respective address ranges for data blocks within the sparse files. The data management system may identify one or more data blocks within one or more sparse files as corresponding to address ranges that overlap with the one or more target address ranges based on the index information. The data management system may transmit, to the cloud environment, one or more read requests for the identified one or more data blocks.