Large Content File Optimization via Parallel Metadata Tree Splitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems face challenges in efficiently managing and modifying large content files, as the process of creating and updating file trees for such files is bottlenecked by the requirement for a single write-lock, leading to sequential operations rather than parallel processing.
Innovation Solution
The implementation of a tree data structure that allows for the generation of multiple file trees for large content files, enabling parallel processing of file metadata operations across multiple storage nodes, thereby reducing the time required for file system operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single write-lock is used to modify file metadata for large content files, then data consistency is maintained, but the time required for metadata operations increases significantly
Solution Approach 1:
The patent divides the file metadata into multiple segments distributed across different storage nodes. Instead of locking the entire metadata structure, individual metadata segments can be modified independently by different storage nodes in parallel, reducing the time required for metadata operations while maintaining data consistency through coordinated updates.
Solution Approach 2:
The patent introduces a distributed dimension to metadata storage by spreading metadata across multiple storage nodes rather than concentrating it in a single location. This dimensional change enables parallel processing of metadata operations across different nodes, transforming a sequential operation into a parallel one that reduces overall operation time.
2Device complexity
If sequential processing is used for file tree operations, then simplicity is maintained, but productivity decreases for large content files
Solution Approach 1:
The patent segments file tree operations into independent units that can be executed in parallel across different storage nodes. Each storage node handles specific portions of the file tree metadata independently, enabling concurrent processing that significantly improves productivity for large content files without requiring complete sequential processing.
Solution Approach 2:
The patent implements dynamic processing where the system can adapt between sequential and parallel processing modes based on the size and characteristics of the content files. For large content files, parallel processing is activated to improve productivity, while simpler sequential processing is used for smaller files, optimizing the balance between complexity and performance.
Data Source
AI summary
Maintenance is performed to determine one or more content files associated with a stored backup snapshot having a size that is greater than a threshold size. It is determined that the size of a content file of the one or more content files is greater than the threshold size. In response to determining that the size of the content file is greater than the threshold size, a new tree data structure is generated and a component file metadata structure corresponding to the content file is split into a plurality of component file metadata structures for the content file.