Queue-Based File Cloning with Priority Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems face inefficiencies in creating and managing copies of files, particularly in de-duplicating data and metadata, and in handling file cloning and checkpointing processes, which can lead to complexities in I/O operations and data integrity.
Innovation Solution
The implementation of a file cloning mechanism that creates clones by referencing source object elements like indirect and direct onodes and data blocks, allowing for both read-only and mutable clones without copying user data or metadata blocks initially, and a checkpoint mechanism that manages file system changes and cloning operations efficiently through priority-based queues and asynchronous processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional file copying is used to create file clones, then complete copies of files are created, but the storage space consumption and copying time increase significantly
Solution Approach 1:
The patent implements a copying mechanism that creates file clones by establishing reference structures (onodes, indirect onodes, and data block pointers) rather than physically copying file data. The clone file system object contains metadata structures that point to the same data blocks as the source file, enabling instant cloning regardless of file size. This resolves the contradiction by making copying speed independent of file size while avoiding duplicate data storage.
Solution Approach 2:
The patent segments the file structure into metadata components (onodes, indirect onodes, data block pointers) and data blocks. By cloning only the metadata structures and sharing data blocks between source and clone, the system achieves rapid cloning without proportional storage consumption. The segmentation allows the clone to reference shared data blocks while maintaining independent metadata for clone-specific operations.
2Quantity of substance
If de-duplication is performed after creating full file copies, then storage efficiency is improved, but the complexity of I/O operations and data management increases
Solution Approach 1:
The patent performs de-duplication preliminarily by creating clones that share data blocks with the source file from the moment of cloning. Instead of creating full copies and then de-duplicating, the system establishes shared data block references upfront. The metadata structures (onodes) contain pointers to data blocks with reference counting, automatically managing de-duplication without adding I/O complexity. When data blocks are modified, the system automatically creates new blocks only when necessary, maintaining storage efficiency while keeping I/O operations simple.
3Reliability
If checkpoints are stored frequently to ensure data integrity, then data recovery capability is improved, but the disk write operations and system overhead increase
Solution Approach 1:
The patent merges checkpoint operations with the existing file cloning and metadata update mechanisms. Checkpoints are integrated into the onode reference structure, allowing the system to leverage the same metadata update paths for both cloning and checkpointing. This consolidation reduces overhead by using unified metadata structures (onodes, indirect onodes) for both purposes, eliminating separate checkpoint data structures and reducing disk write operations.
Solution Approach 2:
The onode metadata structure serves multiple functions: it manages file cloning references, tracks data block sharing, and implements checkpointing simultaneously. The reference counting mechanism in onodes universally handles both clone management and checkpoint verification. This multi-functionality reduces system overhead by eliminating the need for separate checkpoint data structures and operations, while maintaining data integrity through the versatile onode structure.
4Reliability
If all metadata blocks are copied when creating file clones, then clone independence is improved, but the cloning time and computational overhead increase
Solution Approach 1:
The patent applies local quality by creating independent metadata structures (onodes, indirect onodes) only where necessary for clone-specific operations, while sharing data blocks where identical. The clone's onode contains independent pointers and reference counts, providing local independence for metadata management, while the shared data blocks provide global efficiency. This selective independence approach minimizes cloning time by avoiding unnecessary metadata duplication while ensuring clone independence where required.
Data Source
AI summary
A file cloning process allows for quickly cloning files within a filesystem, such as when a user makes a copy of a directory containing files and subdirectories of files. A priority based main queue is filled by a depth first traversal of the directory to be copied with work items corresponding to files and subdirectories. Worker threads refer to the main queue for file cloning, subdirectory copying and directory traversal. Files that have not been modified since a most recent checkpoint are cloned without issuing a checkpoint request. Files that have been modified since the most recent checkpoint are moved to a delay queue. Once a checkpoint of the file system completes, the files in the delay queue are then moved to the main queue to retry cloning thereof. A secondary queue is also provided to prevent worker threads from becoming deadlocked in the main queue.