Filesystem Replication Namespace Collision Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file-based storage systems face issues with namespace collisions during incremental replication, such as failed directory moves and incorrect unlink operations, due to the reversal of parent-child relationships and stale files/directories in snapshots-based filesystems.
Innovation Solution
The technology processes detected changes by deferring operations like link, unlink, and rename, using unique non-conflicting names to resolve namespace collisions before applying changes, thereby avoiding temporary directories and associated performance issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single temp directory is used in the root directory of the target system, then namespace collisions can be temporarily resolved, but performance issues arise due to namespace contention and multi-locker ping pong
Solution Approach 1:
The patent divides the single temp directory into multiple worker-specific temp directories. Each worker process gets its own isolated temp directory (e.g., /target/.replication/worker_<id>/tmp), eliminating namespace contention between workers. This segmentation resolves the contradiction by maintaining reliability through isolated temp spaces while improving productivity by removing locking conflicts and ping-pong scenarios.
2Productivity
If multiple temp directories are created using hash to spread load, then performance issues are reduced, but complexity increases due to extra indirection and file descriptors
Solution Approach 1:
The patent implements dynamic worker process management where each worker process dynamically creates its own temp directory at runtime based on its worker ID. This dynamic approach improves productivity by providing isolated spaces while avoiding the static complexity of pre-configured hash-based directory structures. The system adapts to the number of workers rather than requiring complex routing logic.
Solution Approach 2:
Each worker process self-creates and manages its own temp directory without central coordination or complex routing. The worker process independently handles its own temporary file operations, eliminating the need for centralized temp directory management and reducing overall system complexity while maintaining performance.
3Reliability
If operations are deferred and workers wait for conditions to fix, then namespace collisions are resolved, but time is lost due to waiting and repeated attempts
Solution Approach 1:
The patent performs preliminary actions by having each worker process pre-create its own isolated temp directory before attempting any operations. This preliminary setup prevents namespace collisions from occurring in the first place, eliminating the need for deferred operations and retries. Operations proceed immediately without waiting, maintaining both reliability and time efficiency.
Data Source
AI summary
The described technology is generally directed towards resolving namespace collisions on a target system when applying incremental changes (e.g., between snapshots) as part of an incremental replication from a source system. Detected changes to a filesystem namespace include file link, directory move and file and directory unlink operations. The changes are processed to recognize and overcome failures and other issues, such as to not process an unlink before a corresponding link, which can otherwise result in accidental deletion of a file. Deferred operations are maintained for later processing, including deferred directory moves for directory rename failures, deferred unlink operations and name collision fixes. After initial processing, the deferred move list is iteratively processed to overcome directory rename failures, the deferred unlink list is processed after performing corresponding link operations, and the name fix list is processed to rename previously colliding names.