Filesystem Sync via Directory Chunk Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing filesystem synchronization methods over a network either consume maximum network bandwidth or require significant local storage, and are prone to errors, especially in incremental synchronization approaches.
Innovation Solution
The method involves creating a canonical representation of metadata for each file, sorting directory elements, dividing them into chunks, hashing, and using a hash cache to efficiently compare and synchronize filesystem information across the network, minimizing both network bandwidth usage and local storage requirements while reducing error prone journaling complexities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a full synchronization approach is used, then network bandwidth is maximized for synchronization, but local storage requirements remain minimal
Solution Approach 1:
The patent segments the filesystem metadata into directory chunks, where each chunk represents a portion of the directory structure. This segmentation allows the system to transmit only the necessary metadata portions over the network rather than performing full synchronization, thereby reducing network bandwidth consumption while maintaining synchronization effectiveness.
2Loss of energy
If an incremental synchronization approach using a catalog database is used, then network bandwidth consumption is reduced, but local storage requirements increase
Solution Approach 1:
The patent uses hash values as lightweight, temporary identifiers for directory chunks instead of maintaining a comprehensive catalog database. These hash values serve as disposable markers that indicate whether chunks need synchronization, eliminating the need for extensive local storage while enabling incremental synchronization behavior.
3Quantity of substance
If a journal-based incremental approach is used, then local storage requirements are reduced, but synchronization error risk increases
Solution Approach 1:
The patent implements a feedback mechanism where hash values of directory chunks are compared between client and server to determine synchronization needs. This feedback loop ensures that only chunks with mismatched hashes (indicating changes) are transmitted for synchronization, maintaining accuracy without requiring journal-based tracking.
4Measurement precision
If directory elements are processed individually, then synchronization precision is maintained, but network bandwidth consumption increases
Solution Approach 1:
The patent merges multiple directory elements into directory chunks that are processed collectively. By grouping directory entries into chunks and computing hash values for entire chunks rather than individual files, the system maintains synchronization precision at the chunk level while significantly reducing the overhead associated with processing individual directory elements separately.
Data Source
AI summary
Transmitting filesystem changes over a network is disclosed. A hash of data comprising a chunk of directory elements comprising one or more consecutive directory elements in a set of elements sorted in a canonical order is computed at a client system. One or more directory elements comprising the chunk are sent to a remote server in the event it is determined based at least in part on the computed hash that corresponding directory elements as stored on the remote server are not identical to the directory elements comprising the chunk as stored on the client system.