Hierarchical Hash File System Differencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies are inefficient in describing and discovering metadata changes between datasets, particularly in mirroring and backup systems, as they either take too long to process or fail to provide human-readable, useful information about file-level changes.
Innovation Solution
A method and apparatus that compare hierarchical hashes of datasets to identify metadata differences without moving or copying the data, generating a human-readable report of changes, and can operate across different file systems, including write-anywhere and write-in-place systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If classic backup technologies traverse the entire file system to discover changes, then they can describe metadata changes between datasets, but the processing time becomes extremely slow
Solution Approach 1:
The patent segments the file system traversal into two distinct phases: a fast phase that uses hierarchical hashes to identify potential changes by comparing parent directory hashes, and a slow phase that performs detailed metadata comparison only for suspected changes. This segmentation allows the system to quickly eliminate unchanged directories without exhaustive metadata comparison, thereby reducing overall processing time while maintaining detection accuracy.
Solution Approach 2:
The patent introduces hierarchical hashes as an intermediary mechanism between the file system structure and the metadata comparison process. These hashes serve as a filtering layer that quickly identifies which directories may contain changes, allowing the system to avoid unnecessary detailed metadata comparisons in unchanged directories, thus resolving the contradiction between thorough detection and processing speed.
2Productivity
If differential or incremental backups are used to reduce media consumption, then backup efficiency improves, but discovering and describing changes remains equally slow as full-system backup
Solution Approach 1:
The patent segments the backup process into a fast change-discovery phase using hierarchical hash comparison and a subsequent detailed backup phase. By separating the change discovery function from the data transfer function, the system can quickly identify what has changed without performing the entire slow traversal process, enabling differential and incremental backups to actually run faster than traditional methods.
Solution Approach 2:
The patent performs preliminary change discovery using hierarchical hashes before initiating the actual backup process. This preliminary action identifies the scope of changes in advance, allowing the backup system to prepare optimized backup strategies (full, differential, or incremental) based on pre-analyzed change patterns, thereby reducing the time required during the actual backup execution.
3Productivity
If mirroring applications replicate changes efficiently between datasets, then data synchronization improves, but they cannot describe changes at a human-readable file level
Solution Approach 1:
The patent segments the change information into two levels: block-level changes identified through hierarchical hash comparison (for efficient synchronization) and file-level metadata changes identified through subsequent detailed comparison (for human-readable reporting). This segmentation allows mirroring applications to maintain fast synchronization while also generating meaningful user-level change descriptions.
Solution Approach 2:
The patent merges the strengths of block-level mirroring (fast change detection) with file-level metadata tracking (human-readable information) into a unified change description system. By combining hash-based block comparison with metadata attribute comparison, the system produces both efficient synchronization data and user-comprehensible change reports simultaneously.
4Measurement precision
If the Ting technique compares file names to identify differences, then it can list changed files between datasets, but it cannot describe relationships between differences and processing is expensive and slow
Solution Approach 1:
The patent segments the comparison process into hierarchical levels where parent directory hashes are compared first to identify which subdirectories contain changes, and only then are detailed file-level comparisons performed in the affected subdirectories. This segmentation avoids the expensive full-filesystem comparison of the Ting technique while maintaining the ability to describe detailed change relationships.
Solution Approach 2:
The patent introduces hierarchical hashes as an intermediary that provides a structured, multi-level view of changes. This intermediary structure allows the system to efficiently identify change locations and then describe relationships between differences (such as renames, moves, or modifications) without performing expensive full-comparison operations across the entire dataset.
Data Source
AI summary
A facility for comparing two datasets and identifying metadata differences between the two datasets irrespective of the manner in which the data is stored. In some embodiments, the facility includes a comparison unit and a catalog unit. The comparison unit compares a hierarchical hash of a first dataset with a hierarchical hash of a second dataset, the hierarchical hashes each including a plurality of hierarchical hash values, to identify differences in metadata of the first and second datasets by progressively comparing the hierarchical hash values of the first and second hierarchical hashes without comparing the metadata of the first and second datasets. The catalog unit generates a catalog of differences between the first and second datasets, the catalog indicating differences in metadata of the first and second datasets.