File Synchronization Logging for Faster Data and Metadata Migration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data migration methods for large storage systems take a long time due to the generation of numerous synchronization commands, leading to prolonged user access denial, and lack precision in synchronization accuracy.

Innovation Solution

A computer-implemented method for synchronizing file system items between a source and destination storage system using a synchronization log with metadata and data sections, recording changetimes and integrity levels to generate commands that ensure data and metadata integrity, thereby optimizing synchronization efficiency and accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a single copy of all data is performed during data migration, then data completeness is ensured, but synchronization time increases significantly (days, weeks or months)

Engineering Contradiction:

Improvedata completenessVSAvoidsynchronization time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The synchronization process is divided into multiple stages: initial synchronization (baseline) and multiple incremental synchronizations. The initial sync copies all data to establish a baseline, while subsequent incremental syncs only transfer changed data based on recorded changetimes, significantly reducing total synchronization time while maintaining completeness

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

During the initial synchronization, changetimes of all file system items are recorded in a synchronization log before the actual data transfer begins. This preliminary recording enables subsequent incremental synchronizations to use these pre-recorded timestamps to identify only the changed data, avoiding redundant transfers

Inventive Principle:

Principle #10Preliminary action

2Reliability

If numerous synchronization commands are generated to ensure complete data transfer, then data integrity is maintained, but resource consumption and processing time increase

Engineering Contradiction:

Improvedata integrityVSAvoidsynchronization efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The synchronization log records changetimes from previous synchronization operations and uses this feedback information to determine what data needs to be transferred in subsequent operations. This feedback mechanism prevents redundant commands by comparing current state with recorded history, generating fewer commands while maintaining integrity

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system changes the parameter of comparison from complete data comparison to changetime-based comparison. By using changetimes as the key parameter to determine synchronization needs, the system can efficiently identify only the minimal data that requires transfer, reducing command volume while preserving integrity

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If all file system items are scanned and compared during synchronization, then synchronization accuracy is ensured, but processing time and computational resources increase

Engineering Contradiction:

Improvesynchronization accuracyVSAvoidscan and compare time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system extracts only the necessary information (changetimes) from the complete file system item metadata for comparison purposes. Instead of scanning and comparing all file attributes, it extracts and compares only the changetime parameter, significantly reducing processing time while maintaining sufficient accuracy to identify changes

Inventive Principle:

Principle #2Taking out (Extraction)

4Reliability

If users are denied access to the storage system during cutover synchronization, then data migration completeness is ensured, but business interruption increases

Engineering Contradiction:

Improvemigration completenessVSAvoiduser access denial time

Core Design Contradiction:

ReliabilityVSDuration of action of moving object

Solution Approach 1:

The migration process is segmented into an initial synchronization phase (where users can still access the source system) and a cutover phase. By performing initial sync before cutover, the critical data transfer completes while users are still accessible, reducing the impact of the final cutover interruption

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The synchronization log is updated and changetimes are recorded during the initial synchronization phase before cutover. This preliminary action ensures that the cutover synchronization only needs to transfer minimal changed data, reducing the duration of the critical cutover window where users are denied access

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12475083B2Synchronizing file system items having a data and metadata portion between a source storage system and a destination storage system

Publication Date: 2025.11.18 DATADOBI BV

AI summary

A method for synchronizing file system items having a data and metadata portion between a source and a destination storage system; a synchronization log is maintained having synchronization log items associated with the respective file system items; a synchronization log item includes a metadata and data section comprising changetimes of the file system item on the source and destination storage system upon synchronization of the respective metadata and data portion, and includes a selected integrity level selectable from at least a data integrity level and a metadata integrity level.