Locally Distributed Change Detection for Backup Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional backup systems are time-consuming, labor-intensive, and inadequate for modern computer networks that operate 24/7, as they often require significant downtime for data backup and fail to efficiently manage frequent changes in data.
Innovation Solution
The system employs locally distributed change detection, asynchronous replication correctness validation, and WebDAV-based data transfer to identify and transmit only changed file segments, minimizing network bandwidth usage and enabling rapid backup and recovery with snapshots for version history.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional backup systems copy files from source to destination using central server cataloging, then data backup functionality is provided, but the process is time-consuming and labor-intensive
Solution Approach 1:
The system segments the backup process into distributed client nodes that independently monitor and detect changes locally, rather than relying on a centralized server to catalog all files. Each client maintains its own manifest file tracking changes, enabling parallel change detection across multiple clients simultaneously, thus improving overall backup productivity
Solution Approach 2:
The backup system performs self-service through automated change detection using manifest files. Clients automatically compare their local data against the manifest to identify changes, eliminating the need for manual intervention. The system autonomously determines what needs to be backed up and initiates transfers without human involvement, reducing labor intensity and time
2Productivity
If backup systems operate by turning over network resources during low-activity periods, then data backup can be performed, but network availability is reduced and the process becomes cumbersome
Solution Approach 1:
The system performs preliminary change detection locally at each client before network transfer is needed. By maintaining manifest files and comparing them against current data states, clients identify all changes in advance during normal operations. This preliminary action allows the actual backup transfer to occur more efficiently when network resources are available, without requiring prolonged downtime
Solution Approach 2:
The backup system dynamically adapts to network conditions by using compression and encryption during transfer, and by allowing flexible scheduling of backup operations. The system can perform change detection continuously in the background rather than requiring dedicated backup windows, making network availability less critical while maintaining backup reliability
3Quantity of substance
If traditional backup systems backup files once a week or once a day, then network resources are conserved, but the backup value is reduced for frequently changing data
Solution Approach 1:
The system enables continuous change detection by having clients continuously compare their data against manifest files. This continuous monitoring ensures that even frequently changing data is captured in real-time, making the backup currency current and valuable. The useful action of change detection continues uninterrupted during normal data operations, not just during scheduled backup windows
Solution Approach 2:
The manifest files provide feedback about what has changed since the last backup. Clients use this feedback information to determine exactly what needs to be transferred, enabling incremental backups that capture only recent changes. This feedback mechanism ensures backup currency reflects actual data changes rather than following a fixed schedule
4Reliability
If the system transfers complete files for backup, then data protection is ensured, but network bandwidth consumption increases
Solution Approach 1:
The system extracts only the changed portions of files for backup rather than transferring complete files. By using manifest files to identify changes and then transferring only those specific changes, the system dramatically reduces network bandwidth consumption while maintaining data protection completeness for the modified portions
Solution Approach 2:
The backup system applies local quality by processing and comparing data locally at each client before network transfer. Clients perform change detection and determine transfer requirements locally, then only transmit the necessary changed data to the backup destination. This local processing quality ensures efficient use of network bandwidth by avoiding transmission of unchanged data
Data Source
AI summary
Systems and methods for replicating a client data set on a computer includes replicating the client data set using an application software on the computer; generating a manifest of a target data set during replication containing state of target data reflecting post-update state; and determining differences between the local data and manifest to determine required backup operations.