Backup Indexing with Transaction Logs for Failover
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional backup systems require restarting the entire backup process if interrupted, leading to loss of time and resources, especially for large operations, due to the lack of failover capabilities and load balancing between modules during data backup.
Innovation Solution
A tiered storage system with a catalog and transaction logs that allows media agent modules to resume backup operations without restarting, enabling failover and load balancing by maintaining metadata logs and indices accessible across multiple modules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional backup systems are used without failover capabilities, then the system structure is simple, but the backup process must be restarted if interrupted, resulting in loss of time and resources
Solution Approach 1:
The backup system is segmented into multiple independent media agent modules that can operate autonomously. Each module can handle backup operations independently, and if one module fails, others can continue or take over the operation. This segmentation enables failover capability without requiring a complete system redesign, thus improving reliability while keeping the overall structure manageable.
Solution Approach 2:
The system maintains copies of transaction logs and catalog information across multiple media agent modules. When a module fails, another module can access these copies and resume the backup operation from the last committed state. This copying mechanism enables seamless failover without restarting the entire backup process, improving continuity while adding only minimal redundancy overhead.
2Productivity
If a single media agent module performs all backup operations, then the system is simple to manage, but the module may become overloaded or fail, causing complete backup interruption
Solution Approach 1:
Backup operations are divided into smaller tasks that can be distributed across multiple media agent modules. Each module handles specific portions of the backup workload, increasing overall throughput and preventing any single module from becoming overloaded. The segmentation also enables parallel processing, further enhancing productivity.
Solution Approach 2:
Multiple media agent modules are combined to work together on a single backup operation. They share the workload and can coordinate through common transaction logs and catalog information. This merging increases capacity and resilience, as the collective system can handle larger workloads and survive individual module failures without complete interruption.
3Stability of the object's composition
If transaction logs and catalog are maintained centrally, then data consistency is ensured, but access performance may degrade for distributed media agent modules
Solution Approach 1:
The centralized catalog and transaction log system is segmented into distributed copies across multiple nodes. Each media agent module can access local copies of the catalog and logs, reducing access latency while maintaining consistency through periodic synchronization. This segmentation improves access speed while preserving data integrity through coordinated updates.
Solution Approach 2:
The system implements feedback mechanisms where media agent modules report their state and transaction information to the centralized catalog, which then distributes updated information back to all modules. This feedback loop ensures all modules have consistent views of the backup state while allowing them to operate independently, balancing consistency requirements with access performance.
Data Source
AI summary
In certain embodiments, a tiered storage system is disclosed that provides for failover protection during data backup operations. The system can provide for an index, or catalog, for identifying and enabling restoration of backup data located on a storage device. The system further maintains a set of transaction logs generated by media agent modules that identify metadata with respect to individual data chunks of a backup file on the storage device. A copy of the catalog and transaction logs can be stored at a location accessible by each of the media agent modules. In this manner, in case of a failure of one media agent module during backup, the transaction logs and existing catalog can be used by a second media agent module to resume the backup operation without requiring a restart of the backup process.