Management Database Failover Using Granular Transaction Log Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage management systems face challenges in protecting management databases with aggressive recovery point objectives (RPO) due to the increasing volume and distribution of data, necessitating a streamlined approach for reliable failover scenarios.
Innovation Solution
A data storage management system with enhanced storage managers that perform a 'dump-sweep-and-restore' process, where transaction logs are frequently dumped and transformed into backup data chunks, synchronized with failover storage managers to meet a 1-minute RPO, using a tracking database to manage and restore these chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional backup methods are used for management databases, then data protection is provided, but the recovery time and data loss exceed aggressive RPO requirements
Solution Approach 1:
The management database is segmented into multiple data chunks that are independently backed up and stored. This segmentation allows parallel processing of backup operations and enables selective restoration of specific chunks, significantly reducing overall recovery time and data loss compared to traditional monolithic backup approaches.
Solution Approach 2:
The system performs preliminary actions by continuously maintaining updated copies of management database chunks in the secondary storage system before failures occur. Transaction logs are continuously captured and applied to backup chunks, ensuring that recovery-ready data is already prepared and available, eliminating the need for time-consuming backup operations during failure events.
2Loss of time
If backup frequency is increased to meet aggressive RPO, then data loss is reduced, but system performance and productivity are impacted
Solution Approach 1:
By dividing the management database into multiple chunks that can be backed up independently and in parallel, the system achieves frequent effective backups without proportionally increasing total backup time. Each chunk can be processed simultaneously, maintaining system performance while reducing data loss windows.
Solution Approach 2:
The system creates and maintains multiple copies of management database chunks in the secondary storage system. These copies are updated continuously through transaction log application, providing frequent recovery points without requiring repeated full backup operations that would impact primary system performance.
3Adaptability or versatility
If the management database is distributed across multiple storage managers, then system capacity and scalability are improved, but data protection complexity increases
Solution Approach 1:
The secondary storage system is designed with universal functionality to handle multiple types of data (customer data and management database chunks) using the same backup and recovery mechanisms. This multi-functionality simplifies the overall system architecture by providing a unified data protection approach across distributed storage managers, reducing complexity despite increased scalability.
Solution Approach 2:
The system introduces an intermediary layer (the secondary storage system with chunk management capabilities) that mediates between distributed storage managers and the backup/recovery processes. This intermediary handles the complexity of managing distributed data chunks, transaction logs, and recovery operations, allowing storage managers to focus on their primary functions while maintaining simplified data protection.
Data Source
AI summary
An illustrative data storage management system comprises a management database that stores administrative preferences and system configurations, as well as results and/or statistics of completed secondary storage operations, i.e., information needed by the system to protect customers' data and to track and recover the protected data, including secondary copies such as backup copies, archive copies, etc. The disclosed data storage management system is configured to protect its own system data subject to a very aggressive (short) Recovery Point Objective (RPO), by using an innovative infrastructure that enables the system's storage manager to fail over to any number of other failover destination storage managers, each one comprising a destination management database. An illustrative database granularly tracks whether each and every transaction log file has been successfully applied to each and every destination management database to synchronize with the source management database.