Database Transaction Log Capture for Aggressive RPO Enforcement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage management systems struggle to reliably meet aggressive Recovery Point Objectives (RPO) for production databases due to bottlenecks in storage manager components, leading to potential data loss during backup operations.
Innovation Solution
A data storage management system that autonomously captures database transaction logs and stores them securely away from the DBMS, using a proprietary data pipeline to transmit temporary backup files to secondary storage without involving the storage manager, and generates secondary copies less frequently to avoid processing bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the storage manager computer manages database backups, then centralized control and coordination are achieved, but processing bottlenecks occur that prevent reliable meeting of aggressive RPO
Solution Approach 1:
The backup system is segmented into multiple independent components: database data agents that capture transaction logs, a proprietary data pipeline for transmission, and media agents for storage management. This segmentation eliminates the single-point bottleneck of the traditional storage manager while maintaining coordinated backup operations across distributed components.
Solution Approach 2:
The backup capture function is extracted from the storage manager computer and placed directly at the database host through dedicated data agents. This extraction allows transaction log capture to occur independently and concurrently with storage management operations, removing the processing bottleneck that prevented aggressive RPO achievement.
2Reliability
If secondary copies are generated frequently to meet aggressive RPO, then data loss protection is improved, but system overload occurs in storage components
Solution Approach 1:
A proprietary data pipeline acts as an intermediary between database data agents and storage systems, buffering and managing the flow of transaction log data. This intermediary absorbs peak loads during aggressive RPO intervals while maintaining continuous data protection, preventing system overload even when generating secondary copies frequently.
Solution Approach 2:
The system creates lightweight copies of transaction logs in proprietary backup format with compression and encryption, rather than copying entire databases. These compact copies enable frequent secondary copy generation for aggressive RPO without proportionally increasing storage component load, as the actual data volume transferred and stored is significantly reduced.
3Reliability
If transaction logs are captured and stored away from DBMS, then RPO enforcement is improved, but data transmission security requirements increase
Solution Approach 1:
Transaction logs are transformed into proprietary backup format with built-in compression and encryption parameters applied during the capture process. This parameter change occurs at the source, enabling secure transmission without requiring separate complex security infrastructure, as the data is already protected when leaving the database host.
Data Source
AI summary
A disclosed framework enables virtually any aggressive Recovery Point Objective (RPO) to be enforced for a production database, without limitation on type of database management system (DBMS) or size of the production database. The framework overcomes technological barriers of the prior art, such as bottlenecks presented by a storage manager computer that traditionally managed database backups. To avoid the bottlenecks, an illustrative data storage management system does not engage the storage manager computer in storage operations that could impact the database RPO. Certain components are added and enhanced to autonomously capture database transaction logs, which are stored securely and persistently away from the DBMS. In a separate and less frequent operation, and with involvement of the storage manager computer, the illustrative system generates secondary copies in their final form. Meanwhile, temporary backup files can be used in restore operations without first being converted into secondary copies.