Database Change Log Tracking with Periodic Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database backup systems face inefficiencies due to exponential storage space requirements as databases grow, with differential-based systems becoming costly and computationally inefficient over time, especially when restoring data from longer periods and managing change logs.
Innovation Solution
A system that determines the type of snapshot (full or log-based) based on log size and operation statistics, storing snapshots efficiently as metadata or complete data, and managing change logs by calculating cumulative sizes and removing redundant entries, thereby optimizing storage and computation costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If differential-based backup systems store only differences relative to a reference point, then storage requirements are reduced, but restoration becomes more costly and inefficient over time due to applying differences over longer periods
Solution Approach 1:
The patent segments the change log into fixed-size chunks and creates periodic snapshots at predetermined intervals. This segmentation allows the system to balance between storing detailed differences (for recent changes) and maintaining reference points (for older changes), thereby reducing both storage requirements and restoration time by avoiding the need to apply all differences from a distant reference point.
Solution Approach 2:
The system implements periodic snapshots at predetermined time intervals or after a predetermined number of change log chunks. This periodic action creates intermediate reference points that reduce the accumulation of differences, allowing faster restoration by applying differences only since the most recent snapshot rather than from an older reference point, thus reducing restoration time while maintaining storage efficiency.
2Reliability
If full backups are taken in response to small changes, then data integrity is maintained, but storage space is used inefficiently compared to the actual change size
Solution Approach 1:
Instead of taking full backups for every change, the system applies partial actions by storing only the necessary differences in the change log. Full snapshots are taken periodically or when the change log reaches a size threshold, providing a reference point. This partial action approach maintains data integrity by capturing all changes since the last snapshot while using significantly less storage space than full backups would require.
Solution Approach 2:
The system changes the parameter of backup granularity by switching between storing full snapshots and storing only differences (change logs). By monitoring the size of the change log and the time elapsed since the last snapshot, the system dynamically adjusts whether to take a full snapshot or continue storing differences, optimizing storage space while maintaining data integrity.
3Loss of information
If change logs are stored at a storage service, then data tracking is comprehensive, but storage costs increase over time as change logs accumulate
Solution Approach 1:
The system performs preliminary action by creating periodic snapshots before the change log becomes excessively large. These snapshots serve as new reference points, allowing the system to truncate or archive old change logs while maintaining complete data tracking capability. This preliminary action prevents the change log from growing indefinitely, thereby controlling storage costs while preserving the ability to restore any point in time.
Solution Approach 2:
The system discards old change log data that has been superseded by more recent snapshots, while maintaining the ability to recover any lost information through the snapshot chain. By keeping periodic full snapshots and only storing differences since the last snapshot, the system can discard accumulated change logs older than the most recent snapshot, significantly reducing storage costs while maintaining complete recoverability through the snapshot sequence.
Data Source
AI summary
Various systems and methods of tracking a size of a database change log are described herein. A system is disclosed herein, according to some embodiments. An indication of a plurality of change events that have occurred at the database is received. A new log segment is generated for the change log for the database based on the received indication including the plurality of change events. Metadata for an end log segment from the change log is retrieved to identify a cumulative size for the change log. A new cumulative size for the change log is determined based on a size of the new log segment and the identified cumulative size. The new log segment is stored to the change log as a new end log segment with metadata indicating the new cumulative size for the change log.