Data Recovery System with Dynamic Throughput Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data recovery systems are time-consuming due to the need to move large volumes of data from backup systems to databases, often requiring computationally intensive processing.
Innovation Solution
The system optimizes data recovery by adjusting the throughput of processing functions based on the available capacity of the main memory, preventing intermediate memories from overflowing or becoming too empty, and using parallel processing to manage data pages efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is moved from backup system to database at high speed, then recovery time is reduced, but intermediate memory may overflow causing delays
Solution Approach 1:
The system dynamically adjusts the throughput of processing functions based on real-time monitoring of main memory available capacity. When memory capacity is sufficient, throughput is increased to accelerate recovery; when memory capacity approaches thresholds, throughput is reduced to prevent overflow, creating a dynamic balance between speed and stability
Solution Approach 2:
The system implements continuous feedback monitoring of main memory capacity and uses this information to control the throughput of data processing functions. This closed-loop control ensures that data flow is automatically regulated to match available memory resources, preventing overflow while maximizing recovery speed
2Reliability
If computationally intensive processing is performed on data pages, then data integrity is ensured, but processing speed decreases
Solution Approach 1:
The system divides data recovery into discrete data page units and processes them through multiple parallel processing functions. Each processing function handles specific operations on data pages, allowing computationally intensive integrity checks to be performed concurrently on multiple pages simultaneously, maintaining both integrity and speed
Solution Approach 2:
The system maintains continuous processing of data pages through parallel functions that operate simultaneously. While one function performs computationally intensive operations on a data page, other functions continue processing different pages, ensuring that useful action continues without interruption and overall processing speed is maximized
3Productivity
If parallel processing functions are increased to speed up recovery, then productivity increases, but memory consumption increases risking overflow
Solution Approach 1:
The system dynamically controls the number and throughput of parallel processing functions based on available memory capacity. When memory is abundant, more parallel functions are activated to maximize recovery speed; when memory capacity is limited, the number of active functions is reduced to prevent overflow, creating optimal utilization of available resources
Data Source
AI summary
Embodiments of the present disclosure include techniques for recovering data. In one embodiment, data is copied to a buffer. A plurality of processing functions receive the data in the buffer as data pages and perform processing operations. The processed data pages are then stored in persistent memory. The main memory is monitored so that the main memory of the database is maintained in a state such that a consistent flow of data may be written to persistent memory during the recovery process.