Adaptive Job Queues for Parallel Data Backup Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data in modern computer systems has led to longer backup times, as the processing speed has not kept pace with the increase in data and I/O throughput, causing decryption and checksum calculation processes to become bottlenecks.
Innovation Solution
The implementation of a system that uses a plurality of adaptively managed jobs to perform decryption and checksum calculation processing in parallel, ensuring that job queues remain non-empty and avoiding job sleep, thereby optimizing the backup process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the amount of data stored by computer systems increases, then data storage capacity is improved, but backup time increases
Solution Approach 1:
The backup process is divided into multiple independent jobs that can execute in parallel. Each job handles a portion of the data pages, allowing simultaneous decryption and checksum calculation across multiple threads, thereby reducing total backup time while handling large data quantities
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing. By adding the dimension of concurrency, multiple decryption and validation operations occur simultaneously rather than sequentially, reducing backup time without increasing data storage capacity requirements
2Productivity
If processing speed increases to keep up with data growth, then backup performance is improved, but computational resources are consumed
Solution Approach 1:
The system dynamically manages the number of jobs based on actual workload conditions. Jobs are created and destroyed as needed, and the system adapts to varying data sizes and processing requirements, optimizing resource utilization while maintaining high backup performance
Solution Approach 2:
The patent changes the parameter of job management from static to adaptive. By monitoring system conditions and adjusting the number of active jobs dynamically, the system optimizes the balance between processing speed and resource consumption, preventing resource waste while maintaining high productivity
3Speed
If multiple jobs process data in parallel, then backup speed is improved, but system complexity increases
Solution Approach 1:
The job management system automatically handles job creation, task distribution, and resource allocation without manual intervention. The system self-regulates based on available resources and data size, reducing the operational complexity burden on users while maintaining high backup speed through parallel processing
Solution Approach 2:
The system incorporates feedback mechanisms that monitor job execution and resource utilization. Based on this feedback, the system dynamically adjusts the number of active jobs and their task assignments, optimizing parallel processing efficiency while managing system complexity through automated control loops
Data Source
AI summary
Embodiments of the present disclosure include techniques for backing up data. In one embodiment, a plurality of read requests are issued. In response to the read requests, a plurality of data pages are retrieved. The plurality of data pages are stored in a plurality of buffers. During said storing, for each data page, an indication that storage of a particular data page of the plurality of data pages has been completed is generated. In response to an indication that storage of a particular data page has been completed, the data page is processed with one of a plurality of jobs, where a plurality of data pages are processed by the plurality of jobs in parallel.