Dynamic Batch Job Sizing for Repository Data Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The initial indexing process in large data repositories is time-consuming, causing system resources to be unavailable or slow, and existing sequential batch data retrieval methods can be delayed by a single batch of data, leading to inefficiencies and prolonged index updates.
Innovation Solution
A method that dynamically adjusts batch jobs by splitting or merging them based on control parameters to optimize data transmission, monitoring system performance parameters to ensure they meet predefined standards, and using optimization operations to determine the best approach for data distribution during transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential batch data retrieval is used to transmit repository data, then data transmission can be performed systematically, but the process is delayed by a single batch of data that takes a long time to retrieve and transmit
Solution Approach 1:
The patent divides the data transmission process into multiple parallel batch jobs instead of a single sequential process. Each batch job processes a portion of the data independently, allowing concurrent retrieval and transmission operations to proceed simultaneously, thereby eliminating the bottleneck where one slow batch delays the entire process.
Solution Approach 2:
The patent introduces dynamic control parameters that allow the system to automatically adjust batch job characteristics based on real-time performance monitoring. When performance standards are not met, the system can dynamically modify data amounts, batch sizes, or retrieval strategies to optimize transmission speed while maintaining reliability.
2Quantity of substance
If batch jobs are executed to transmit large amounts of data, then data can be transferred efficiently in volume, but system resources become unavailable or slow for a significant amount of time
Solution Approach 1:
Large data volumes are divided into multiple smaller batch jobs that can execute in parallel. This segmentation allows the system to transmit large quantities of data without locking up all system resources, as multiple batches can proceed concurrently using available resources more efficiently.
Solution Approach 2:
The system uses periodic performance monitoring during batch job execution to detect when resources are becoming constrained. When performance standards are not met, the system can pause or adjust batch processing to free up resources for other operations, then resume when resources are available, maintaining continuous progress without complete resource occupation.
3Reliability
If initial indexing is performed on entire repository contents, then complete index coverage is achieved, but the indexing process takes a full day or more depending on system size
Solution Approach 1:
The initial indexing process is divided into multiple parallel batch jobs that process different portions of the repository simultaneously. Each batch job indexes a specific data segment, and all results are aggregated to produce the complete index, achieving full coverage in a fraction of the time required for sequential processing.
Solution Approach 2:
The system performs preliminary actions by pre-dividing the repository into manageable data segments and pre-configuring multiple batch jobs to process these segments in parallel. This preparation allows the indexing process to proceed concurrently through all segments without sequential delays, significantly reducing total indexing time while maintaining completeness.
Data Source
AI summary
A method of organizing a transmission of repository data includes receiving, in a computer system, a request to transmit data from a data repository to a data recipient, the data to be transmitted by executing batch jobs that each identify a data amount to be transmitted to the data recipient. Control parameters are individually assigned to each of the batch jobs. Upon detecting during transmission that a performance parameter of the computer system does not meet a predefined standard, the data amount identified by at least one of the batch jobs is automatically changed in accordance with the control parameter assigned to the at least one of the batch jobs. An optimizer may determine the control parameters by solving a constraint optimization problem. The optimizer may take into account system characteristics such as priorities, transmission constraints or the volume of the data to be transmitted.