Dynamic Batch Job Segmentation for Data Repository Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data repository systems face inefficiencies in initial indexing due to sequential batch data retrieval, leading to prolonged system resource unavailability and delayed index updates, especially when handling large volumes of data, with inadequate handling of failed batch jobs and lack of dynamic adjustment to performance parameters.
Innovation Solution
A method that dynamically adjusts batch jobs by splitting or merging them based on performance parameters, using predictive models to optimize data transmission, and implementing recovery procedures for failed jobs, allowing for dynamic changes in the number of batch jobs and resource allocation to maintain system efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential batch data retrieval is used for initial indexing, then data transmission can be performed systematically, but system resources remain unavailable for extended periods and indexing time increases significantly
Solution Approach 1:
The patent divides the initial indexing process into multiple parallel batch jobs that can execute simultaneously. Instead of processing data sequentially in one large batch, the system segments the data into smaller portions and creates multiple batch jobs that run in parallel, significantly reducing the total indexing time and making system resources available more quickly.
Solution Approach 2:
The patent implements dynamic adjustment of batch job parameters during execution. The system monitors performance parameters and automatically adjusts the number of batch jobs, data amounts per batch, or other control parameters to optimize indexing speed while managing system resource consumption, allowing the indexing process to adapt to changing system conditions.
2Reliability
If sequential batch data retrieval is used, then data transmission can be controlled, but a single slow batch job can hold up the entire indexing process
Solution Approach 1:
The patent segments the data transmission into multiple independent batch jobs that execute in parallel. This segmentation ensures that a slow or failing batch job does not block the entire indexing process, as other batch jobs can continue executing independently, maintaining overall productivity while preserving control through job management.
Solution Approach 2:
The system implements feedback mechanisms that monitor the performance of individual batch jobs and adjust the overall indexing process accordingly. When a batch job performs poorly or fails, the system receives feedback and can adjust parameters such as the number of concurrent jobs or data portion sizes to maintain optimal throughput.
3Productivity
If batch jobs process large data amounts, then transmission efficiency per job increases, but system resources are occupied longer and other indexes must wait
Solution Approach 1:
The patent segments large data amounts into smaller batches processed in parallel. Instead of processing one large data portion that occupies resources for a long time, the system divides the data into multiple smaller portions that can be processed simultaneously by multiple batch jobs, reducing the duration each job holds resources while maintaining overall efficiency.
Solution Approach 2:
The system dynamically adjusts the data amount per batch job based on system conditions and performance monitoring. It can increase or decrease batch sizes in real-time to optimize the balance between transmission efficiency and resource occupation duration, allowing flexible adaptation to changing system load and performance characteristics.
4Quantity of substance
If the indexing process takes a long time, then comprehensive data can be processed, but system efficiency reduces and updates are delayed
Solution Approach 1:
The patent segments the comprehensive data processing into multiple parallel batch jobs that execute simultaneously. This allows the system to process large data volumes quickly by distributing the workload across multiple jobs rather than sequentially, maintaining high system efficiency while completing comprehensive indexing.
Solution Approach 2:
The system uses dynamic parameter adjustment to optimize processing speed without sacrificing completeness. By monitoring performance and adapting batch parameters in real-time, the system ensures that all data is processed thoroughly while minimizing total processing time, balancing comprehensive data processing with high system efficiency.
Data Source
AI summary
A method of organizing a transmission of repository data includes receiving, in a computer system, a request to transmit data from a data repository to a data recipient. In response to the request, batch jobs for transmitting the data are performed. Each of the batch jobs identifies a data amount that is to be transmitted to the data recipient. Upon detecting during transmission that a performance parameter of the computer system does not meet a predefined standard, the data amount that is identified by at least one of the batch jobs is automatically changed. Changing the data amount may include splitting the batch job into at least two batch jobs, or merging the batch job with another batch job. A predictive model may be created for determining a control parameter that can be assigned to the batch jobs. The control parameter may help the system assume an optimal organization state.