Runtime Task Inheritance for Batch Processing Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current batch processing frameworks lack support for parallelism, require a client for job execution, and do not allow for the creation of dependency trees, limiting their ability to efficiently process tasks and reduce execution time.
Innovation Solution
The introduction of runtime tasks, which are dynamically created to maximize parallelism by splitting tasks based on available processing power across multiple batch server instances, allowing for concurrent execution and propagation of dependencies and batch group configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch jobs are executed serially with a single task framework, then the system structure is simple, but the execution time is long and throughput is low
Solution Approach 1:
The batch job is segmented into multiple sub-tasks that can be executed in parallel. The framework divides the overall batch job into independent sub-tasks that can be distributed across multiple servers, enabling concurrent execution and improving throughput while maintaining manageable complexity through modular task organization
Solution Approach 2:
The system transitions from single-dimension serial execution to multi-dimensional parallel execution by introducing the dimension of multiple servers and tasks. Tasks can be distributed across different servers and executed simultaneously, effectively utilizing hardware resources and reducing execution time
2Extent of automation
If a client is required to manually launch the batch engine, then the control flow is simple, but the automation level is low and execution reliability is poor
Solution Approach 1:
The batch framework is designed to automatically execute tasks without requiring manual client intervention. The system self-manages task scheduling, server selection, and execution monitoring, enabling automated batch job processing while reducing dependency on external client connections
Solution Approach 2:
An automatic task scheduling mechanism serves as an intermediary between job submission and execution. This mediator automatically manages the workflow, allocating tasks to appropriate servers and coordinating execution without requiring manual client engagement, thereby improving automation while maintaining systematic control
3Loss of time
If batch jobs are not broken into sub-jobs across multiple servers, then the framework is simple, but the execution time is long and hardware utilization is poor
Solution Approach 1:
Batch jobs are divided into smaller sub-tasks that can be independently executed on multiple servers simultaneously. This segmentation enables parallel processing, significantly reducing overall execution time while distributing the coordination complexity across manageable task units
Solution Approach 2:
Multiple server resources are merged to work together on a single batch job. The framework combines the computing power of multiple servers by distributing sub-tasks across them, achieving better hardware utilization and faster execution through coordinated parallel processing
Data Source
AI summary
A batch job processing architecture that dynamically creates runtime tasks for batch job execution and to optimize parallelism. The task creation can be based on the amount of processing power available locally or across batch servers. The work can be allocated across multiple threads in multiple batch server instances as there are available. A master task splits the items to be processed into smaller parts and creates a runtime task for each. The batch server picks up and executes as many runtime tasks as the server is configured to handle. The runtime tasks can be run in parallel to maximize hardware utilization. Scalability is provided by splitting runtime task execution across available batch server instances, and also across machines. During runtime task creation, all dependency and batch group information is propagated from the master task to all runtime tasks. Dependencies and batch group configuration are honored by the batch engine.