Batch Process Partitioning for Parallel Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing enterprise database systems face challenges in parallelizing business calculations due to complexity and errors, especially in a SaaS architecture where multiple tenants share hardware, requiring efficient processing power distribution across clients of varying sizes and requirements.
Innovation Solution
A system is developed that includes a processor configured to verify batch jobs for compatibility and execute jobs by determining appropriate nodes based on user requests, customer size, and resource availability, allowing for parallel processing and error checking of process definitions to ensure successful execution in a distributed architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallel computing is used to process large data sets, then processing speed is improved, but software design complexity and error difficulty increase
Solution Approach 1:
The system automatically partitions batch processes into independent tasks and assigns them to available processors without requiring manual intervention. The process manager dynamically creates partitions, distributes them to processors, and monitors execution, enabling the system to self-manage parallel computing complexity while maintaining high processing speed.
2Use of energy by moving object
If multiple tenants share hardware resources, then resource utilization is improved, but processing power distribution complexity increases
Solution Approach 1:
The system segments batch processes into independent partitions that can be independently assigned to different processors. This segmentation allows flexible distribution of work across multiple tenants sharing hardware resources, enabling efficient resource utilization while simplifying the distribution management through automated partition-based allocation.
3Reliability
If batch processes are executed sequentially, then error checking is simplified, but processing time increases
Solution Approach 1:
The system divides batch processes into independent partitions that can be executed in parallel. Each partition maintains its own error checking mechanisms, allowing simultaneous execution across multiple processors while keeping error detection manageable through modular design. This segmentation reduces processing time without compromising error checking effectiveness.
Data Source
AI summary
A system for processing a batch job comprises a processor and a memory. The processor is configured to receive a job name for a job submitted to execute, to receive one or more job parameters, and to determine one or more nodes to run the job. The processor is configured to determine one or steps, where for each step: a step is executed on a node using a state of data associated with a start state of the step; and upon completion of executing the step, a result is stored to a durable storage. The durable storage stores the state of data associated with the start state of the step and the completion state of the step and are accessible by other execution processes as associated with either the start state of the step or the completion state of the step. The memory of the system is coupled to the processor and configured to provide processor with instructions.