Parallel Processing Framework for Batch Work Packages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional batch processing methods require significant time and resources due to sequential processing of work packages, leading to inefficiencies in handling large amounts of data.
Innovation Solution
Implementing a Mass Data Run Object (MDRO) in a service-oriented architecture that enables parallel processing of independent work packages by instantiating a framework for parallel processing (FPP), allowing for simultaneous execution and clustering of work packages outside the requesting application.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential batch processing is used, then the system structure is simple and easy to implement, but the processing time is long and productivity is low
Solution Approach 1:
The batch process is divided into multiple independent work packages that can be processed in parallel. Each work package represents a discrete business object action and can be executed independently by the parallel processing framework, thereby increasing processing throughput without requiring complete system redesign.
Solution Approach 2:
A Mass Data Run Object (MDRO) is introduced as an intermediary between the application and the parallel processing framework. The MDRO manages the compilation of work packages, instantiation of the parallel processing framework, and coordination of parallel execution, thereby hiding the complexity of parallel processing from the application layer.
2Loss of time
If parallel processing is implemented, then processing time is reduced and productivity increases, but the system complexity and resource requirements increase
Solution Approach 1:
Work packages are compiled and prepared in advance before parallel execution begins. The MDRO compiles the batch process into discrete work packages and confirms their independence beforehand, allowing the parallel processing framework to execute them simultaneously without runtime compilation overhead.
Solution Approach 2:
The parallel processing framework automatically manages the execution of work packages without requiring continuous application intervention. Each work package is executed independently in its own session, and the framework handles session management, error handling, and result aggregation automatically.
3Productivity
If work packages are processed within the application, then the application has full control over processing, but the application resources are consumed and performance degrades
Solution Approach 1:
The batch processing workload is extracted from the application and transferred to an external parallel processing framework. The application only needs to initiate the batch process through the MDRO interface, while the actual processing is performed by the framework using separate computing resources, thereby preserving application performance.
Solution Approach 2:
Processing is moved from the application dimension to a framework dimension. The MDRO acts as a bridge, allowing the application to request batch processing without the processing occurring within the application's execution context. This dimensional separation allows independent resource allocation and avoids application resource exhaustion.
Data Source
AI summary
The present disclosure relates to methods, systems, and software for performing parallel processing on a set of business object actions forming a batch process. The batch process may be performed by compiling a group of independent work packages, each work package representing a discrete business object action. The work packages are clustered and an individual session for each work package is created. The work packages are then executed in parallel by a framework for parallel processing, external of an application requesting the batch process. Each session is closed after processing of the work package is completed, and the cluster is closed after processing of the work packages is completed.