Parallelized Batch Data Processing via Dynamic Worker Scaling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Enterprise computing environments face challenges in efficiently and effectively processing large or variably-sized sets of work items, particularly in maintaining secure and efficient data processing.

Innovation Solution

A computing platform initializes multiple processing workers to perform queries on a work queue database, initiating parallel processing of work items, and dynamically scales the availability of processing workers based on monitored workloads.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple processing workers are initialized to process work items in parallel, then processing speed and productivity are improved, but system complexity and resource management difficulty increase

Engineering Contradiction:

Improveprocessing speedVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments the work queue into multiple partitions and assigns different processing workers to different partitions. Each worker independently processes work items from its assigned partition, enabling parallel processing while maintaining manageable system complexity through clear division of responsibilities.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A database engine acts as an intermediary between processing workers and the work queue. The database engine manages work item distribution, locking, and coordination, shielding workers from the complexity of direct interaction with each other and simplifying resource management.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If processing workers read and lock work items from the work queue database, then data consistency and reliability are improved, but database performance and processing throughput may deteriorate

Engineering Contradiction:

Improvedata consistencyVSAvoidprocessing throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The work queue database is segmented into multiple partitions, and each processing worker is assigned to specific partitions. This segmentation reduces contention for database locks because workers operate on different partitions simultaneously, maintaining data consistency within each partition while improving overall processing throughput.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Processing workers read and lock only the specific work items assigned to them rather than accessing the entire work queue. This partial action approach reduces the scope of database locking, minimizing blocking effects on other workers and maintaining high throughput while ensuring data consistency for the assigned items.

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If the system dynamically scales the availability of processing workers based on monitored workload, then adaptability and efficiency are improved, but system complexity and overhead increase

Engineering Contradiction:

Improveworkload adaptabilityVSAvoidsystem overhead

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system implements a monitoring process that continuously observes the workload in the work queue database and provides feedback to the processing workers. Based on this feedback, workers can dynamically adjust their behavior or the system can scale worker availability, enabling adaptability to changing workload conditions while maintaining manageable overhead through automated control.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

Processing workers are designed to autonomously monitor their own workload and self-regulate by picking up new work items from the database when capacity is available. This self-service approach enables dynamic adaptation to workload changes without requiring complex centralized control mechanisms, reducing system overhead while maintaining efficiency.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12204518B2Optimizing distributed and parallelized batch data processing

Publication Date: 2025.01.21 BANK OF AMERICA CORP

AI summary

Aspects of the disclosure relate to providing and maintaining efficient and effective processing of sets of work items in enterprise computing environments by optimizing distributed and parallelized batch data processing. A computing platform may initialize a monitoring process configured to monitor a pending workload in a work queue database. Subsequently, the computing platform may cause the monitoring process to query the work queue database and create one or more historical records indicative of a workload processing status associated with one or more processing workers. Then, the computing platform may identify one or more new parameter values for one or more processing parameters associated with the one or more processing workers based on the one or more historical records. Thereafter, the computing platform may configure the one or more processing workers based on the one or more new parameter values identified for the one or more processing parameters.