Policy Workflow Worker Routing for Data Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing pipelines face bottlenecks and inefficiencies due to computationally-intensive tasks stalling less intensive ones, leading to slower processing and scalability issues, along with data inconsistencies and complex resource management.
Innovation Solution
A distributed data processing system utilizing a policy workflow worker and multiple workflow process workers, where messages are dynamically routed based on workflow identifiers and file locations, allowing for iterative processing and the generation of new workers when thresholds are met, to improve efficiency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is processed via a single serial data processing pipeline, then resource management is simplified, but processing speed decreases and bottlenecks occur when computationally-intensive tasks stall less intensive ones
Solution Approach 1:
The patent segments the monolithic data processing pipeline into multiple independent workflow process workers, each capable of handling specific tasks autonomously. This segmentation allows parallel processing of multiple data items simultaneously, eliminating the bottleneck where one task stalls others. Each worker can independently execute its assigned workflow without waiting for other workers to complete their tasks.
Solution Approach 2:
The patent transitions from a single-dimensional serial processing pipeline to a multi-dimensional parallel processing architecture. By introducing multiple workflow process workers operating concurrently, the system adds a spatial dimension to processing, enabling simultaneous execution of multiple workflows rather than sequential processing through a single pipeline.
2Productivity
If multiple workflow process workers are used to improve processing speed, then data processing efficiency increases, but data inconsistencies and resource management complexity increase
Solution Approach 1:
The patent implements a policy workflow worker that acts as a central coordinator and supervisor. This worker receives status updates and processed data from multiple workflow process workers, evaluates their outputs against predefined policies, and ensures data consistency across all workers. The feedback mechanism allows the policy workflow worker to monitor and regulate the activities of multiple workers, maintaining reliability while enabling parallel processing.
Solution Approach 2:
The policy workflow worker serves as an intermediary between multiple workflow process workers and the central system. It mediates the interactions among workers, coordinates their activities, and ensures that data processed by different workers remains consistent. This intermediary layer abstracts the complexity of managing multiple workers while maintaining data integrity and reliability.
3Loss of time
If computationally-intensive tasks are processed in a serial pipeline, then resource management is simpler, but processing time increases and bottlenecks occur
Solution Approach 1:
The patent introduces dynamic workflow management where the policy workflow worker can adaptively assign and redistribute tasks among multiple workflow process workers based on current system state and workload conditions. This dynamic approach allows the system to optimize processing time by dynamically balancing the load across workers, while the automated nature of this dynamic management prevents excessive operational complexity.
Solution Approach 2:
The workflow process workers operate autonomously, with each worker independently executing its assigned workflows without requiring manual intervention or complex coordination. The workers self-manage their execution, and the policy workflow worker automatically handles coordination and consistency checks, reducing the operational burden on users while enabling parallel processing to minimize processing time.
Data Source
AI summary
Some examples include a machine-readable storage medium with instructions executable by a processing resource of a device to process a file according to a pre-defined workflow. The machine-readable storage medium comprises instructions to receive a message to process the file at a workflow queue, to forward the file, by a policy workflow worker, to a first workflow process worker that corresponds to the file, and to perform, at the first workflow process worker, a first workflow process on the message. The machine-readable storage medium further comprises instructions to return, by the first workflow process worker, the processed message to the workflow queue, wherein the processed message includes the file and a first result, to evaluate, by the policy workflow worker, the processed message against the pre-defined workflow and, based on the evaluation, to forward the processed message, by the policy workflow worker, to a second workflow process worker or an output queue.