Configurable Data Pipeline Orchestration via Dynamic Stage Sequencing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing technologies struggle to efficiently orchestrate data pipelines with multiple stages, often requiring sequential execution that may not be optimal for processing workflows and utilizing microservices effectively.
Innovation Solution
A method that involves receiving data pipeline stages from a trigger, performing operational functions, publishing completion messages, querying configuration modules for subsequent functions, and determining the next operational function to be performed, allowing for non-sequential execution based on pre-configured settings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If sequential execution of data pipeline stages is used, then system complexity is reduced and ease of operation is improved, but productivity and workflow optimization are worsened
Solution Approach 1:
The patent applies dynamics by making the data pipeline execution order configurable and flexible rather than fixed. The system allows runtime determination of stage execution sequences through configuration files and dynamic graph generation, enabling the pipeline to adapt its execution flow based on specific workflow requirements while maintaining operational simplicity through centralized configuration management.
2Productivity
If non-sequential execution of data pipeline stages is implemented, then productivity and workflow optimization are improved, but device complexity and orchestration difficulty are worsened
Solution Approach 1:
The patent introduces an intermediary configuration module that manages the complexity of non-sequential pipeline orchestration. This module stores execution order configurations, generates dynamic execution graphs, and coordinates stage transitions, thereby enabling flexible non-sequential execution without directly increasing the complexity of individual pipeline stages or their interconnections.
Solution Approach 2:
The system performs preliminary actions by pre-configuring execution orders and generating execution graphs before pipeline runtime. Configuration files define stage dependencies and execution sequences in advance, allowing the system to optimize workflow productivity without managing complex runtime orchestration logic during actual data processing.
3Adaptability or versatility
If microservices architecture is utilized, then adaptability and versatility are improved, but ease of operation and orchestration are worsened
Solution Approach 1:
The patent applies universality by creating a unified configuration management system that handles diverse microservice orchestration requirements through a single interface. The configuration module can manage different pipeline stages, execution modes, and service compositions using consistent configuration files and graphs, thereby providing adaptability for various microservice architectures while maintaining ease of operation through standardized management procedures.
Data Source
AI summary
Orchestrating data pipelines in a pre-orchestrated manner. In some instances, workflows and microservices are performed in stages. The order in which these stages are performed are pre-dominantly non-sequential in order to ensure that the most relevant stages are performed in a manner that allows the workflows to be processed and microservices to be utilized in the most efficient manner possible. In some instances, when the processes in the first stage is complete, a broadcast message is published to a topic that indicates that the first stage is completed and the second stage can commence. In order to determine which processes can be performed in the second stage, a configuration table is utilized. This general process is repeated until each stage in the data pipeline is complete.