Dynamic Data Pipeline Orchestration for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud platforms face increased system latency and degraded user experience due to sequential execution of tasks in data processing, which restricts flexibility and user input, especially when task dependencies are not dynamically managed.
Innovation Solution
Implementing a process orchestration system that generates and dynamically updates execution plans, allowing for concurrent execution of independent tasks and user input during task execution, thereby managing task dependencies and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If tasks are performed sequentially to ensure process dependencies are enforced, then task dependency management is reliable, but system latency increases and user experience degrades
Solution Approach 1:
The patent segments the execution plan into multiple subsets of tasks that can be executed concurrently. The orchestration server identifies independent tasks that do not have dependencies on each other and groups them into separate subsets, allowing parallel execution while maintaining dependency constraints within each subset.
Solution Approach 2:
The execution plan is dynamically updated based on user input and task execution results. The system allows users to modify the execution plan during task execution, adding, removing, or modifying tasks and subsets, thereby adapting the execution strategy in real-time to optimize performance while maintaining reliability.
2Device complexity
If a fixed execution plan is used to simplify task management, then system complexity is reduced, but system flexibility and user ability to update the system are reduced
Solution Approach 1:
The execution plan transitions from a static to a dynamic structure that can be modified during task execution. Users can provide input to add, remove, or modify tasks and subsets, and the orchestration server updates the execution plan accordingly, maintaining both simplicity and flexibility.
Solution Approach 2:
The system incorporates feedback mechanisms where user input during task execution is processed to update the execution plan. The orchestration server monitors task progress and allows users to provide feedback that triggers dynamic updates to the execution plan, enabling adaptive task management.
3Ease of operation
If users are not provided with information about process execution to simplify system operation, then ease of operation is improved, but user understanding and ability to update the system are reduced
Solution Approach 1:
The system provides feedback to users about task execution progress and allows users to interact with the execution plan during processing. Users can view the current state of task execution and provide input to modify the plan, maintaining simplicity while improving information transparency.
Data Source
AI summary
Methods, systems, and devices supporting dynamic process orchestration are described. An orchestration server may receive a request defining a data modification process from a user device. The orchestration server may generate an execution file based on the request, and the execution file may include a set of tasks for performing the data modification process and an order for performing the set of tasks. The orchestration server may execute, for the execution file, a first set of tasks according to the order for performing the set of tasks and, in some cases, may update the execution file based on executing the first subset of tasks. For example, updating the execution file may involve modifying a second subset of tasks of the set of tasks. The orchestration server may execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks.