Hybrid Flow Conversion to Data Flow Graph
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing analytic flow design tools and execution engines face challenges in optimizing hybrid flows, which are complex and involve mixed control and data flows, making it difficult to improve efficiency and operation without manual intervention.
Innovation Solution
The process of converting a hybrid flow into a single data flow graph, allowing for automated optimization by combining task and job flow graphs, separating operators, and adding connector operators, while preserving control flow semantics, enables global pushdown optimization across multiple execution engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hybrid flows are implemented as a collection of scripts and code with mixed control flows, then the system can execute complex analytic processes on multiple execution engines, but the system becomes difficult to optimize and maintain without manual intervention
Solution Approach 1:
The patent introduces a hybrid flow translator as an intermediary component that converts mixed control flow definitions into equivalent data flow graph representations. This translator mediates between the complex mixed control flow execution model and the optimization capabilities of data flow analysis tools, enabling automated optimization without requiring manual intervention in the complex script-based control logic.
2Productivity
If manual intervention is used to optimize hybrid flows, then some optimization can be achieved, but the process is time-consuming and not scalable
Solution Approach 1:
The patent enables the hybrid flow system to self-optimize by automatically translating mixed control flow definitions into data flow graph representations that can be processed by automated optimization algorithms. The hybrid flow translator and analyzer work together to perform optimization without human intervention, making the system self-sufficient in terms of optimization tasks.
3Productivity
If global optimization techniques are applied across multiple execution engines, then overall performance improves, but the complexity of managing control flow semantics increases
Solution Approach 1:
The patent creates a copied representation of the mixed control flow in the form of a data flow graph. The hybrid flow translator generates an equivalent data flow graph that mirrors the control flow semantics, allowing optimization techniques to operate on this copied representation without directly manipulating the original complex control flow structure. This copying approach enables global optimization while preserving control flow semantics through the translated representation.
Data Source
AI summary
Converting a hybrid flow can include combining each of a plurality of task nodes with a plurality of corresponding operators of the hybrid flow and converting the combined plurality of task nodes and the plurality of corresponding operators of the hybrid flow to a data flow graph using a code template.