Hybrid Flow Data Structure for Global Graph Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing analytic flow design tools and execution engines face challenges in efficiently processing hybrid flows across multiple execution engines, as they lack the ability to globally optimize the data flow graph while preserving control flow semantics, leading to suboptimal execution and resource allocation.
Innovation Solution
Converting hybrid flows into data flow graphs that combine task and job flow structures, allowing for global processing and optimization, including the addition of new operators and metadata management to maintain control flow semantics, enabling efficient distribution and execution across various execution engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hybrid flows are processed as separate scripts and code fragments with independent control flow, then each fragment can be executed on different execution engines, but global optimization of the data flow graph cannot be achieved
Solution Approach 1:
The patent merges the control flow graph and data flow graph into a unified representation where control flow semantics are embedded within the data flow graph structure. This integration allows global optimization algorithms to operate on the complete hybrid flow while preserving the ability to execute fragments across multiple execution engines, thus resolving the contradiction between adaptability and optimization capability.
2Ease of manufacture
If existing analytic flow design tools process hybrid flows with separate control flow orchestration, then implementation flexibility is maintained, but execution efficiency and resource allocation are suboptimal
Solution Approach 1:
The patent introduces a transformation process that converts the hybrid flow into an optimized data flow graph representation while maintaining the original control flow semantics through metadata preservation. This intermediary transformation enables global optimization without losing the implementation flexibility of the original design, thus resolving the contradiction between ease of manufacture and productivity.
3Reliability
If control flow semantics are preserved during data flow graph processing, then execution correctness is maintained, but optimization opportunities are limited
Solution Approach 1:
The patent resolves this contradiction by adding a metadata dimension to the data flow graph nodes and edges, where control flow semantics are stored as attributes rather than structural constraints. This dimensional transformation allows optimization algorithms to operate freely on the graph structure while preserving control flow information in the metadata layer, thus achieving both reliability and productivity.
Data Source
AI summary
Processing a data flow graph of a hybrid flow can include a data structure to store metadata for each of a plurality of nodes of a data flow graph of a hybrid flow, processing the data flow graph of the hybrid flow, and defining metadata in the data structure for a particular node affected during processing of the data flow graph of the hybrid flow.