Multi-Level Graph APIs for Conditional Image Processing on AI Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph-based programming models for computer vision applications, such as OpenVX, can be cumbersome to use, leading to inefficient execution and increased computational overhead due to unnecessary operations in conditional branches.
Innovation Solution
A graph application programming interface (API) that allows developers to create multi-level graphs with control flow nodes, enabling conditional execution of subgraphs based on runtime conditions, thereby optimizing the image processing pipeline.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing graph-based programming models (e.g., OpenVX) are used for computer vision applications, then functional portability and hardware acceleration access are improved, but the API becomes cumbersome and execution efficiency deteriorates due to unnecessary operations in conditional branches
Solution Approach 1:
The patent segments the graph into subgraphs that can be independently controlled. Each subgraph represents a distinct computational block that can be selectively executed based on conditions, allowing the system to maintain hardware acceleration access while avoiding unnecessary operations in certain branches, thus improving execution efficiency.
Solution Approach 2:
The patent introduces dynamic control flow capabilities to the static graph structure. By enabling conditional execution of subgraphs based on runtime conditions, the system transitions from a fixed execution model to a dynamic one, allowing efficient adaptation to different computational scenarios and hardware architectures.
2Adaptability or versatility
If graph-based programming models are used to represent pipelined operations, then functional abstraction is improved, but operational complexity increases making the API cumbersome to use
Solution Approach 1:
The patent divides the graph into manageable subgraphs that can be added and controlled independently. This segmentation reduces the operational complexity by allowing developers to work with smaller, more manageable units rather than entire complex graphs, while maintaining the functional abstraction benefits.
Solution Approach 2:
The patent introduces control flow nodes as intermediaries between subgraphs. These nodes simplify the API by providing a standardized interface for conditional execution, making it easier to control the flow between subgraphs without exposing the underlying complexity of the graph structure.
3Reliability
If all subgraphs are executed in the image processing pipeline, then computational completeness is improved, but computational overhead increases due to unnecessary operations
Solution Approach 1:
The patent enables dynamic selection of subgraphs to execute based on runtime conditions. This allows the system to maintain computational completeness for required operations while avoiding unnecessary computations in conditional branches, thereby reducing energy consumption and computational overhead.
Solution Approach 2:
The patent implements the ability to skip execution of specific subgraphs when conditions are not met. This skipping mechanism allows the system to maintain reliability by executing only necessary computational steps, avoiding the waste of computational resources on unnecessary operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A graph application programming interface (API) is used to control an image processing flow. A system (700) receives graph API calls to add nodes to respective subgraphs. The system (700) further receives a given graph API call to add a control flow node to a main graph. The given graph API call identifies the subgraphs as parameters. The main graph includes the control flow node connected to other nodes by edges that are directed and acyclic. A graph compiler (760) compiles the main graph and the subgraphs into corresponding executable code (750). At runtime, a condition is evaluated before the subgraphs identified in the given graph API call are executed. One or more target devices (710) execute the corresponding executable code (750) to perform operations of an image processing pipeline while skipping execution of one or more of the subgraphs depending on the condition.