Graph API Complex Node Expansion for Portable Vision Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph-based programming interfaces for computer vision applications are cumbersome and inefficient, requiring significant overhead in integrating diverse programming models into image processing pipelines.
Innovation Solution
A graph application programming interface (API) that allows software developers to add complex nodes representing multiple operations, which are expanded into individual nodes by a graph compiler at compile time, reducing overhead and simplifying the integration of different programming models into an image processing pipeline.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing graph-based APIs are used to integrate diverse programming models, then functional portability across hardware platforms is achieved, but development complexity and overhead increase significantly
Solution Approach 1:
The patent segments complex nodes into multiple simpler nodes, each representing a single operation. This segmentation allows the graph compiler to process and compile each node independently, reducing the complexity of integrating diverse programming models while maintaining functional portability across hardware platforms.
Solution Approach 2:
The patent introduces a graph compiler as an intermediary between the high-level graph API and the hardware execution layer. The compiler automatically expands complex nodes, manages the integration of diverse programming models, and generates hardware-specific code, thereby reducing development complexity while preserving adaptability.
2Productivity
If complex nodes are expanded into multiple individual nodes, then compilation and execution efficiency improve, but the number of compilation steps increases
Solution Approach 1:
The patent performs preliminary actions by pre-defining complex nodes with their corresponding operation sets during the API design phase. When a complex node is encountered during compilation, the compiler retrieves the pre-defined expansion pattern and applies it automatically, significantly improving compilation efficiency while minimizing the time required to process complex nodes.
3Ease of operation
If a unified graph API is used to simplify development, then ease of operation improves, but the ability to handle diverse programming models decreases
Solution Approach 1:
The patent implements a unified graph API that is universal in design, capable of accepting nodes from multiple programming models (OpenVX, TensorFlow, PyTorch, etc.) through a common interface. The graph compiler then translates these universal nodes into hardware-specific operations, maintaining ease of development while preserving the ability to handle diverse programming models.
Data Source
AI summary
An image processing system includes one or more processors operative to receive a graph application programming interface (API) call to add a complex node to a graph. The graph includes at least the complex node connected to other nodes by edges that are directed and acyclic. The one or more processors are further operative to process, by a graph compiler at compile time, the complex node by iteratively expanding the complex node into multiple nodes with each node corresponding to one operation in an image processing pipeline. The system further includes one or more target devices to execute executable code compiled from each node to perform operations of the image processing pipeline. The system further includes memory to store the graph compiler and the executable code.