Dynamic Control Flow in Tensor Computation Graphs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing AI and machine learning frameworks face challenges in efficiently optimizing control flow in compiled tensor computation graphs, particularly in supporting dynamic control flow primitives and reducing computational overhead.

Innovation Solution

The method involves defining an intermediate representation (IR) of a computation graph, identifying fusable layer nodes, removing and duplicating these nodes into control flow sub-graphs, and enabling dynamic control flow primitives to optimize control flow.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If dynamic control flow primitives are added to static computation graphs, then adaptability and execution efficiency are improved, but device complexity and compiler support requirements worsen

Engineering Contradiction:

Improvedynamic control flow supportVSAvoidcompiler support complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The computation graph is segmented into a main graph and multiple sub-graphs, with control flow primitives serving as entry points. This segmentation allows dynamic control flow to be integrated without requiring the entire graph to be rewritten, reducing compiler complexity while maintaining adaptability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Sub-graphs are nested within the main computation graph structure, with control flow primitives acting as containers that can dynamically select and execute different sub-graphs. This nesting approach enables dynamic behavior while keeping the overall graph structure manageable and compiler-friendly.

Inventive Principle:

Principle #7Nested doll (Nesting)

2Productivity

If fusable layer nodes are removed and duplicated into control flow sub-graphs, then layer fusion optimization is improved, but graph structure complexity increases

Engineering Contradiction:

Improvelayer fusion optimizationVSAvoidgraph structure complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

Fusable layer nodes are extracted from the main graph and duplicated into appropriate sub-graphs based on control flow requirements. This extraction allows each sub-graph to be optimized independently for layer fusion while maintaining the overall graph structure for dynamic execution.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The graph structure is made dynamic by allowing fusable nodes to be selectively duplicated into different sub-graphs based on runtime control flow conditions. This dynamic duplication enables optimization without requiring a fixed, complex static structure.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250045029A1Dynamic pathways for artificial intelligence and tensor computation graphs

Publication Date: 2025.02.06 NEC CORP

AI summary

A method for optimizing control flow in compiled computation graphs includes defining an intermediate representation (IR) of a computation graph, the computation graph IR including a main computation graph having at least one control flow primitive layer node pointing to one or more control flow sub-graph nodes. Fusable layer nodes of the main computation graph are identified and removed from the main computation graph, and the removed fusable layer nodes are duplicated into each of the one or more control flow sub-graph nodes. The method can be applied to machine learning frameworks, for example, for scientific computations such as in medical AI.