Neural Network Compilation with Dynamic Tensor Shapes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Dynamic tensor shapes in neural networks pose optimization challenges for compilers, leading to inefficient runtime code and performance penalties due to the need for runtime checks and recompilation with shape changes, as conventional methods struggle to statically evaluate and optimize dynamic shapes.

Innovation Solution

The method involves parsing neural networks using global virtual dimension IDs to define dynamic shapes, performing shape checks during computation graph construction, and generating runtime code, allowing for precomputation and auto-tuning of tensor shapes, thereby reducing runtime overhead and improving performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If dynamic tensor shapes are used in neural networks, then flexibility and adaptability are improved, but compilation optimization and runtime performance deteriorate due to unknown sizes at compile time

Engineering Contradiction:

ImproveflexibilityVSAvoidruntime performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent performs shape checks and validations during the compilation phase before runtime execution. By establishing shape relationships and constraints in advance using symbolic shape inference, the system prepares optimization opportunities beforehand, allowing for better runtime performance despite dynamic shapes.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces symbolic shape inference as an intermediary mechanism between the compiler and runtime execution. This intermediary performs static analysis to infer shape relationships, creating a bridge that enables optimization without requiring fixed shapes, thus resolving the contradiction between flexibility and performance.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If dynamic tensor shapes are used, then model adaptability is improved, but code generation complexity increases due to unknown tensor sizes

Engineering Contradiction:

Improvemodel adaptabilityVSAvoidcode generation complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the shape inference process into distinct phases: static shape analysis during compilation, dynamic shape tracking during execution, and runtime shape validation. This segmentation breaks down the complex code generation task into manageable components, reducing overall complexity while maintaining adaptability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Symbolic shape inference acts as an intermediary layer that automatically analyzes and determines tensor shape relationships, replacing complex manual code generation logic. This intermediary handles the complexity of dynamic shapes, allowing the compiler to generate optimized code without directly managing the complexity of unknown sizes.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If runtime shape checks are performed, then correctness is improved, but execution overhead increases

Engineering Contradiction:

ImprovecorrectnessVSAvoidexecution overhead

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent performs shape checks and validations during the compilation phase before runtime execution. By establishing shape relationships and constraints in advance, the system ensures correctness is verified beforehand, eliminating or minimizing the need for extensive runtime checks and reducing execution overhead.

Inventive Principle:

Principle #10Preliminary action

4Adaptability or versatility

If dynamic shapes are supported, then versatility is improved, but optimization capability deteriorates due to inability to precompute

Engineering Contradiction:

ImproveversatilityVSAvoidoptimization capability

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

The patent performs symbolic shape inference and optimization during the compilation phase, before runtime execution. By analyzing shape relationships and constraints in advance, the system can precompute optimization strategies and generate optimized code specifically tailored to the neural network's shape characteristics, even when shapes are dynamic.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes the approach from fixed numerical shape values to symbolic shape representations that capture shape relationships. This parameter change allows the compiler to perform symbolic analysis and optimization, transforming the optimization capability from being unable to handle dynamic shapes to being able to optimize based on symbolic shape constraints.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20230281433A1Compilation of neural networks with dynamic shapes of tensors

Publication Date: 2023.09.07 NEC LAB EURO GMBH

AI summary

A computer-implemented method for compiling a neural network with tensors having dynamic shapes includes parsing the neural network using a set of global virtual dimension identifications (IDs) that define the dynamic shapes of one or more of the tensors of the neural network. The method further includes performing shape checks while building a computation graph using the set of global virtual dimension IDs, and generating a runtime code of the neural network based on the computation graph.