Neural Network Intermediate Representation for Distributed Training

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing deep learning frameworks are not flexible or effective for large-scale deep neural network model training on distributed devices due to complex parallelism requirements, complicating the use and implementation of distributed deep learning.

Innovation Solution

A method and apparatus for neural network model computation-oriented intermediate representation that parses input model files to acquire topological structure information, constructs logical and physical computation graphs, infers meta attributes, and optimizes memory using a three-level memory structure, enabling efficient distributed training through meta attribute-based abstract representations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If distributed training interfaces are developed to enhance model parallelism, then the parallelism capability is improved, but the complexity of use and implementation increases

Engineering Contradiction:

Improveparallelism capabilityVSAvoidcomplexity of use and implementation

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a computation-oriented intermediate representation (IR) as an intermediary layer between the high-level model definition and the low-level distributed training execution. This intermediate computation graph serves as a mediator that automatically handles the complexity of parallelism decomposition, data distribution, and device mapping, allowing users to define models simply while the system manages the complex distributed training details automatically.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If complex parallelism devices are used for large-scale deep neural network model training, then the training capability is improved, but the flexibility and effectiveness of the framework deteriorates

Engineering Contradiction:

Improvetraining capabilityVSAvoidflexibility and effectiveness

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent implements a dynamic computation graph that can be automatically transformed and optimized for different distributed training scenarios. The system dynamically adjusts the computation graph based on the target devices and training requirements, enabling flexible adaptation to various parallelism configurations (data parallelism, model parallelism, pipeline parallelism) without requiring users to manually configure complex parallelism devices.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the representation parameters of the computation graph from a static mathematical model to a dynamic execution plan with explicit device mappings, data distribution specifications, and parallelism annotations. This parameter transformation allows the same computation graph to be effectively executed on different distributed device configurations while maintaining framework flexibility.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If existing deep learning frameworks are used for single device training, then the simplicity of interface is maintained, but the effectiveness for distributed devices deteriorates

Engineering Contradiction:

Improvesimplicity of interfaceVSAvoideffectiveness for distributed training

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent performs preliminary actions by automatically generating and optimizing the computation-oriented intermediate representation before distributed training execution. The system pre-processes the model definition into a detailed computation graph with explicit parallelism annotations and device mappings, so that when distributed training is initiated, the framework is already prepared and can execute efficiently without requiring users to manually configure complex distributed training parameters.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11823053B2Method of neural network model computation-oriented intermediate representation by constructing physical computation graph, inferring information of input and output tensor edges of each node therein, performing memory optimization on tensor edges, and optimizing physical computation graph

Publication Date: 2023.11.21 ZHEJIANG LAB

AI summary

The disclosure discloses a method of neural network model computation-oriented intermediate representation and apparatus thereof. The method includes the following steps: S1, parsing an input model file so as to acquire topological structure information of a neural network; S2, constructing a logical computation graph; S21, inferring physical layout information of each operator in the logical computation graph; S22, inferring meta attributes of each operator in the logical computation graph; S23, inferring description information of input and output logical tensors of each operator in the logical computation graph; S3, constructing a physical computation graph; S31, generating a physical computation graph, etc.