Deep-Learning Model Branching for SoC Inference Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing data size of deep-learning models and limited memory and bandwidth in system-on-chip (SoC) devices, such as neural processing units (NPUs), hinder efficient inference performance and energy efficiency in on-device AI applications.
Innovation Solution
An automated method for modifying the architecture of deep-learning models by replacing dependent layers with independent branches, allowing slice inputs to be processed sequentially, optimizing the model for SoC environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the deep-learning model uses a standard sequential layer architecture, then the model can maintain its original computational flow and accuracy, but the memory bandwidth requirement increases and inference performance deteriorates under limited SoC resources
Solution Approach 1:
The patent applies segmentation by dividing the sequential layer architecture into multiple independent branches. Each branch processes a subset of layers independently, allowing parallel computation and reducing the memory bandwidth requirement. The input tensor is split into multiple slices that are processed simultaneously by different branches, thereby improving inference performance under limited memory bandwidth conditions.
Solution Approach 2:
The patent transforms the single-dimensional sequential processing into a multi-dimensional parallel structure by introducing branch dimensions. The model architecture is reorganized from a linear sequence of layers into a tree-like structure with multiple branches, enabling data to flow through multiple paths simultaneously. This dimensional transformation reduces the bottleneck effect of limited memory bandwidth.
2Use of energy by moving object
If the deep-learning model architecture is modified to reduce memory usage, then energy efficiency improves, but the model complexity and implementation difficulty increase
Solution Approach 1:
The patent segments the model into independent branches that can be processed sequentially, reducing the need for large on-chip memory and thereby lowering energy consumption. The segmentation allows for more efficient memory utilization without significantly increasing architectural complexity, as each branch maintains a similar internal structure to the original layers.
Solution Approach 2:
The system performs automated architecture modification that autonomously generates the branched model structure from the original sequential model. This self-service approach reduces the manual effort and complexity involved in designing and implementing the modified architecture, making it easier to deploy energy-efficient models on SoC devices.
3Productivity
If the deep-learning model processes all layers sequentially as in the original architecture, then the implementation is straightforward, but the inference time increases and productivity decreases on SoC devices
Solution Approach 1:
The patent segments the sequential layer structure into multiple parallel branches, enabling simultaneous processing of different parts of the input data. This segmentation increases inference speed by utilizing parallel computation capabilities of SoC devices, while the modular nature of the branches keeps the implementation complexity manageable.
Solution Approach 2:
The patent creates multiple copies of branch structures that process different slices of the input data in parallel. Each branch is essentially a copy of a subset of the original layers, configured to handle specific portions of the computation. This copying strategy accelerates inference by distributing the computational load across multiple identical or similar processing units.
Data Source
AI summary
The present disclosure relates to automated methods for modifying an architecture of a deep-learning model for improving inference performance based on the deep-learning model performed in a system-on-chip (SoC). An example method for modifying an architecture of a deep-learning model, executed by a computing system, comprises determining a target module among a plurality of layers included in an original deep-learning model, the target module including a plurality of layers having dependency, configuring a plurality of branches using the target module, the plurality of branches being independent of one another, and replacing the target module with the plurality of branches, thereby modifying the architecture of the original deep-learning model.