Built-In Edge Compiler for Framework-Independent NPU Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models are often trained without considering the hardware characteristics of neural processing units (NPUs), leading to inefficient memory management, high power consumption, and low computational processing speed when executed on NPUs.
Innovation Solution
An integrated circuit with a neural processing unit (NPU) and central processing unit (CPU) that compiles neural network models into machine code executable by the NPU, optimizing and converting models to framework-independent and hardware-independent formats, and performing operations like pruning, quantization, and retraining to enhance compatibility and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network models are trained without considering NPU hardware characteristics, then model development is simpler and faster, but execution efficiency on NPU deteriorates with high power consumption and low processing speed
Solution Approach 1:
The patent applies preliminary action by performing model compilation and optimization before NPU execution. The CPU compiles the neural network model into framework-independent format and generates optimized machine code for the NPU in advance, ensuring the model is ready for efficient execution without runtime compilation overhead.
Solution Approach 2:
The patent introduces an intermediary compilation process that translates models from framework-specific formats to framework-independent intermediate representations, then to hardware-specific machine code. This intermediary step enables compatibility across different frameworks while optimizing for specific NPU architectures.
2Productivity
If models are compiled and optimized for specific NPU hardware, then execution efficiency improves, but compatibility across different machine learning frameworks deteriorates
Solution Approach 1:
The patent implements universality by creating a framework-independent intermediate representation that can represent neural network models from different machine learning frameworks uniformly. This intermediate format serves as a universal bridge that maintains compatibility across frameworks while enabling hardware-specific optimization.
Solution Approach 2:
The framework-independent intermediate representation acts as an intermediary that decouples framework-specific details from hardware-specific optimizations, allowing the same intermediate model to be compiled for different NPUs while maintaining framework agnosticism.
3Use of energy by moving object
If model optimization operations like pruning and quantization are applied, then power consumption reduces and execution speed improves, but model complexity and optimization process time increase
Solution Approach 1:
The patent applies preliminary action by performing optimization operations such as pruning and quantization during the compilation phase before NPU execution. This allows the model to be optimized in advance, reducing power consumption during runtime without impacting execution performance.
Solution Approach 2:
The patent utilizes parameter changes by applying quantization to reduce precision of model parameters and pruning to remove redundant connections. These parameter modifications reduce model complexity and computational requirements, enabling efficient execution on resource-constrained NPUs.
Data Source
AI summary
A system includes a substrate on which a first memory, a neural processing unit (NPU) including a plurality of processing elements (PEs) with multiplier-accumulator circuits, a controller, and a second memory, and a central processing unit (CPU) are disposed. The CPU may be configured to execute a universal compiler to perform a conversion for a particular neural network model into a machine code executable by the NPU and store the machine code in the first memory or the second memory. When the particular neural network model, generated by one among a plurality of machine learning frameworks that are incompatible with each other, is received and stored in the first memory, the universal compiler may perform the conversion based on mapping information indicating mapping between elements of machine learning frameworks and functions or operations executable by the CPU or NPU.