Neural Network Data Conversion for MV Product Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ASIC devices and GPUs face inefficiencies in executing matrix vector products (MV products) due to longer memory access times compared to product-sum operations, making it difficult to process neural networks with MV products in real-time, especially in resource-constrained environments like edge devices.
Innovation Solution
A data conversion apparatus and method that acquires structural data of neural networks, extracts nodes for MV products, converts them into nodes suitable for convolutional layers, and outputs the converted data, allowing for the execution of MV products as matrix matrix products (MM products) to enhance processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If existing ASIC devices and GPUs are used for neural network processing, then device complexity is reduced by using standardized hardware, but processing speed deteriorates due to longer memory access times compared to product-sum operations
Solution Approach 1:
The patent segments the neural network processing into two distinct parts: (1) MV product calculations are extracted and handled separately using optimized routines, while (2) the remaining network processing continues on the standardized hardware. This segmentation allows the time-critical MV operations to be optimized without redesigning the entire system.
Solution Approach 2:
The patent introduces an intermediary component that acts as a bridge between the standardized hardware and the MV product calculations. This intermediary handles the data transformation and coordination, allowing the hardware to operate in its optimal MM product mode while MV products are processed through a specialized interface or software layer.
2Productivity
If devices are tailored for MM products with sufficient memory capacity, then productivity is improved for large-scale neural networks, but adaptability deteriorates for small 1D vector MV products with limited memory
Solution Approach 1:
The patent implements dynamic adaptability by allowing the system to automatically adjust its processing mode based on the input characteristics. When MV products are detected, the system dynamically switches to the optimized processing path, while maintaining the ability to handle standard MM products when appropriate. This dynamic behavior enables the same hardware to serve multiple purposes efficiently.
Solution Approach 2:
The patent creates a universal processing framework that can handle both MM products and MV products through a unified interface. The optimized MV product processing is integrated as a specialized capability within the broader hardware ecosystem, allowing the device to function effectively across different neural network layers and input types without requiring separate dedicated hardware for each case.
3Speed
If dedicated RNN circuits are implemented to speed up MV products, then processing speed is improved for RNN layers, but device complexity increases and applicability to FC layers and convolutional layers is limited
Solution Approach 1:
Instead of creating dedicated hardware circuits for MV products, the patent uses software-based optimization by copying and adapting existing MM product processing routines to handle MV products. This software copying approach achieves the necessary speedup without the complexity of hardware redesign, and the same software optimizations can be applied across different layer types including FC layers and convolutional layers.
Solution Approach 2:
The patent replaces the mechanical/hardware approach of dedicated RNN circuits with a software-based solution. By substituting hardware specialization with optimized software routines, the system achieves similar performance benefits without the associated hardware complexity, while maintaining flexibility to apply the same optimizations across various neural network architectures.
Data Source
AI summary
A new technology capable of processing a neural network including layers including MV products at a high speed is provided. A data conversion apparatus (1) includes a structural data acquisition unit (2) configured to acquire structural data representing a structure of a neural network, a node extraction unit (3) configured to extract a plurality of nodes for a matrix vector product from the structural data, a converting unit (4) configured to convert the extracted plurality of nodes into nodes in a convolutional layer, and a structural data output unit (5) configured to outputting the converted structural data.