Neural Network Input Packing for Fused Graph Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Wide & Deep network model's complex structure leads to inefficient data access and execution on hardware accelerators like GPUs and DSAs due to numerous small, discrete data inputs and operators, resulting in reduced efficiency and increased running time.
Innovation Solution
A data processing method involving packing processing to create instance input data for fused computation nodes, reducing data carrying overhead and improving hardware efficiency by optimizing graph compilation through operator fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the Wide & Deep network model uses numerous small, discrete data inputs and operators, then the model structure can be detailed and precise, but the data access efficiency on hardware accelerators deteriorates and running time increases
Solution Approach 1:
The patent merges multiple small, discrete data inputs into larger data batches and combines multiple operators into fused computation nodes. This reduces the number of separate data access operations and operator executions, thereby improving data access efficiency and reducing running time while maintaining model precision through careful design of the fused operations
2Measurement precision
If the Wide & Deep network model uses numerous small, discrete data inputs and operators, then the model structure can be detailed and precise, but the configuration overhead and running time increase
Solution Approach 1:
The patent combines multiple operators into fused computation nodes, reducing the total number of operator executions required. This merging strategy decreases configuration overhead and reduces running time by eliminating redundant operations while preserving the computational precision needed for accurate model predictions
3Measurement precision
If the Wide & Deep network model uses numerous small, discrete data inputs, then the model can process detailed information, but hardware utilization efficiency deteriorates
Solution Approach 1:
The patent merges multiple small data inputs into larger batches and combines multiple operators into fused computation nodes. This approach improves hardware utilization efficiency by reducing the number of separate memory access operations and operator executions, thereby simplifying the hardware execution complexity while still processing detailed information through the fused operations
Data Source
AI summary
A data processing method, an electronic device, and a storage medium. The data processing method is applied to a compiled neural network model, a compiled computation graph corresponding to the neural network model includes M fusion computing nodes, M is a positive integer, and the data processing method includes: packaging data in a plurality of input data groups to obtain at least one instance input data, wherein each instance input data at least includes data required by at least one of M fusion computing nodes when the neural network model executes one-time model inference; reading one instance input data of the at least one instance input data; based on the instance input data, executing an execution instruction corresponding to at least one fusion computing node, to obtain an output of the at least one fusion computing node; and outputting the output of the at least one fusion computing node.