NPU Latency Lookup Table via Auxiliary Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining layer latency in neural networks are inefficient and impractical, especially when dealing with billions of neural network architectures, as they involve expensive and time-consuming measurements, and include significant overhead latency related to data processing and transportation.
Innovation Solution
A method that adds an auxiliary layer to the neural network, allowing for the measurement of total latency and overhead latency, with the latter modeled using linear regression based on input and output data sizes, to accurately estimate the latency of individual layers and generate a lookup table for faster latency estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If platform-specific latency measurements are performed for all possible neural network architecture variations, then measurement precision is improved, but productivity deteriorates due to the prohibitively expensive and time-consuming nature of measuring billions of architectures
Solution Approach 1:
The patent segments the latency measurement problem by decomposing the total latency into individual layer latencies. Instead of measuring entire network latencies for billions of architectures, the system measures and stores latency for each layer type separately in a lookup table, then combines these pre-measured layer latencies to evaluate complete architectures, dramatically improving evaluation speed while maintaining accuracy
Solution Approach 2:
The patent applies preliminary action by pre-measuring the latency of individual neural network layers before actual architecture evaluation. These pre-measured layer latencies are stored in a lookup table, allowing rapid evaluation of billions of architectures by simply summing the pre-computed layer latencies without performing new measurements during the evaluation process
2Productivity
If a layer-wise latency lookup table is created using conventional methods, then productivity is improved through fast latency estimation, but measurement precision deteriorates due to inclusion of overhead latency from data processing and transportation
Solution Approach 1:
The patent extracts and separates the overhead latency component from the total measured latency. By adding auxiliary layers with known computational characteristics and measuring their latency contribution, the system identifies and removes the overhead portion (data processing and transportation) from the measurements, leaving only the pure neural network layer latencies in the lookup table
Solution Approach 2:
The patent introduces auxiliary layers as intermediaries to isolate and measure overhead latency. These auxiliary layers serve as mediators between the measurement system and the neural network layers, allowing the system to distinguish between overhead operations and actual layer computations by comparing measurements with and without these intermediary elements
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and a method are disclosed for estimating a latency of a layer of a neural network. A host processing device adds an auxiliary layer to a selected layer of the neural network. A neural processing unit executes an inference operation over the selected layer and the auxiliary layer. A total latency is measured for the inference operation for the selected layer and the auxiliary layer, and an overhead latency is measured for the inference operation. The overhead latency is subtracted from the total latency to generate an estimate of the latency of the layer. In one embodiment, measuring the overhead latency for the inference operation that is associated with the auxiliary layer involves modeling the overhead latency based on a linear regression of an input data size that is input to the selected layer, and an output data size that is output from the auxiliary layer.