ANN Bypass Compiler for Low-Latency Layer Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks (ANNs) executed by deep learning accelerators (DLAs) in memory devices are inefficient in terms of latency, power consumption, and throughput due to the execution of unnecessary layers, which can be improved by implementing bypass paths between layers.
Innovation Solution
Incorporating bypass paths in ANNs to selectively execute certain layers, allowing the compiler to determine the placement of these paths based on resource criteria such as energy consumption, latency, and bandwidth, thereby optimizing the execution process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all layers of the ANN are executed as traditionally designed, then the model accuracy is maintained, but the latency increases and throughput decreases
Solution Approach 1:
The patent implements dynamic bypass paths that can be selectively activated or deactivated based on runtime conditions. The compiler generates multiple execution paths through the ANN layers, and the control circuitry dynamically selects which path to execute based on input data characteristics, previously computed results, or resource availability. This dynamic approach allows the system to skip unnecessary layers when possible, reducing latency, while maintaining full layer execution when accuracy requirements demand it.
2Use of energy by stationary object
If all layers of the ANN are executed, then computational accuracy is preserved, but power consumption increases
Solution Approach 1:
The patent applies partial action by executing only the necessary subset of ANN layers required to achieve the desired accuracy threshold. The compiler analyzes the ANN architecture and identifies bypass paths that can be safely activated without compromising the final output quality. The control circuitry monitors execution and determines when sufficient accuracy has been achieved, allowing the system to terminate early by skipping remaining layers, thereby reducing power consumption while maintaining acceptable computational accuracy.
3Productivity
If bypass paths are implemented to skip layers, then processing speed improves, but the device complexity increases
Solution Approach 1:
The patent implements preliminary action by having the compiler pre-analyze the ANN architecture and pre-generate all possible bypass path configurations before runtime. The compiler performs static analysis to identify which layers can be safely bypassed based on the network topology, data flow dependencies, and accuracy requirements. This preprocessing step creates a compact representation of bypass opportunities that the control circuitry can efficiently evaluate at runtime without performing complex calculations, thereby reducing the runtime complexity burden on the control circuitry.
4Loss of energy
If bypass paths are used to skip layers, then energy consumption decreases, but the loss of time in making decisions increases
Solution Approach 1:
The compiler performs preliminary analysis to pre-compute and store bypass decision criteria in lookup tables or decision trees during the compilation phase. At runtime, the control circuitry simply queries these pre-computed structures based on input data characteristics or execution state, avoiding complex real-time analysis. This preliminary preparation transforms complex decision-making problems into simple table lookups or threshold comparisons, dramatically reducing the time required to make bypass decisions while maintaining energy-efficient execution.
Data Source
AI summary
Apparatuses and methods can be related to compiling instructions for implementing an artificial neural network (ANN) bypass. The bypass path can be used to bypass a portion of the ANN such that the ANN generates an output with a particular level of confidence while utilizing less resources than if the portion of the ANN had not been bypassed. A compiler can determine where to place the bypass path in an ANN.