Neural Network Layer Evaluation via Persistent Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating the output of neural networks consumes significant processor time and computing resources, particularly due to the high execution time associated with depthwise separable convolution and pointwise convolution layers.
Innovation Solution
Implementing a persistent fusion technique that fuses layers into blocks, divides them into sub-blocks, and assigns these sub-blocks to groups, allowing filter data to be loaded once from main memory and reused across sub-blocks within a group, reducing memory access and redundant computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional neural network evaluation is used with separate layer processing, then computational accuracy is maintained, but processor time and computing resources are excessively consumed
Solution Approach 1:
The patent merges multiple convolution layers (depthwise separable convolution layer and pointwise convolution layer) into a single fused block that is evaluated together. This combining of layers reduces the number of separate evaluation operations, thereby decreasing processor time and computing resource consumption while maintaining computational accuracy.
Solution Approach 2:
The fused block serves multiple functions by integrating both depthwise separable convolution and pointwise convolution operations within a single evaluation unit. This multi-functionality allows the system to perform multiple computational tasks in one processing pass, improving productivity without sacrificing accuracy.
2Measurement precision
If filter data is loaded for each sub-block separately, then computational precision is maintained, but memory access pressure and loading latency increase
Solution Approach 1:
The patent loads filter data from main memory into L2 cache in advance before it is needed for sub-block evaluations. This preliminary action ensures that when sub-blocks are processed, the required filter data is already available in faster memory, eliminating loading latency and maintaining computational precision.
Solution Approach 2:
Once filter data is loaded into L2 cache, it is continuously reused across multiple sub-block evaluations without being reloaded from main memory. This continuous availability of data in fast memory maintains computational precision while eliminating repeated memory access delays.
3Reliability
If multiple memory accesses are performed for each layer evaluation, then data availability is ensured, but L2 cache and main memory pressure increase significantly
Solution Approach 1:
By merging multiple layers into a single fused block, the patent reduces the total number of memory access operations. The filter data is loaded once for the entire fused block rather than multiple times for separate layers, ensuring data availability while significantly reducing memory and cache pressure.
4Measurement precision
If layers are evaluated separately with individual memory accesses, then computational accuracy is preserved, but redundant computations and memory operations occur
Solution Approach 1:
The patent combines multiple convolution layers into a fused block that is evaluated in a single operation. This merging eliminates redundant memory accesses and computations that would occur if layers were processed separately, improving execution efficiency while preserving computational accuracy through the unified evaluation approach.
Data Source
AI summary
Apparatuses, systems, and techniques to evaluate layers of a neural network. In at least one embodiment, one or more layers of a neural network are evaluated based, at least in part, on a single memory access.