Iterative Normalization for Neural Network Overflow Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-layer neural networks face overflow issues in limited precision arithmetic during batch normalization operations, particularly in calculating the L2-norm of vectors, which can lead to inaccurate results and system failures.
Innovation Solution
Implementing an iterative process for batch normalization that calculates a running value for the square root of the sum of squares operation, using fixed-point arithmetic to prevent overflows and approximate the L2-norm without calculating large values, thereby avoiding overflow risks and optimizing performance in limited precision systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If batch normalization is implemented using standard floating-point arithmetic in multi-layer neural networks, then normalization accuracy is improved, but overflow risks increase in limited precision systems
Solution Approach 1:
The patent changes the numerical representation parameter from standard floating-point to custom precision formats (e.g., 8-bit or 16-bit fixed-point arithmetic). This parameter change allows the system to maintain sufficient normalization accuracy while preventing overflow by constraining the numerical range to fit within limited precision constraints.
Solution Approach 2:
The batch normalization operation is segmented into multiple discrete steps: calculating mean, calculating variance, applying normalization, and applying gamma/beta transformations. Each step is implemented as a separate computational module that can be independently optimized for limited precision arithmetic, preventing cumulative overflow errors.
2Measurement precision
If high precision arithmetic is used for batch normalization calculations, then calculation accuracy is improved, but power consumption increases in resource-constrained devices
Solution Approach 1:
The patent changes the precision parameter from high (32-bit or 64-bit floating-point) to low (8-bit or 16-bit fixed-point) arithmetic. This parameter reduction directly decreases the computational complexity and power consumption of each arithmetic operation while maintaining sufficient accuracy through careful design of the normalization algorithm.
Solution Approach 2:
The patent employs simplified arithmetic operations that are computationally cheaper and consume less power. By using fixed-point arithmetic instead of floating-point, the system trades off some precision for significant reductions in power consumption, making batch normalization viable on mobile and embedded devices.
3Adaptability or versatility
If standard batch normalization is implemented in limited precision systems, then device compatibility is improved, but calculation reliability deteriorates due to overflow
Solution Approach 1:
The patent changes the numerical precision parameter to match the constraints of limited precision systems (e.g., 8-bit or 16-bit fixed-point). This adaptation allows the batch normalization to run on resource-constrained devices while maintaining calculation reliability through overflow prevention mechanisms built into the fixed-point arithmetic implementation.
Solution Approach 2:
The patent implements preventive measures before overflow can occur by using fixed-point arithmetic with predetermined range constraints. The system is designed beforehand to accommodate the limited precision by scaling inputs and intermediates appropriately, cushioning against potential overflow errors before they can affect calculation reliability.
Data Source
AI summary
An embodiment of a semiconductor package apparatus may include technology to process one or more vectors with a sum of squares operation with a layer of a multi-layer neural network, and determine a fixed-point approximation for the sum of squares operation. Other embodiments are disclosed and claimed.