Batch Normalization Fusion and Quantization for Neural Network Inference

The patent merges the batch normalization layer and convolutional layer into a single fused operation. The batch normalization parameters (mean, variance, scale, shift) are integrated with the convolutional kernel and bias parameters, allowing both operations to execute simultaneously in one pass through the data. This eliminates the need for separate layer implementations and intermediate memory storage, directly resolving the contradiction by improving computation efficiency while maintaining the functional complexity of both layers.

The patent performs preliminary computation of batch normalization parameters (mean and variance) during the training phase and stores them for inference. During inference, these pre-computed parameters are used directly in the fused operation without requiring additional computation passes. This preliminary action reduces the computational burden during inference while maintaining accuracy, addressing the efficiency- complexity contradiction.

By fusing batch normalization and convolutional operations, the patent eliminates intermediate memory storage between these layers. The output of the batch normalization operation is directly fed into the convolutional operation within the same computational pass, removing the need to write intermediate results to memory and read them back. This directly reduces memory access time while maintaining computational accuracy through proper data flow management.

The fused operation enables continuous computation flow where data flows directly from batch normalization to convolution without interruption or memory storage. This continuous action maintains data in registers or fast buffers throughout the computation chain, eliminating memory access bottlenecks while preserving the mathematical accuracy of each operation through proper floating-point arithmetic management.

The patent quantizes the batch normalization parameters (mean, variance, scale, shift) and convolutional parameters (kernel, bias) to 8-bit integer representations. This parameter change from 32-bit floating point to 8-bit integer enables the use of faster integer arithmetic operations while maintaining sufficient precision for the neural network inference task. The quantization process includes calculating scale and zero-point values to map floating-point ranges to integer ranges, preserving computational accuracy.

The patent implements a phase transition in the data representation format from floating-point to integer domain. During the fused batch normalization and convolution operation, data is transformed through quantization to integer format for computation, then converted back to floating-point for the output. This phase transition enables exploitation of faster integer arithmetic hardware while maintaining the precision requirements through careful quantization design and dequantization processes.