Binary Convolution Packing for Low-Latency Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computing resources and high throughput, making them unsuitable for devices with low power consumption and computation capability, such as embedded devices, due to high computation costs and latency in convolutional computations.
Innovation Solution
Binarizing and bit-packing input data and convolution kernels along the channel direction to compress data, dividing input data into blocks matching kernel sizes, and performing convolutional computations on these blocks using unsigned integer operations, which reduces computation cost and improves speed, particularly suitable for CPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks with large number of model parameters are used, then classification accuracy and detection performance are improved, but computing resource consumption and power consumption increase significantly
Solution Approach 1:
The patent changes the parameter representation from full-precision floating-point numbers to binarized integers (±1), fundamentally altering the data format to enable efficient computation. By representing weights and activations as binary values, the network maintains its functional behavior while enabling bitwise operations that consume significantly less energy and computational resources
Solution Approach 2:
The patent substitutes traditional floating-point arithmetic operations with bitwise logical operations (XOR, AND, population count). This replacement transforms the computational mechanism from energy-intensive floating-point unit operations to efficient bitwise operations that can be executed using simpler, lower-power circuitry, directly addressing the power consumption issue
2Measurement precision
If deep neural networks with large number of model parameters are used, then detection performance is improved, but computation time and latency increase
Solution Approach 1:
The patent changes the computational parameters from floating-point arithmetic to binary integer operations. This parameter transformation enables the use of bitwise operations that are significantly faster than traditional floating-point multiplications and additions, directly reducing computation latency while preserving detection performance
Solution Approach 2:
The patent replaces the mechanical arithmetic operation system with a bitwise logical operation system. By substituting multiply-accumulate operations with bitwise XOR and population count operations, the computation time is dramatically reduced, enabling real-time processing applications
3Measurement precision
If conventional convolutional computation is performed with full-precision data, then computation accuracy is maintained, but memory consumption and data transfer requirements increase
Solution Approach 1:
The patent changes the data representation parameters from full-precision floating-point to binarized integers. This parameter change reduces the memory footprint of weights and activations by several times while maintaining sufficient computational accuracy for vision tasks, directly addressing the memory consumption issue
Solution Approach 2:
The patent extracts only the essential computational information needed for accurate processing, discarding the redundant precision of floating-point representations. By keeping only the sign and magnitude information in binary form, the network achieves efficient memory utilization without sacrificing functional accuracy
4Productivity
If binarized data and bitwise operations are used, then computation speed and energy efficiency are improved, but data representation complexity increases
Solution Approach 1:
The patent segments the binarized data into structured formats that facilitate efficient bitwise operations. By organizing binary tensors in specific patterns and grouping operations systematically, the complexity of handling binarized data is managed through structured segmentation rather than random or unorganized representation
Solution Approach 2:
The patent performs preliminary binarization and data reorganization before the main computational process. By pre-processing the data into the optimal binary format and arranging it in computation-friendly structures ahead of time, the actual forward propagation and backward propagation operations can proceed efficiently without dealing with representation complexity during critical computation phases
Data Source
AI summary
The embodiments of this application provide a method and device for optimizing neural network. The method includes: binarizing and bit-packing input data of a convolution layer along a channel direction, and obtaining compressed input data; binarizing and bit-packing respectively each convolution kernel of the convolution layer along the channel direction, and obtaining each corresponding compressed convolution kernel; dividing the compressed input data sequentially in a convolutional computation order into blocks of the compressed input data with the same size of each compressed convolution kernel, wherein the data input to one time convolutional computation form a data block; and, taking a convolutional computation on each block of the compressed input data and each compressed convolution kernel sequentially, obtaining each convolutional result data, and obtaining multiple output data of the convolution layer according to each convolutional result data.