Binary Tensor Scale Factor Pairs for Neural Network Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks (CNNs), face challenges in being deployed on resource-constrained devices due to their high computational and memory requirements, with existing binary-weight approaches often resulting in unsatisfactory accuracy and inefficient training procedures.
Innovation Solution
The proposed solution involves exploiting binary structures in pre-trained filter banks through a coarse-to-fine model approximation and tensor expansion to generate binary-weight CNNs, reducing floating-point operations and memory usage while maintaining near-state-of-the-art recognition accuracy, as demonstrated by improved performance on AlexNet and ResNet models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If binary-weight approaches are used to reduce computational requirements, then efficiency is improved, but accuracy deteriorates
Solution Approach 1:
The patent applies parameter changes by transforming the weight representation from continuous floating-point values to discrete binary values (±1). This parameter transformation enables efficient hardware implementation using simple adders and XNOR gates, achieving significant computational speedup and energy reduction while maintaining competitive accuracy through optimized training algorithms and initialization schemes
Solution Approach 2:
The patent uses copying by creating binary copies of the original neural network weights. Instead of directly quantizing weights, the method generates binary weight matrices that replicate the functional behavior of full-precision weights. This copying approach allows the binary network to approximate the original network's predictions while enabling efficient inference on resource-constrained devices
2Quantity of substance
If full-precision CNNs are converted to binary models, then computational requirements are reduced, but model precision deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the weight matrix into multiple binary components or layers. Instead of representing weights with a single binary value, the method segments the weight representation across multiple binary matrices or layers, each contributing to the final computation. This segmentation enables more nuanced weight approximation while maintaining binary efficiency, reducing memory requirements by storing only binary values and segmentation indices rather than full-precision weights
3Productivity
If binary-weight CNNs are trained, then training speed is improved, but training effectiveness deteriorates
Solution Approach 1:
The patent applies preliminary action by performing full-precision training first to obtain optimized weight values, then using these trained weights to initialize or guide the binary network training process. This preliminary full-precision training establishes a good starting point for the binary network, ensuring that the binary weights are initialized in a region of the parameter space that leads to effective convergence. This two-stage approach combines the benefits of full-precision training effectiveness with binary network inference efficiency
Data Source
AI summary
Methods and apparatus are disclosed for enhancing a neural network using binary tensor and scale factor pairs. For one example, a method of optimizing a trained convolutional neural network (CNN) includes initializing an approximation residue as a trained weight tensor for the trained CNN. A plurality of binary tensors and scale factor pairs are determined. The approximation residue is updated using the binary tensors and scale factor pairs.