Binary Ternary Neural Network Weight Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning-based recognition systems require significant memory and computing power, making them unsuitable for low-end or embedded computing devices.
Innovation Solution
The implementation of low-bit approximation using binary/ternary neural networks with specific matrix structures, which imposes a binary/ternary constraint on the weights of deep neural networks, resulting in much greater model size compression and higher speedup compared to full-precision neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deep learning-based recognition systems are implemented, then recognition accuracy is improved, but memory consumption and computing power requirements increase significantly
Solution Approach 1:
The patent applies parameter changes by transforming the weight parameters of neural networks from continuous full-precision values to discrete binary or ternary values. This quantization process reduces the memory footprint of model parameters while maintaining the network's recognition accuracy, directly resolving the contradiction between accuracy and memory consumption
Solution Approach 2:
The patent introduces elaborate matrix structures with specific local patterns (such as structured sparsity or specific binary/ternary configurations) that optimize both compression and accuracy. By imposing structured constraints on weight matrices rather than uniform quantization, the system achieves better local optimization that preserves important features while reducing overall memory requirements
2Reliability
If conventional deep learning-based recognition systems are implemented, then recognition accuracy is improved, but computing power requirements increase significantly
Solution Approach 1:
By changing the parameter representation from full-precision floating-point to binary or ternary discrete values, the computational operations become simpler and more efficient. Binary/ternary neural networks enable the use of optimized hardware operations (such as XNOR and population count operations) that reduce computing power requirements while maintaining recognition accuracy
Solution Approach 2:
The patent substitutes complex floating-point arithmetic operations with simpler discrete operations suitable for binary/ternary networks. This replacement enables more efficient hardware implementation with reduced computational complexity, lowering the power and computing resources needed while preserving recognition performance
3Adaptability or versatility
If model size compression is applied to reduce memory requirements, then compatibility with low-end devices is improved, but processing speed may be affected
Solution Approach 1:
The parameter quantization to binary/ternary values compresses model size significantly, enabling deployment on low-end devices. Simultaneously, the discrete nature of these parameters enables faster hardware operations, thus improving processing speed rather than degrading it, resolving the apparent contradiction between compression and speed
Data Source
AI summary
A mechanism is described for facilitating slimming of neural networks in machine learning environments. A method of embodiments, as described herein, includes learning a first neural network associated with machine learning processes to be performed by a processor of a computing device, where learning includes analyzing a plurality of channels associated with one or more layers of the first neural network. The method may further include computing a plurality of scaling factors to be associated with the plurality of channels such that each channel is assigned a scaling factor, wherein each scaling factor to indicate relevance of a corresponding channel within the first neural network. The method may further include pruning the first neural network into a second neural network by removing one or more channels of the plurality of channels having low relevance as indicated by one or more scaling factors of the plurality of scaling factors assigned to the one or more channels.