Binarization Matrix Optimization for Deep Learning Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The pruning optimization method in deep learning results in low learning accuracy, and there is a need to improve inference speed and accuracy by reducing the number of parameters in learning models.
Innovation Solution
A learning model optimization device and method that involves setting a binarization matrix with elements of '0' or '1', transforming the parameter matrix using this binarization matrix, performing machine learning to optimize the binarization matrix, and re-randomizing parameters set to '0' to improve learning accuracy and inference speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pruning optimization is applied to reduce the number of parameters, then inference time is shortened and memory usage is reduced, but learning accuracy becomes low
Solution Approach 1:
The patent changes the parameter representation from continuous floating-point values to discrete binarized values (0 or 1). This parameter transformation enables the model to achieve both compression and maintained accuracy through the binarization matrix optimization process, directly resolving the contradiction between parameter reduction and learning accuracy
Solution Approach 2:
The patent applies preliminary binarization to the parameter matrix before the main learning process. By pre-transforming parameters into binary form and optimizing the binarization matrix in advance, the system prepares a compressed representation that maintains learning capability, thus improving inference speed without sacrificing accuracy
2Measurement precision
If the number of parameters is increased to improve learning accuracy, then learning accuracy is improved, but inference time becomes longer and memory usage increases
Solution Approach 1:
The patent transforms parameters into binary form (0 or 1), which reduces the memory footprint and computational complexity while maintaining learning accuracy through optimized binarization matrices. This parameter change enables faster inference without requiring increased parameter counts
Solution Approach 2:
The patent creates a binarized copy of the parameter matrix that serves as an optimized representation. This binary copy requires less memory and processes faster while the optimization process ensures it maintains the necessary learning accuracy, avoiding the need to increase the original parameter count
Data Source
AI summary
A learning model optimization device includes a binarization matrix setting unit configured to set a binarization matrix m in which each element is a numerical value of “0” or “1” and a transformed matrix setting unit configured to set a transformed matrix M having, as an element, a product of each element of the parameter matrix and each element of the binarization matrix in the same row and the same column. The learning model optimization device further includes a learning unit configured to perform machine learning using the transformed matrix M and change a numerical value of each element of the binarization matrix m such that a result of the machine learning approaches teacher data, thereby optimizing the binarization matrix m, and a re-randomization processing unit configured to change again a parameter of the parameter matrix w.