Convolutional Neural Network Model Decomposition Layer Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model generation techniques for convolutional neural networks face limitations in increasing processing speed due to the complexity of the network, as they maintain the original layer structure during matrix decomposition, which restricts further optimization.
Innovation Solution
The method involves replacing the convolution layer with a decomposition layer through matrix decomposition, sorting weight parameters to form an equivalent weight matrix, extracting ranks, and building the decomposition layer based on selected ranks, reducing the number of weight parameters and optimizing the layer structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If matrix decomposition is performed while maintaining the original layer structure, then the model can be compressed, but the processing speed cannot be sufficiently increased due to structural complexity
Solution Approach 1:
The convolution layer is segmented into multiple decomposition layers through matrix decomposition. The original weight matrix W is decomposed into multiple factor matrices (e.g., W = A × B × C), creating a sequence of simpler decomposition layers that are easier to process hardware-wise while maintaining the overall transformation function, thus improving processing speed without sacrificing compression
Solution Approach 2:
The patent transforms the problem from maintaining the original 2D layer structure to a multi-dimensional decomposition structure. By introducing intermediate decomposition layers and reorganizing weight parameters across multiple dimensions (input channels, output channels, depth), the system achieves better hardware compatibility and processing speed while preserving model accuracy
2Weight of stationary object
If the number of weight parameters is reduced through matrix decomposition, then hardware downsizing is enabled, but model accuracy may deteriorate
Solution Approach 1:
The patent changes the parameters of the decomposition layers by introducing multiple ranks (r1, r2, ..., rk) with different decomposition factors. By optimizing the rank parameters and the number of decomposition layers, the system finds the optimal balance between reducing weight parameters and maintaining model accuracy, allowing hardware downsizing without significant accuracy loss
3Quantity of substance
If convolution layers are replaced with decomposition layers, then the number of weight parameters is reduced, but the layer structure becomes more complex
Solution Approach 1:
The complex decomposition structure is segmented into multiple independent decomposition layers, each with its own weight matrices and operations. This segmentation makes the overall complex structure more manageable and hardware-friendly, as each layer can be processed independently while collectively achieving the desired parameter reduction
Data Source
AI summary
A model generation method is for generating a machine learning model by replacing a convolution layer of a convolutional neural network with a decomposition layer by matrix decomposition. The model generation method includes sorting weight parameters constituting an original layer of the convolution layer to constitute an equivalent weight matrix equivalent to a weight matrix product which is a product of matrices of weight parameters constituting the decomposition layer, extracting a plurality of ranks by matrix decomposition on the equivalent weight matrix, and building the decomposition layer based on convolution of the weight matrix product corresponding to at least one selected ranks selected from the plurality of ranks.