Neural Network Virtual Layer Insertion for Parameter Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network models, particularly deep neural networks, require substantial memory space for storage due to their large number of parameters, which is a challenge especially on mobile devices with limited memory resources, necessitating effective compression methods that minimize prediction performance loss.
Innovation Solution
The method involves determining target layers in a neural network based on neuron counts, adding virtual layers between these layers to reduce connections, and training the model to obtain an optimized version with reduced parameter count and storage needs, using compression ratios to select models with minimal prediction loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of neurons and hidden layers are increased to improve data fitting capability, then the model prediction accuracy is improved, but the memory space required for storing the model parameters increases substantially
Solution Approach 1:
The patent divides the original dense neural network into multiple stages: initial pruning to remove redundant connections, virtual layer insertion to restructure the network, and iterative refinement. This segmentation allows the model to maintain accuracy while reducing parameter quantity through progressive optimization.
Solution Approach 2:
The patent changes the structural parameters of the neural network by inserting virtual layers with specific neuron counts and adjusting connection patterns. This transforms the original dense structure into a sparser equivalent structure that requires fewer parameters while preserving predictive capability.
2Quantity of substance
If the number of connections between layers is reduced to compress the model size, then the storage requirements are reduced, but the model prediction performance may be degraded
Solution Approach 1:
The patent introduces virtual layers as intermediary structures between existing neural network layers. These virtual layers act as mediators that redistribute and reorganize connections, allowing the network to maintain information flow and prediction accuracy while reducing the total number of direct connections between original layers.
Solution Approach 2:
The patent adds a structural dimension to the neural network by inserting virtual layers, transforming the network from a direct layer-to-layer connection model into a multi-stage structure. This dimensional change allows for more efficient parameter organization and reduced connectivity requirements while preserving functional equivalence.
Data Source
AI summary
The present disclosure provides a method and a device for optimization processing of neural network models. The method includes the following: determining one or more target layers of the neural network model based on the number of neurons at each layer of the neural network model; for each of the one or more target layers, adding a virtual layer between the target layer and a preceding layer of the target layer, where neurons at the virtual layer are separately connected to neurons at the target layer and neurons at the preceding layer of the target layer, and addition of the virtual layer reduces the number of connections between the target layer and the preceding layer of the target layer; and training the neural network model after having added the virtual layers, to obtain an optimized neural network model.