Neural Network Parameter Compression via Mask Layer Node Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large neural network models, such as Deep Neural Networks, consume significant memory storage space and processing resources, posing challenges for mobile devices and resource-constrained computing systems due to redundancy in neural network parameters, which existing compression techniques like Compressed Sparse Rows (CSR) formats do not adequately address.
Innovation Solution
The method involves identifying and removing redundant nodes from neural network layers by adding mask layers, training them to determine unimportant nodes, and then removing these nodes to create a compressed neural network model, thereby reducing memory storage and processing requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing compression techniques like Compressed Sparse Rows (CSR) formats are used, then storage efficiency is improved, but processing efficiency deteriorates due to the complexity of handling sparse matrix operations
Solution Approach 1:
The patent extracts and removes redundant nodes from the neural network model through mask layers that identify and eliminate unimportant nodes. This extraction process reduces the quantity of parameters without creating sparse matrix structures, thereby maintaining processing efficiency while achieving compression.
Solution Approach 2:
The patent changes the parameter representation by using mask layers with Boolean variables to identify redundant nodes. This parameter transformation allows the system to compress the network by removing entire nodes rather than sparsifying weight matrices, avoiding the processing overhead of sparse matrix operations.
2Quantity of substance
If neural network model size is reduced by removing redundant parameters, then memory storage requirements are improved, but model accuracy may deteriorate
Solution Approach 1:
The patent implements feedback mechanisms during the training of mask layers to identify redundant nodes. The mask layers are trained iteratively with feedback from the neural network's performance, allowing the system to determine which nodes can be removed while maintaining accuracy. This feedback-driven approach ensures that only truly redundant nodes are eliminated.
Solution Approach 2:
The patent performs preliminary identification of redundant nodes through trained mask layers before actual removal. The mask layers are trained in advance to predict which nodes are unimportant, allowing the system to pre-screen and selectively remove only those nodes that will not negatively impact model accuracy.
3Quantity of substance
If mask layers are added to identify redundant nodes, then compression effectiveness is improved, but device complexity increases
Solution Approach 1:
The patent introduces mask layers as temporary, disposable components used during the compression process. These mask layers are trained to identify redundant nodes and then discarded after serving their purpose. The mask layers themselves are removed along with the nodes they identified, leaving no permanent overhead in the final compressed model.
Solution Approach 2:
The patent employs a process where mask layers are introduced, trained, and then discarded after identifying redundant nodes. The temporary complexity of the mask layers is recovered by removing both the mask layers and the identified redundant nodes, resulting in a net reduction of model complexity while achieving compression goals.
Data Source
AI summary
Subject matter disclosed herein may relate to storage and/or processing of signals and/or states representative of neural network parameters in a computing device, and may relate more particularly to compressing signals and/or states representative of neural network nodes in a computing device.