Self-adaptive Neural Networks Using Segmented Parameter Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural networks face inefficiencies due to large network requirements during training and unnecessary redundancy in inference, leading to waste of computation power and expensive hardware, with existing methods failing to effectively reduce memory usage and improve network structure.
Innovation Solution
The implementation of dictionary-based, self-adaptive neural networks (DSN) that utilize an indexing matrix and mapping dictionary, where gradients are computed and matrices are updated using clustering methods like K-means, allowing for adaptive stochastic gradient descent and reduction of redundant parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional neural networks are used with large network requirements during training, then model accuracy and performance can be improved, but memory usage and computation resources are excessively consumed
Solution Approach 1:
The patent segments the neural network parameters into two distinct matrices: an operating matrix that remains fixed during inference and an indexing matrix that enables dynamic parameter selection. This segmentation allows the network to achieve high accuracy with reduced memory footprint by only storing essential parameters permanently while deriving other parameters computationally during runtime.
Solution Approach 2:
The indexing matrix serves as an intermediary between the operating matrix and the final network parameters. Instead of directly storing all parameters, the system uses the indexing matrix to selectively access and combine parameters from the operating matrix, enabling flexible parameter adaptation without increasing storage requirements.
2Adaptability or versatility
If traditional neural networks are used with fixed structure, then implementation is straightforward, but the network cannot adapt to different tasks or reduce redundant parameters
Solution Approach 1:
The patent introduces dynamic adaptability through the indexing matrix, which can be updated during training to reflect different task requirements. This allows the same base operating matrix to serve multiple functions by changing how parameters are indexed and combined, enabling the network to adapt to different tasks without increasing the fundamental structure.
Solution Approach 2:
The operating matrix is designed to be universal and task-agnostic, serving as a shared parameter store for multiple different tasks. By combining this universal operating matrix with task-specific indexing matrices, the system achieves multi-functionality without duplicating parameters across tasks, thereby reducing redundancy while maintaining versatility.
3Reliability
If comprehensive parameters are stored in traditional neural networks, then all necessary information is available, but redundant parameters waste computation power and hardware resources
Solution Approach 1:
The patent extracts only the essential parameters into the operating matrix, removing redundant parameters from storage. The indexing mechanism then reconstructs the necessary parameter set dynamically during inference by selectively accessing and combining entries from the compact operating matrix, ensuring information completeness without storing all possible parameters explicitly.
Data Source
AI summary
A method of updating a neural network may be provided. A method may include computing gradients for an operating matrix of a current layer of the neural network based on data of at least one of the current layer and at least one other layer of the neural network. The method may also include updating the operating matrix based on the computed gradients. Further, the method may include updating an indexing matrix of the current layer based on the updated operating matrix.