Neural Network Architecture Reduction via Weight Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for identifying optimal artificial neural network (ANN) architectures are either hindered by the need for expert knowledge or require extensive computational optimization, which is computationally expensive and impractical due to the need for repetitive network trainings.
Innovation Solution
A method that automatically reduces the input parameter space and the number of hidden units in neural networks using regularization techniques and Gram-Schmidt weight transformation, allowing for the identification of an optimal subspace and set of neurons per hidden layer without expensive re-trainings, and selecting the best model through competition frameworks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If brute force computational optimization approach is used to identify optimal network topology and hyperparameter configuration, then the quality and reliability of the ANN model is improved, but the computational cost and time required for repetitive network trainings increases significantly
Solution Approach 1:
The patent applies preliminary action by performing a reduced space transformation on the weight space before conducting the full optimization search. The device transforms the full space ANN architecture into a reduced space architecture with fewer nodes, thereby pre-processing the search space to reduce the number of required trainings while maintaining model quality
Solution Approach 2:
The patent extracts and removes unimportant neurons from the network architecture through the reduced space transformation process. By identifying and eliminating redundant nodes in the weight space, the method reduces computational complexity without significantly impacting model performance
2Measurement precision
If brute force computational optimization approach is used to identify optimal network topology andhyperparameter configuration, then the accuracy of the ANN model is improved, but the time required for repetitive network trainings increases significantly
Solution Approach 1:
The device performs a reduced space transformation as a preliminary step before the main optimization process. This transformation reduces the dimensionality of the weight space and eliminates redundant neurons upfront, so that subsequent optimization requires fewer iterative trainings, thereby reducing total training time while preserving accuracy
Solution Approach 2:
The patent changes the parameters of the neural network by transforming the architecture from full space to reduced space, modifying the number of nodes and the structure of weight connections. This parameter transformation enables faster convergence during optimization while maintaining predictive accuracy
3Use of energy by stationary object
If expert knowledge is used to identify optimal parameterization of artificial neural networks, then the computational cost is reduced, but the requirement for detailed knowledge about neural network theory and practical application increases
Solution Approach 1:
The system performs self-service by automatically conducting the reduced space transformation and identifying optimal architecture parameters through computational algorithms. The device autonomously transforms the weight space and selects important neurons without requiring external expert intervention, making the process accessible to users without specialized neural network knowledge
Data Source
AI summary
A method and apparatus of a device of determining a reduced space neural network architecture is described. In an exemplary embodiment, the device receives a full space neural network architecture, wherein the full space architecture includes a first plurality of nodes and a set of weights. The device may further transform the set of weights. In addition, the device may also reduce the first plurality of nodes using the transformed set of weights to create second plurality of nodes. Furthermore, the device can create the reduced space neural network architecture using the second plurality of nodes.