Auxiliary Neural Network On-Demand Parameter Generation Circuit
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks are limited by the memory and power requirements of electronic devices due to the large size of weight matrices, which restricts their implementation and usage.
Innovation Solution
A circuit and method that utilize an auxiliary neural network to generate a set of parameters for a main neural network by applying a non-linear operation to vectors produced by a number generator, reducing the memory and power needs by storing only a smaller set of initial parameters and generating the rest on demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks use large numbers of hidden layers and artificial neurons to improve performance, then the network's computational power and accuracy increase, but the memory and power requirements of electronic devices increase
Solution Approach 1:
The patent applies dynamics by transitioning from static weight matrices to dynamic parameter generation. Instead of storing fixed weights for all neurons, the system generates weights on-demand using a number generator and auxiliary neural network, allowing the same hardware to function as a large network only when needed while using minimal memory at any given time.
Solution Approach 2:
The patent extracts the weight generation function from the traditional neural network structure. By separating the weight matrix storage requirement from the network computation, the invention removes the need to store all weights simultaneously, extracting only the essential parameters (initial weights of the auxiliary network) while generating the rest computationally.
2Reliability
If deep neural networks increase the number of parameters to improve accuracy, then the network's computational power increases, but the power consumption of electronic devices increases
Solution Approach 1:
The system dynamically generates network parameters only when needed for inference, rather than maintaining all parameters in memory simultaneously. This reduces the active computational workload and memory access requirements, thereby lowering power consumption while preserving the ability to perform complex neural network operations.
Solution Approach 2:
Instead of storing unique weight values for all neurons, the system uses a compact set of initial parameters that are computationally transformed into the required weight matrices. This copying approach generates equivalent weight information on-demand, reducing the energy required for parameter storage and retrieval.
3Speed
If traditional neural networks store all weight matrices in memory to enable fast computation, then computational speed increases, but the memory capacity requirements increase
Solution Approach 1:
The system performs preliminary action by pre-computing and storing only the initial parameters of the auxiliary neural network. These initial parameters are then used to generate all subsequent weight matrices through deterministic transformations, eliminating the need to store complete weight matrices while maintaining computational efficiency.
Solution Approach 2:
The patent replaces the mechanical storage system (memory) with a computational generation system. Instead of relying on memory capacity to store all weights, the system uses computational algorithms (number generator and auxiliary network) to produce weights on-demand, substituting memory resources with processing resources.
Data Source
AI summary
The present description concerns a circuit comprising: a number generator (205) configured to generate a sequence of vectors (207, 219) of size , the vector sequence being the same at each start-up of the number generator; a memory (211) configured to store a set of first parameters (Ω) of an auxiliary neural network (204); a processing device configured to generate a set of second parameters of a layer (201) of a main neural network by the application a plurality of times of a first operation (g), by the auxiliary neural network, performing a generation operation from each vector generated by the number generator, each generation delivering a vector of second parameters, the set of the vectors of second parameters forming said set of second parameters; and wherein the number of second parameters is greater than the number of first parameters.