Equilibrium Solver for Deep Neural Network Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant memory for training due to large weight and temporary data storage needs, making them infeasible for lower-spec devices and increasing training time and latency, especially as model complexity grows.
Innovation Solution
The use of a numerical root-finding algorithm to determine an equilibrium point, which replaces the iterative execution of weight-tied layers, reducing memory footprint and computational complexity during training and inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks with many layers are implemented, then model accuracy and capability are improved, but memory requirements increase significantly
Solution Approach 1:
The patent merges the functionality of multiple weight-tied layers into a single layer by representing the iterative function as a whole. Instead of storing and processing separate layer outputs for each layer in the stack, the equilibrium solver computes the fixed point of the iterative function directly, combining the computations of L layers into one operational unit that requires minimal temporary storage.
Solution Approach 2:
The patent extracts the essential computational core from the deep neural network by identifying and solving the fixed point equation of the iterative function. This extraction allows the system to achieve deep network capabilities without storing all intermediate layer representations, removing the memory-intensive portion of the traditional deep network architecture.
2Reliability
If deep neural networks with many layers are implemented, then model accuracy and capability are improved, but training time increases
Solution Approach 1:
The patent replaces the mechanical iterative execution of multiple layers during training with a mathematical root-finding approach. Instead of sequentially processing L layers and storing intermediate states for backpropagation, the system uses numerical methods to directly compute the equilibrium point, substituting the computational mechanics of deep network propagation with more efficient mathematical operations.
3Reliability
If deep neural networks with many layers are implemented, then model accuracy and capability are improved, but computational complexity increases
Solution Approach 1:
The patent creates a universal representation where a single layer with an iterative function can replace any number of weight-tied layers (L=1, 2, 3, ...). This universal layer structure performs the same computational function as deep networks but with constant complexity regardless of the effective depth, making the system adaptable to various task requirements without increasing computational burden.
Data Source
Figure 1~2
Figure 3
Figure 4A~4B
AI summary
A neural network may comprise an iterative function (z[i+1] = f(z[i],θ,c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. As described in this specification, this stack of layers may during training be replaced by the use of a numerical root-finding algorithm to find an equilibrium of the iterative function in which a further execution of the iterative function would not substantially further change the output of the iterative function. Effectively, the stack of layers may be replaced by a numerical equilibrium solver 480. The use of the numerical root-finding algorithm is demonstrated to greatly reduce the memory footprint during training while achieving similar accuracy as state-of-the-art prior art models.