Tensorized Neural Networks with Local Nonlinearities for Scalable Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for machine learning are memory greedy and not scalable, limiting their use in monitoring and controlling complex machines, systems, and processes.
Innovation Solution
Convert neural networks into tensorized neural networks with local non-linearities applied to each tensor, reducing memory requirements and computational resources by keeping feature data in a compact tensor network representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks are used for machine learning to improve monitoring and control accuracy, then characterization precision is improved, but memory requirements increase making the system not scalable
Solution Approach 1:
The patent segments the neural network into a tensor network representation where the weight matrix is decomposed into multiple smaller tensors. This segmentation allows the model to maintain high characterization precision while reducing memory requirements by storing tensors in a factored form rather than as a complete weight matrix.
Solution Approach 2:
The patent transitions from a standard neural network weight matrix to a tensor network representation, adding dimensional structure to the weight parameters. This dimensional change allows the same model capacity to be achieved with fewer stored parameters, resolving the contradiction between precision and memory usage.
2Measurement precision
If neural networks become larger and more complex to monitor and control more complex targets, then characterization precision is improved, but device complexity increases making the system unfeasible
Solution Approach 1:
The patent segments complex neural networks into tensor network components, where the weight matrix is decomposed into multiple smaller tensors. This segmentation maintains the model's ability to characterize complex targets accurately while reducing the computational and memory complexity of storing and processing the network parameters.
Solution Approach 2:
The patent changes the parameter representation from dense weight matrices to factored tensor forms. This parameter transformation allows the system to handle more complex targets with higher detection accuracy while the underlying storage and computation complexity remains manageable due to the efficient tensor representation.
3Measurement precision
If intermediate feature vectors are stored in neural networks to process target features, then inference accuracy is improved, but memory allocation increases making computing devices unable to cope
Solution Approach 1:
The patent applies tensor network decomposition to the weight matrices in the neural network, transforming them into multi-dimensional tensor structures. This dimensional transformation enables the network to maintain accurate inference capabilities while significantly reducing the memory allocation required to store the network parameters and intermediate computations.
Data Source
Figure 1~3
Figure 4A~4B
Figure 5~6
AI summary
An apparatus or system configured to: provide a predetermined machine learning routine in the form of a tensorized neural network and associated with a target machine or system or process, with the tensorized neural network comprising a plurality of layers and one or more non-linearities per layer applicable to each tensor of the tensor network of the respective layer; and produce at least one output about the target machine or system or process, the at least one output being inferred by the provided predetermined machine learning routine upon inputting a data set therein.