Higher-Order Neural Networks Using Symmetric Tensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks require multiple layers of linear units to solve non-linear problems, increasing memory and computational complexity, and are limited by having only one hidden layer, making training computationally expensive.
Innovation Solution
Implementing higher-order neural networks with multiple layers that include second-order neurons combined using symmetric tensors, reducing the number of layers and neurons needed, and utilizing a gradient descendant algorithm for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple layers of linear units are used to solve non-linear problems, then the network can resolve non-linear problems, but memory and computational complexity increase
Solution Approach 1:
The patent changes the fundamental parameters of neural network units from linear to higher-order (quadratic and cubic) operations. By using second-order neurons with symmetric tensor products and third-order neurons with symmetric tensor products, the network can represent non-linear relationships more efficiently, reducing the number of layers and neurons needed while maintaining computational tractability through structured parameter organization.
Solution Approach 2:
The patent employs composite neural network architectures combining different orders of neurons (first-order, second-order, and third-order) within the same network. This composite approach allows the network to leverage the computational efficiency of lower-order units while incorporating the non-linear representation capabilities of higher-order units, achieving a balance between complexity and functionality.
2Device complexity
If only one hidden layer is used, then the network structure is simpler, but training becomes computationally expensive
Solution Approach 1:
The patent transforms the training process by changing the mathematical parameters and operations used in each layer. By using symmetric tensor products for second-order and third-order neurons, the network achieves more efficient parameter updates during training, reducing computational cost while maintaining the ability to learn complex non-linear patterns.
Solution Approach 2:
The patent replaces traditional mechanical gradient descent optimization with a specialized training algorithm tailored for higher-order neural networks. This substitution involves using the structural properties of symmetric tensors to guide parameter updates, making the training process more efficient and scalable compared to conventional approaches.
3Device complexity
If higher-order neurons with symmetric tensors are used, then the number of layers and neurons is reduced, but the complexity of implementing symmetric tensor operations increases
Solution Approach 1:
The patent simplifies the implementation of symmetric tensor operations by changing how parameters are organized and computed. By exploiting the symmetry properties of the tensors, the network reduces the number of independent parameters to learn and streamlines the computation, making higher-order operations more manageable and easier to implement in practice.
Data Source
AI summary
Methods and apparatuses for implementing a neural network using symmetric tensors. In embodiments, a system may include a higher order neural network with a plurality of layers that includes an input layer, one or more hidden layers, and an output layer. Each of the input layer, the one or more hidden layers, and the output layer includes a plurality of neurons, where the plurality of neurons includes at least first order neurons and second order neurons, and where inputs at a second order neuron are combined using a symmetric tensor.