Deep Multiplicative Networks for Long-Range Time-Series Prediction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional time-series prediction using neural networks is limited by their reliance on additive elements, making it difficult to effectively combine feed-forward and feedback information for accurate long-term predictions, especially in complex data sets like natural language and video.

Innovation Solution

The use of deep multiplicative networks with encoders and decoders that multiplicatively combine current and past information, allowing for the generation of feed-forward and feedback information across layers, enabling more accurate and invariant representations of time-series data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If traditional neural networks use additive elements to process time-series data, then the network structure is simpler and easier to implement, but the ability to effectively combine feed-forward and feedback information for accurate long-term predictions deteriorates

Engineering Contradiction:

Improveease of implementationVSAvoidprediction accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent changes the fundamental operation type from additive to multiplicative in the neural network layers. This parameter change allows the network to effectively combine feed-forward and feedback information through multiplication, enabling accurate long-term predictions while maintaining a structured layered architecture with encoders and decoders.

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If deep multiplicative networks are used to predict future motor intents and time-series data, then prediction accuracy improves, but computational requirements and energy consumption increase

Engineering Contradiction:

Improveprediction accuracyVSAvoidcomputational energy consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent segments the computational process into distinct encoder and decoder layers, where encoders process feed-forward information and decoders process feedback information. This segmentation allows for efficient organization of multiplicative operations across layers, reducing redundant computations and energy consumption while maintaining high prediction accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies multiplicative operations selectively in specific layers (encoders and decoders) rather than uniformly across the entire network. This partial application of complex operations reduces overall computational burden and energy consumption while still achieving accurate predictions in critical layers.

Inventive Principle:

Principle #16Partial or excessive action

3Device complexity

If traditional neural networks process complex data sets like natural language and video, then the network structure remains simple, but the ability to effectively combine feed-forward and feedback information deteriorates

Engineering Contradiction:

Improvenetwork structure complexityVSAvoidinformation combination effectiveness

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent implements a nested structure where encoder layers and decoder layers are organized hierarchically. Each layer contains both encoder and decoder components that process information at different levels of abstraction. This nesting allows effective combination of feed-forward and feedback information while maintaining an organized, manageable network structure suitable for complex data like natural language and video.

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS11353833B2Systems and methods for learning and predicting time-series data using deep multiplicative networks

Publication Date: 2022.06.07 GOLDMAN SACHS & CO LLC

AI summary

A method includes using a computational network to learn and predict time-series data. The computational network includes one or more layers, each having an encoder and a decoder. The encoder of each layer multiplicatively combines (i) current feed-forward information from a lower layer or a computational network input and (ii) past feedback information from a higher layer or that layer. The encoder of each layer generates current feed-forward information for the higher layer or that layer. The decoder of each layer multiplicatively combines (i) current feedback information from the higher layer or that layer and (ii) at least one of the current feed-forward information from the lower layer or the computational network input or past feed-forward information from the lower layer or the computational network input. The decoder of each layer generates current feedback information for the lower layer or a computational network output.