Memory Augmented Neural Network Trajectory Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for predicting future trajectories of moving objects in autonomous vehicles or advanced driver assistance systems fail to effectively capture multimodal possibilities, relying on single hidden representations and requiring offline retraining, which limits their ability to adapt to online data and remember specific instances from experience.
Innovation Solution
A computer-implemented method using a Memory Augmented Neural Network (MANN) with trained encoder and decoder networks and a key-value database to store and retrieve past and future trajectory encodings, allowing for the generation of multiple future trajectories by conditioning predictions on past observations and leveraging associative memory for online learning and refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If common machine learning models such as Recurrent Neural Networks are used to predict future trajectories, then the system can store past information and make predictions based on long term patterns, but the memory is a single hidden representation that is only addressable as a whole, making memory inspection and focused prediction difficult
Solution Approach 1:
The patent segments the single hidden representation memory into multiple key-value pairs, where each key represents a specific past state and each value represents a corresponding future outcome. This segmentation allows the system to store and retrieve specific trajectory patterns independently, enabling focused prediction on particular scenarios while maintaining overall memory retention capability.
Solution Approach 2:
The patent transitions from a single-dimensional hidden state representation to a multi-dimensional key-value memory structure. By introducing the key dimension (past states) and value dimension (future outcomes) separately, the system enables targeted access to specific trajectory patterns through key matching, rather than relying on the entire hidden state.
2Productivity
If LSTM networks are used to encode temporal information, then the system can process sequential data, but they encode all temporal information in a single hidden state vector which makes it difficult to address individual elements of knowledge and require offline retraining to accommodate new data
Solution Approach 1:
The patent segments the monolithic LSTM hidden state into discrete key-value pairs that can be independently accessed and updated. Each key-value pair represents a specific temporal pattern that can be learned and refined separately, allowing the system to adapt to new data by adding or updating specific keys without retraining the entire model.
Solution Approach 2:
The patent introduces an external memory structure as an intermediary between the LSTM processor and the prediction output. This memory acts as a buffer that stores learned temporal patterns in a structured format, allowing new patterns to be added without disrupting the existing processing pipeline, thus enabling online learning while maintaining efficient temporal information processing.
3Device complexity
If a single hidden representation is used for memory storage, then the system structure is simplified, but memory inspection and focused prediction become difficult
Solution Approach 1:
The patent segments the memory into distinct key-value pairs where each key can be independently inspected and matched against current states. This segmentation enables focused prediction by allowing the system to retrieve only the relevant future outcomes corresponding to specific past patterns, rather than processing the entire hidden state, thereby improving prediction precision without significantly increasing structural complexity.
4Adaptability or versatility
If complex systems with multiple modules such as Variational Autoencoder and Inverse Optimal Control are used, then multiple trajectories can be predicted, but the double complexity makes deployment difficult and requires full offline retraining when new data arrives
Solution Approach 1:
The patent merges the trajectory prediction and memory learning functions into a unified key-value memory structure. Instead of using separate modules like Variational Autoencoder and Inverse Optimal Control, the system combines these functionalities by storing trajectory patterns directly as key-value pairs, where the key is the encoded past state and the value is the encoded future trajectory. This unified approach maintains multimodal prediction capability while reducing architectural complexity and enabling online learning.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
A computer-implemented method for predicting multiple future trajectories of moving objects of interest in the environment of a monitoring device comprising a memory augmented neural network (MANN) comprising at least one trained encoder deep neural network, one trained decoder deep neural network and a key-value database storing keys corresponding to past trajectory encodings and associated values corresponding to associated future trajectory encodings.