Asynchronous RNN Fusing Multimodal Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI systems face challenges in automatically discovering correlations between numerical time series data and timestamp textual data, which limits their ability to effectively analyze and predict behaviors over time.
Innovation Solution
An AI platform employing deep learning techniques, specifically using an asynchronous recurrent neural network (AsyncLSTM) to fuse and analyze encoded vectors from both data modalities, iteratively and asynchronously combining features to represent correlated temporal behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If traditional AI systems are used to analyze multi-modal data, then the system structure is simple, but the ability to automatically discover correlations between numerical time series data and timestamp textual data is limited
Solution Approach 1:
The patent combines multiple data modalities (numerical time series data and timestamp textual data) into a unified analysis framework. The system merges heterogeneous data sources with different characteristics into a single multi-modal dataset that can be processed together to discover correlations that would be invisible when analyzing each modality separately.
Solution Approach 2:
The patent introduces an asynchronous recurrent neural network as an intermediary component that bridges the gap between different data modalities. This RNN-based mediator processes and aligns temporal patterns from numerical data with textual information, enabling the discovery of correlations without requiring direct transformation of the original diverse data formats.
2Loss of information
If all textual data is processed and analyzed, then comprehensive analysis is achieved, but computational resources are wasted on irrelevant textual data
Solution Approach 1:
The patent extracts only the relevant features and patterns from the multi-modal dataset. The asynchronous RNN selectively processes textual data that exhibits temporal correlations with numerical time series data, while filtering out irrelevant textual information. This extraction approach maintains comprehensive analysis of meaningful patterns while reducing computational overhead.
Solution Approach 2:
The system performs partial processing on the textual data by focusing computational efforts only on portions of the text that show temporal alignment and correlation with numerical data patterns. Rather than uniformly processing all textual data, the system applies selective attention mechanisms that concentrate resources on the most informative segments.
3Adaptability or versatility
If synchronous processing is used for multi-modal data, then the processing logic is simple, but it cannot handle data with different temporal frequencies and arrival times
Solution Approach 1:
The patent implements dynamic processing where the system adapts its processing rhythm to match the temporal characteristics of each data modality. The asynchronous RNN allows different data streams to be processed at their native temporal frequencies, with the network dynamically adjusting its internal state transitions to handle varying arrival times and frequencies without requiring synchronization to a common clock.
Solution Approach 2:
The system transitions from synchronous time-step processing to an asynchronous event-driven processing model. By introducing an additional dimension of temporal flexibility, the patent allows data to be processed based on event occurrence rather than fixed time intervals, enabling the handling of multi-modal data with different temporal frequencies through a dimensional transformation of the processing approach.
4Difficulty of detecting and measuring
If correlation analysis is performed on all data features, then comprehensive correlation discovery is achieved, but the computational complexity increases significantly
Solution Approach 1:
The patent segments the correlation analysis task into modular components handled by the asynchronous RNN architecture. Instead of computing correlations across all data features simultaneously, the system divides the analysis into sequential processing steps where the RNN progressively builds temporal patterns and identifies correlations in an incremental manner, reducing peak computational requirements while maintaining detection accuracy.
Data Source
AI summary
Embodiments relate to a system, program product, and method for employing deep learning techniques to fuse data across modalities. A multi-modal data set is received, including a first data set having a first modality and a second data set having a second modality, with the second modality being different from the first modality. The first and second data sets are processed, including encoding the first data set into one or more first vectors, and encoding the second data set into one or more second vectors. The processed multi-modal data set is analyzed, and the encoded features from the first and second modalities are iteratively and asynchronously fused. The fused modalities include combined vectors from the first and second data sets representing correlated temporal behavior. The fused vectors are then returned as output data.