Temporal Convolution in Spatiotemporal Neural Networks for Edge AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks struggle to efficiently process both spatial and temporal data, particularly in edge devices, due to high computational requirements and the inability to flexibly encode temporal data, leading to inefficiencies in processing spatiotemporal data formats common in applications like NLP and video streams.
Innovation Solution
A neural network system with temporal and spatial layers, utilizing FIFO buffers and polynomial expansion kernels, enabling efficient temporal convolution operations in both buffer and recurrent modes, reducing computational complexity and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If CNNs are used for spatial data processing, then spatial correlation learning capability is improved, but temporal data processing capability deteriorates
Solution Approach 1:
The patent merges CNNs for spatial processing with RNNs for temporal processing into a unified spatiotemporal network architecture. The network integrates convolutional layers for spatial feature extraction with recurrent layers for temporal sequence modeling, enabling simultaneous processing of both spatial and temporal dimensions in video and other spatiotemporal data.
Solution Approach 2:
The spatiotemporal network is designed with multi-functional capabilities to handle diverse data types including video sequences, natural language processing, and other spatiotemporal formats. The architecture can adapt to different applications by configuring appropriate spatial and temporal processing components within the same framework.
2Adaptability or versatility
If RNNs are used for temporal data processing, then temporal sequence modeling capability is improved, but computational complexity and power consumption increase
Solution Approach 1:
The patent segments the temporal processing into discrete time steps with localized recurrent connections. By processing temporal sequences in manageable segments rather than requiring global temporal context, the computational burden is reduced while maintaining effective temporal modeling capability for edge device deployment.
3Adaptability or versatility
If 2D spatial convolution combined with state-based RNNs is used, then spatiotemporal processing capability is improved, but parameter quantity and device complexity increase
Solution Approach 1:
The patent applies local quality by using localized receptive fields in both spatial and temporal dimensions. The convolutional kernels operate on local spatial regions and the recurrent connections operate on local temporal windows, reducing the need for large global parameters while maintaining effective spatiotemporal feature extraction for edge device deployment.
Data Source
AI summary
Disclosed is a neural network system generally relates to the field of neural networks (NNs). In particular, the present disclosure relates to event-based convolutional neural networks (NNs) that are trained to process spatial and temporal data using kernels represented by polynomial expansion. The event-based convolutional neural networks (NNs) are spatiotemporal neural networks. According to an embodiment, an explicit temporal convolution capability is added through Temporal Event-based Neural Networks (TENN) models. or TENNs in the spatiotemporal neural networks. The TENNs includes a plurality of temporal and spatial convolution layers that combine spatial and temporal features of data for low-level and high-level features. The TENNs as disclosed herein are configured to perform in a buffer mode and recurrent mode that effectively learns both spatial and temporal correlations from the input data.