2D Convolution of 4D Video Tensors for Edge Device Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video-processing convolutional neural networks (CNNs) face limitations due to the computational expense and memory demands of 3D convolution operations, making real-time computer vision tasks on edge devices infeasible, as they often require hardware support for 4D or 5D tensors which many edge devices do not have.
Innovation Solution
The approach involves performing 2D convolution on 4D video data tensors instead of 3D convolution on 5D tensors, splitting the 3D kernels into smaller 2D kernels for spatial and temporal processing, and using parallel structures with appropriate reshaping and down-sampling to combine output tensors efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D convolution operations are used on 5D video data tensors, then video processing accuracy is maintained, but computational expense and memory demands increase significantly
Solution Approach 1:
The patent segments the 3D convolution operation into separate 2D spatial convolution and 1D temporal convolution operations. The 5D video tensor is processed by first applying 2D convolutions to spatial dimensions (H, W) and then 1D convolutions to the temporal dimension (T), breaking down the computationally intensive 3D operation into smaller, more manageable steps that reduce memory requirements and computational expense while maintaining accuracy.
Solution Approach 2:
The patent changes the dimensionality of the convolution kernels from 3D to 2D by separating spatial and temporal processing. Instead of using 3D kernels that operate across all dimensions simultaneously, the method uses 2D kernels for spatial features and 1D kernels for temporal features, effectively reducing the dimensional complexity of each individual convolution operation while preserving the full 5D video data structure.
2Measurement precision
If 3D convolution operations are used on 5D video data tensors, then video processing accuracy is maintained, but memory demands increase making edge device deployment infeasible
Solution Approach 1:
The patent segments the 3D convolution operation into separate 2D spatial convolution and 1D temporal convolution operations. The 5D video tensor is processed by first applying 2D convolutions to spatial dimensions (H, W) and then 1D convolutions to the temporal dimension (T), breaking down the computationally intensive 3D operation into smaller, more manageable steps that reduce memory requirements and computational expense while maintaining accuracy.
3Adaptability or versatility
If 3D convolution operations are used on 5D video data tensors, then comprehensive spatiotemporal feature extraction is achieved, but hardware support for 4D or 5D tensors is required which many edge devices lack
Solution Approach 1:
The patent changes the dimensionality of the convolution kernels from 3D to 2D by separating spatial and temporal processing. Instead of using 3D kernels that operate across all dimensions simultaneously, the method uses 2D kernels for spatial features and 1D kernels for temporal features, effectively reducing the dimensional complexity of each individual convolution operation while preserving the full 5D video data structure.
Data Source
AI summary
Methods, systems and media for computer vision using 2D convolution of 4D video data tensors are described. 3D convolution operations performed on 5D input tensors are simulated by performing 2D convolution of 4D tensors instead. A convolution block of a CNN performs two parallel operations: a spatial processing branch performs spatial feature extraction on a 4D tensor using 2D convolution, whereas a temporal processing branch performs temporal feature extraction on a different 4D tensor using 2D convolution. The output tensors of the spatial processing branch and the temporal processing branch are combined to generate an output tensor of the convolution block. The convolution block may include additional operations such as reshaping and/or further convolution operations to generate identically-sized output tensors for each branch, thereby eliminating the need for post-processing of the branches' output tensors prior to combining them.