Mixed-Convolution Network Decomposes 3D Convolutions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing convolutional neural networks (CNNs) for image processing are inefficient due to a large number of parameters, which limits their depth and accuracy in tasks like video analysis, as they treat each pixel independently, leading to increased complexity and computational requirements.
Innovation Solution
The use of 3D residual networks and mixed-convolution networks that decompose 3D convolutions into 2D and 1D convolutions, allowing for more efficient parameter sharing and additional nonlinearities, resulting in lower training and testing losses, and enabling better representation of complex functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fully-connected feedforward neural networks are used for image processing, then features can be learned and data can be classified, but the number of parameters increases significantly due to each pixel being a variable and weights being based on image size
Solution Approach 1:
The image is segmented into multiple patches or regions, and each patch is processed independently by separate neural network branches. This segmentation reduces the number of parameters by avoiding full connectivity across all pixels while still capturing local features effectively.
Solution Approach 2:
The patent transitions from treating images as 2D grids of pixels to processing them as 3D volumetric data by adding a depth dimension through multiple patches or layers. This dimensional change enables more efficient parameter sharing and reduces overall parameter count while maintaining feature learning capability.
2Device complexity
If the number of parameters in CNNs is reduced through convolution operations, then the network can be deeper with fewer parameters, but the computational complexity and depth limitations still restrict accuracy in video analysis tasks
Solution Approach 1:
The video data is segmented into multiple frames or temporal segments, which are processed in parallel by separate network branches. This segmentation enables the network to capture temporal dynamics more effectively while maintaining a reasonable parameter count through shared weights across segments.
Solution Approach 2:
The patent employs nested residual blocks within the network architecture, where smaller functional units are nested within larger hierarchical structures. This nesting allows for deeper networks with more layers while controlling parameter growth through weight sharing and residual connections.
3Ease of operation
If traditional CNNs treat each pixel independently, then the processing is simpler, but this leads to increased complexity and computational requirements when trying to capture spatial and temporal relationships
Solution Approach 1:
The input data is divided into multiple segments or patches that are processed in parallel. This segmentation maintains processing simplicity for each individual patch while improving overall computational efficiency through parallelization and better capture of local spatial relationships.
Solution Approach 2:
The patent applies preliminary processing steps such as patch extraction, feature preprocessing, or temporal segmenting before the main network processing. This preliminary action simplifies the subsequent processing by pre-organizing data in a way that reduces computational requirements during the main inference phase.
Data Source
AI summary
In one embodiment, a method includes, by a computing device, receiving a plurality of inputs for a convolution layer of a convolutional neural network, the convolution layer having one or more input channels and one or more output channels, wherein the inputs are received via the input channels, generating, by convolving the inputs with one or more two-dimensional filters, a plurality of intermediate values, and generating, by convolving the intermediate values with one or more one-dimensional filters, a plurality of outputs, wherein the one-dimensional filters receive the intermediate values from the two-dimensional filters via intermediate channels. The method may provide the outputs to a subsequent layer of the convolutional neural network via the output channels. Each of the two dimensions of the two-dimensional filter may correspond to a spatial dimension, and the one dimension of the one-dimensional filter may correspond to a temporal dimension.