Transposed Convolution Input Tensor Layout Without Zero Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current transposed convolution solutions suffer from reduced matrix multiplication efficiency due to zero-padding and insertion of zeros in input tensors, leading to low computational efficiency and high memory footprint.
Innovation Solution
Optimize input data by reducing zero padding and dividing kernels into sub-kernels for direct convolution, eliminating zero multiplications and flattening, thereby improving matrix multiplication efficiency and reducing memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If zero-padding and insertion of rows and columns of zeros is applied to upsample input data, then spatial dimensions are increased, but matrix multiplication efficiency is reduced due to sparse input tensor
Solution Approach 1:
The input tensor is segmented into non-zero elements only, excluding the zero-padded regions. By identifying and processing only the meaningful non-zero data portions, the system avoids unnecessary multiplications with zero while maintaining the upsampling function. This segmentation approach divides the original tensor into actionable segments that contribute to the output.
Solution Approach 2:
The method extracts and removes the zero elements from the input tensor, keeping only the non-zero elements for processing. This extraction eliminates the harmful factor (zero multiplications) while preserving the essential information needed for transposed convolution, thereby improving computational efficiency without sacrificing output quality.
2Productivity
If input data is flattened into linearly-stored non-zero data to eliminate zeros, then computational efficiency improves, but memory footprint increases and time consumption increases
Solution Approach 1:
Instead of flattening the tensor into a 1D linear structure (which increases memory footprint), the patent maintains the multi-dimensional structure of the input tensor while selectively processing only non-zero elements. This approach operates in the original dimensional space, avoiding the memory overhead associated with flattening while still achieving efficient computation by skipping zero elements.
3Volume of moving object
If conventional transposed convolution is used with zero-padding, then upsampling is achieved, but computational time increases due to unnecessary zero multiplications
Solution Approach 1:
The patent converts the harmful effect of zero-padding (which causes unnecessary computations) into a benefit by using the known zero positions to optimize the computation. By identifying zero regions and excluding them from processing, the system transforms what was previously a computational burden into an opportunity for optimization, reducing computational time while maintaining the desired upsampling output size.
Data Source
AI summary
A new approach is proposed that contemplates system and method to support efficient implementation of transposed convolution for machine learning (ML). Under the proposed approach, input data/tensor to a transposed convolution operation is optimized before the transposed convolution operation and each of a plurality of original kernels used for the transposed convolution operation is divided into a plurality of smaller sub-kernels. A plurality of direct sub-convolutions are then performed by sequentially applying each of the plurality of sub-kernels of each of the original kernels over the optimized input tensor without flattening either the input tensor or the plurality of sub-kernels. The output from the sub-convolutions using the plurality of sub-kernels are then combined as the final output tensor for each of the original kernels for the transposed convolution operation.