DNN Head Architecture for Local Feature Dependency Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing DNN architectures fail to effectively capture local feature dependencies, leading to suboptimal performance, particularly in resource-constrained environments, due to their reliance on global feature processing that disregards spatial layout cues.
Innovation Solution
A head architecture that partitions output feature maps into feature groups, processes them separately or in parallel, and aggregates local tensors into a global vector for improved feature abstraction, applicable to various backbone networks including CNN, ViT, and MLP.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If global feature processing is used in existing DNN architectures, then the network can process features uniformly, but it fails to capture local feature dependencies and spatial layout cues, leading to suboptimal performance
Solution Approach 1:
The patent divides the feature map into multiple local regions (e.g., quadrants or grid cells) and processes each region separately to capture local feature dependencies. This segmentation allows the network to preserve spatial layout information while maintaining computational efficiency, directly resolving the contradiction between global processing uniformity and local feature capture capability
2Measurement precision
If DNNs use high accuracy models, then they achieve better performance, but they require hundreds of millions of operations per inference, leading to significant computation cost
Solution Approach 1:
By segmenting the feature map into local regions and processing them independently, the patent reduces the computational complexity from O(N^2) global processing to O(k*N^2/d) where d is the number of segments and k is a small constant. This segmentation strategy maintains accuracy by preserving local patterns while significantly improving inference speed
Solution Approach 2:
The patent applies partial processing by focusing computational resources on local regions rather than processing the entire feature map globally. This partial action approach achieves sufficient accuracy for local feature capture while reducing overall computation cost, effectively trading complete global processing for targeted local processing
Data Source
AI summary
A head of a DNN receives an OFM from a backbone network of the DNN. The head can partition the OFM into feature groups having same sizes. The head can further generate local tensors from the features group. To generate a local tensor from a feature group, the head may further partition the feature group into two subgroups, e.g., based on a splitting factor. The spatial sizes of the subgroups depend on the splitting factor. One subgroup can be converted into an attention tensor. The other subject can be converted into a value tensor, which may have the same size as the attention tensor. The attention tensor and value tensor are mixed to produce the local tensor. The local tensors of all the feature groups can be aggregated to form a global vector, which can be fed into a classifier to output one or more classification determined by the DNN.