Learned Tokenization in Vision Neural Networks for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision neural networks face challenges in efficiently processing images and videos due to high computational requirements and latency, despite needing to encode significant visual information.
Innovation Solution
Incorporating TokenLearner layers that apply input-dependent, learned transformations to generate a reduced number of token vectors, allowing subsequent layers to operate with fewer parameters and lower latency while maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If traditional computer vision neural networks process images with full resolution and detail, then measurement precision and information retention are improved, but computational resources and processing time increase significantly
Solution Approach 1:
The image is segmented into multiple patches, which are then processed independently through the neural network. This segmentation allows the model to focus computational resources on local features while maintaining overall image understanding, reducing total computational load while preserving visual information.
Solution Approach 2:
The patent extracts only the most salient and informative patches from the image for detailed processing, while less important regions are processed with lower resolution or skipped entirely. This selective extraction maintains critical visual information while significantly reducing computational resource consumption.
2Measurement precision
If traditional computer vision neural networks use sufficient model capacity to capture visual information, then measurement precision is improved, but device complexity increases
Solution Approach 1:
By segmenting the image into patches and processing them separately, the model can use smaller, less complex network components for each patch while collectively maintaining high accuracy. This divides the complexity burden across multiple simpler processing units.
Solution Approach 2:
Different patches receive different levels of processing complexity based on their importance. Salient regions are processed with higher model capacity while less important regions use simpler processing, optimizing the balance between accuracy and complexity.
3Measurement precision
If traditional computer vision neural networks process high-resolution images, then measurement precision is improved, but speed decreases due to increased latency
Solution Approach 1:
The image is divided into multiple smaller patches that can be processed in parallel, significantly reducing processing latency while maintaining the ability to detect fine details in each patch. This parallelization approach preserves visual detail detection capability while improving processing speed.
Solution Approach 2:
The model applies full processing resolution only to salient patches that require detailed analysis, while other patches receive reduced processing. This partial application of high-resolution processing maintains necessary visual detail detection while reducing overall processing time and latency.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing a network input using a computer vision neural network with learned tokenization.