Saliency-Based Frame Resampling for Faster Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks for video processing, particularly in resource-constrained devices, face challenges with high computational costs and inefficiencies due to the increased resolution demands of 4K cameras, leading to prolonged processing times and power consumption, while conventional down-sampling methods degrade detection performance.
Innovation Solution
Implement saliency-based input resampling, where salient regions are sampled at higher resolution and less important regions are down-sampled, using a framework that includes object detection models, saliency map generation, resampling modules, and inverse transformation to maintain detection accuracy with reduced computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional uniform down-sampling is applied to reduce computational load, then processing time and computational cost are reduced, but detection precision and manufacturing precision deteriorate due to loss of fine-grained details
Solution Approach 1:
The patent applies local quality by differentiating resolution requirements across different regions of the input image. Salient regions containing objects of interest are processed at high resolution to preserve detection precision, while non-salient background regions are down-sampled to reduce computational load. This spatially varying resolution strategy resolves the contradiction between processing speed and detection precision.
Solution Approach 2:
The patent segments the input image into salient and non-salient regions using saliency maps generated by neural networks. This segmentation enables selective processing where different resolution treatments are applied to different segments, allowing the system to maintain high detection precision for important regions while achieving faster processing through down-sampling of less important regions.
2Manufacturing precision
If high-resolution input from 4K cameras is processed, then detection precision is improved, but computational cost and energy consumption increase significantly
Solution Approach 1:
The system applies local quality by processing only salient regions at high resolution while down-sampling non-salient regions. This selective high-resolution processing maintains detection precision for objects of interest while significantly reducing the total computational load and energy consumption compared to processing the entire 4K image at full resolution.
Solution Approach 2:
The patent implements partial action by applying high-resolution processing only to the necessary salient regions rather than the entire image. The saliency-based resampling module generates a down-sampled version of the input where only regions containing objects of interest retain high resolution, thereby reducing overall energy consumption while maintaining sufficient detection precision.
3Measurement precision
If full-resolution video frames are processed through deep neural networks, then object detection accuracy is maintained, but processing time increases leading to real-time performance degradation
Solution Approach 1:
The patent applies local quality by maintaining high resolution only in salient regions where objects of interest are located, while down-sampling non-salient regions. This approach preserves object detection accuracy in critical areas while reducing overall processing time by decreasing the computational burden on non-important regions.
Solution Approach 2:
The system performs preliminary action by generating saliency maps and creating down-sampled versions of the input video frames before processing them through the deep neural network. This pre-processing step identifies and preserves only the necessary high-resolution information, enabling faster subsequent processing without sacrificing detection accuracy.
4Device complexity
If uniform down-sampling is applied to reduce input size, then computational complexity is reduced, but fine-grained details and detection performance are degraded
Solution Approach 1:
The patent resolves this contradiction by applying local quality through saliency-based resampling. The system identifies salient regions and preserves their fine-grained details at high resolution while applying down-sampling only to non-salient regions. This selective approach reduces overall computational complexity while maintaining detection performance in critical areas.
Data Source
Figure 1
Figure 2A~2C
Figure 2D
AI summary
A processor-implemented method of video processing using includes receiving, via an artificial neural network (ANN), a video including a first frame and a second frame. A saliency map is generated based on the first frame of the video. The second frame of the video is sampled based on the saliency map. A first portion of the second frame is sampled at a first resolution and a second portion of the second frame is sampled at a second resolution. The first resolution is different than the second resolution. A resampled second frame is generated based on the sampling of the second frame. The resampled second frame is processed to determine an inference associated with the video.