Memory-Guided Video Object Detection for Mobile Real-Time Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video object detection models struggle to achieve real-time performance on mobile devices without significant accuracy loss, as they are constrained by computation and energy limitations.
Innovation Solution
An interleaved object detection model utilizing multiple feature extractor networks and a shared memory layer, where one network prioritizes accuracy and the other prioritizes speed, with an adaptive or fixed interleaving policy to optimize frame analysis, allowing for efficient and accurate object detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single elaborate convolutional neural network design is used to improve accuracy, then object detection accuracy improves, but processing speed and real-time performance deteriorate
Solution Approach 1:
The patent divides the object detection task into multiple sub-tasks by using separate feature extractor networks (e.g., ResNet for accuracy, MobileNet for speed) that process different aspects of the input image. Each network is specialized for specific detection requirements, allowing the system to achieve both high accuracy and real-time performance by segmenting the computational workload rather than using a single monolithic network.
Solution Approach 2:
The patent implements dynamic selection between different feature extractor networks based on the specific detection task requirements. The system can switch between accurate but slow networks and fast but less accurate networks depending on the temporal and computational constraints, enabling adaptive performance optimization that resolves the accuracy-speed tradeoff dynamically rather than statically.
2Productivity
If neural networks are deployed on mobile devices with extreme computation and energy constraints, then portability and real-time operation improve, but detection accuracy deteriorates
Solution Approach 1:
The patent applies partial action by using lightweight feature extractor networks (such as MobileNet or EfficientNet with reduced complexity) that perform sufficient detection at lower computational cost. Instead of always using the most accurate but computationally expensive network, the system uses partial models that provide adequate accuracy for mobile device constraints while enabling real-time operation, accepting that maximum accuracy is not always achieved but sufficient performance is maintained.
3Speed
If multiple feature extractor networks are used with different accuracy-speed tradeoffs, then real-time performance improves, but model complexity and computational cost increase
Solution Approach 1:
The patent implements multi-functionality by designing a unified detection framework that can accommodate multiple feature extractor networks with different characteristics. The system uses a common detection head and shared computational graph that can work with various backbone networks (ResNet, MobileNet, EfficientNet), allowing one detection system to serve multiple accuracy-speed requirements without requiring separate complete detection pipelines for each network type, thus managing complexity through universality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for detecting objects in a video are provided. A method can include inputting a video comprising a plurality of frames into an interleaved object detection model comprising a plurality of feature extractor networks and a shared memory layer. For each of one or more frames, the operations can include selecting one of the plurality of feature extractor networks to analyze the one or more frames, analyzing the one or more frames by the selected feature extractor network to determine one or more features of the one or more frames, determining an updated set of features based at least in part on the one or more features and one or more previously extracted features extracted from a previous frame stored in the shared memory layer, and detecting an object in the one or more frames based at least in part on the updated set of features.