Edge-Assisted Object Detection for Mobile AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Augmented Reality (AR) and Mixed Reality (MR) systems struggle to detect and classify complex objects in real-time due to high latency in offloading processes, which reduces detection accuracy and degrades the AR experience.
Innovation Solution
The system employs a low latency offloading process that decouples the rendering pipeline from the offloading pipeline, using dynamic RoI encoding and Motion Vector Based Object Tracking to maintain detection accuracy, allowing for adaptive offloading and parallel streaming and inference processes on commodity hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If object detection is offloaded to edge or cloud, then detection capability is improved, but latency increases significantly
Solution Approach 1:
The system segments the object detection process into two parts: lightweight preliminary detection performed locally on the mobile device and comprehensive detection performed on the edge cloud. This segmentation allows the device to perform basic object identification while offloading complex processing to the cloud, thereby reducing end-to-end latency compared to fully cloud-based approaches.
Solution Approach 2:
The mobile device performs preliminary object detection and classification before offloading to the edge cloud. By completing basic detection tasks locally in advance, the system reduces the amount of data and computation that needs to be transmitted and processed remotely, significantly cutting down offloading latency.
2Measurement precision
If deep Convolutional Neural Networks are used for object detection, then detection accuracy is improved, but computational resources required increase
Solution Approach 1:
The system extracts and offloads the computationally intensive deep CNN processing to the edge cloud, while retaining only the lightweight preliminary detection logic on the mobile device. This extraction allows high-accuracy detection using complex networks without burdening the device with significant computational requirements.
Solution Approach 2:
The edge cloud acts as an intermediary between the mobile device and the deep CNN processing. The device sends captured images to the cloud, which performs the computationally intensive detection using deep Convolutional Neural Networks, and returns results to the device. This intermediary approach enables high accuracy without requiring the device to handle complex computations.
3Loss of time
If offloading process latency is reduced, then detection accuracy is maintained, but system complexity increases
Solution Approach 1:
The system segments the processing pipelines into separate rendering pipeline and offloading pipeline, each operating independently. This segmentation allows for optimized latency handling in the offloading pipeline without complicating the rendering pipeline, as each can be tuned and optimized separately for its specific requirements.
Solution Approach 2:
The system implements dynamic adjustment of offloading parameters based on real-time conditions. The mobile device can dynamically modify detection parameters, resolution, and offloading timing based on current computational resources, network conditions, and scene complexity, thereby reducing latency without requiring complex static pipeline designs.
Data Source
AI summary
Systems and methods for edge assisted real-time object detection for mobile augmented reality are provided. The system employs a low latency offloading process, decouples the rendering pipeline from the offloading pipeline, and uses a fast object tracking method to maintain detection accuracy. The system can operate on a mobile device, such as an AR device, and dynamically offloads computationally-intensive object detection functions to an edge cloud device using an adaptive offloading process. The system also includes dynamic RoI encoding and motion vector-based object tracking processes that operate in a tracking and rendering pipeline executing on the AR device.