Neural Network Screen Coordinate Detection for Mobile AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current screen detection methods for vision-based computing devices face challenges in accurately detecting the shape and coordinates of a display in real-time, especially under varying lighting conditions and on devices with limited hardware capabilities, as they require high computational resources and are not reliable for offline deployment.
Innovation Solution
A system and method using a neural network model on a mobile device to detect corner coordinates of a screen display within the camera's field of view, processing input frames to predict corner coordinates and uncertainty values, updating predictions based on confidence thresholds, and normalizing coordinates for accurate pose estimation and augmented reality applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional threshold segmentation techniques are used for screen detection, then the method is simple to implement, but it relies on high contrast images and is not reliable under varying lighting conditions
Solution Approach 1:
The patent replaces conventional mechanical computer vision techniques (thresholding, edge detection) with a neural network-based system. The neural network model learns to detect screen displays and their corner coordinates directly from image data, substituting the need for hand-crafted image processing algorithms and eliminating reliance on high-contrast conditions.
Solution Approach 2:
The patent transforms the detection problem by changing the approach from analyzing pixel intensity gradients and edges to using a neural network that processes entire image frames and predicts corner coordinates directly. This parameter transformation allows the system to be robust to lighting variations by learning invariant features during training.
2Measurement precision
If image segmentation techniques are used to achieve pixel-precise boundaries, then detection accuracy is improved, but computational resource requirements increase significantly
Solution Approach 1:
The patent extracts only the essential information needed for screen detection - the corner coordinates of the display - rather than performing full image segmentation. The neural network is trained to predict these specific coordinates directly, eliminating the need for computationally intensive pixel-level segmentation while maintaining sufficient accuracy for pose estimation and augmented reality applications.
Solution Approach 2:
The patent applies partial action by focusing computational effort only on detecting the four corner points of the screen rather than segmenting every pixel. This selective approach provides sufficient precision for the intended applications while dramatically reducing computational complexity compared to full segmentation.
3Measurement precision
If Mask R-CNN architecture is used for instance segmentation, then detection results are relatively good, but computation requirements are high and it is not reliable for offline deployment on devices with limited hardware capabilities
Solution Approach 1:
The patent employs a lightweight neural network architecture specifically designed for mobile devices with limited hardware capabilities. This simplified model trades some of the computational complexity of Mask R-CNN for faster inference speed, enabling real-time screen detection on consumer mobile devices while maintaining adequate detection accuracy for practical applications.
Solution Approach 2:
The patent applies local quality by designing a neural network architecture optimized for the specific task of screen corner detection rather than general-purpose instance segmentation. The model focuses computational resources on predicting the four corner coordinates, using a streamlined architecture that is efficient for mobile deployment while achieving sufficient local precision for screen pose estimation.
4Productivity
If real-time video processing is performed on vision-based computing devices, then user experience is improved, but computational load increases making accurate screen detection difficult or prohibitive
Solution Approach 1:
The patent applies preliminary action by pre-training the neural network model offline on large datasets of screen images. This pre-trained model can then be deployed on mobile devices for real-time inference, separating the computationally intensive training phase from the real-time detection phase. The pre-trained model achieves both real-time performance and accurate screen detection when deployed on resource-constrained devices.
Data Source
AI summary
Systems, methods, and devices are provided for detecting coordinate information about a screen display shown at a monitor unit. The monitor unit is located within the field of view of a vision-based computing device. A set of source images captured by the camera representative of the screen display shown at the monitor unit are stored. A neural network model is stored in memory. A processor is configured to preprocess the set of source images to obtain an input frame and to process the input frame with the loaded neural network model to predict corner coordinates of the screen display in real-time. In a further feature, the processor may be configured to estimate a pose of the vision-based computing device relative to the monitor unit when a level of confidence in the updated corner coordinates exceeds a threshold.