Physical Keyboard Tracking Using Gradient Feature Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AR/VR systems face challenges in accurately tracking physical keyboards due to low-resolution, high-noise images from outward-facing cameras, occlusion by user hands, and suboptimal lighting conditions, which hinder precise feature detection and pose determination.
Innovation Solution
The method involves detecting predefined T, X, and L features formed by the space between keys using a gradient- and variance-based approach, correcting distorted images, and comparing these features to pre-mapped models to render a virtual model that matches the physical keyboard's pose with sub-millimeter accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine-learning approaches are used for keyboard tracking, then tracking can be performed, but the system becomes too slow for real-time applications and imprecise due to low camera quality
Solution Approach 1:
The patent replaces machine-learning approaches with a computer-vision-based gradient and variance detection system. This substitution enables real-time processing by using mathematical gradient calculations and variance thresholds instead of computationally intensive machine learning models, thereby achieving both speed and precision requirements for AR/VR keyboard tracking.
2Difficulty of detecting and measuring
If outward facing cameras are used for tracking, then keyboard location can be detected, but the captured images have low-resolution and high-noise that hinder accurate feature detection
Solution Approach 1:
The patent transforms the image processing approach by changing from direct pixel-based recognition to gradient-based feature detection. By calculating gradients of pixel intensities and using variance thresholds, the system becomes insensitive to image resolution and noise levels, enabling accurate keyboard detection even from low-quality camera captures.
Solution Approach 2:
The patent introduces gradient calculations and variance metrics as intermediary parameters between the raw camera images and the keyboard detection process. These intermediaries serve as robust features that can be reliably extracted from noisy, low-resolution images and used for accurate keyboard localization without requiring high image quality.
3Area of stationary object
If fisheye lenses are used to maximize tracking coverage, then field of view is increased, but the captured images become warped or distorted
Solution Approach 1:
The patent employs gradient-based detection that is inherently invariant to geometric transformations and distortions. By detecting features based on gradient patterns and variance rather than absolute pixel positions or shapes, the system can accurately identify keyboard features even in severely distorted fisheye lens images, maintaining detection accuracy across the entire field of view.
4Reliability
If keyboard features are detected in low-contrast images, then tracking can proceed, but the low-contrast conditions reduce detection reliability
Solution Approach 1:
The patent detects keyboard features by analyzing gradients of pixel intensities and calculating variance across regions, rather than relying on absolute intensity values or contrast. This approach allows the system to identify keyboard features based on relative changes and patterns, maintaining detection reliability even when overall image contrast is low due to lighting conditions.
Data Source
AI summary
In one embodiment, a method includes the steps of capturing an image from a camera viewpoint, the image depicting a physical keyboard, detecting one or more shape features of the physical keyboard depicted in the image by comparing pixels of the image to a predetermined shape template, the predetermined shape template representing visual characteristics of spaces between keyboard keys, accessing predetermined shape features of a keyboard model associated with the physical keyboard, and determining a pose of the physical keyboard based on comparisons between (1) the detected one or more shape features of the physical keyboard and (2) projections of the predetermined shape features of the keyboard model toward the camera viewpoint.