Camera-Calibrated Neuromuscular Inference for XR Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for tracking user movements in augmented and virtual reality environments face challenges in accurately interpreting neuromuscular signals and calibrating models to represent realistic body movements, particularly in providing precise spatial positioning and orientation of body parts, especially when parts are out of the camera's field of view or occluded.
Innovation Solution
A computerized system that combines neuromuscular signals from wearable sensors with camera data to improve the interpretation of user movements, using inference models to generate accurate musculoskeletal representations, including hand states and gestures, by calibrating and updating models based on camera-captured images and neuromuscular signals, and providing feedback to users through XR environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If wearable sensors are used to track user movements, then spatial positioning and orientation information can be obtained, but the accuracy of interpreting neuromuscular signals deteriorates when body parts are out of camera field of view or occluded
Solution Approach 1:
The patent combines data from multiple sources including wearable sensors (IMUs, neuromuscular sensors) and camera systems into a unified tracking framework. This fusion allows the system to maintain accurate spatial positioning and neuromuscular signal interpretation even when body parts are occluded, as each sensor type compensates for the limitations of others
Solution Approach 2:
The patent introduces intermediate processing layers including muscle activation models and skeletal models that act as mediators between raw sensor data and final movement interpretation. These models help infer hidden states of body parts that are not directly visible to cameras, maintaining reliability during occlusion events
2Manufacturing precision
If inference models are used to generate musculoskeletal representations, then realistic body movement representation is improved, but model calibration complexity increases
Solution Approach 1:
The patent implements preliminary calibration procedures where users perform standardized movements before actual tracking sessions. This preliminary action establishes baseline parameters for individual users, reducing the complexity of real-time model calibration while maintaining high accuracy in musculoskeletal representations
Solution Approach 2:
The system dynamically adjusts model parameters based on detected movement patterns and sensor data quality. By changing parameters adaptively rather than requiring full recalibration, the system maintains high representation accuracy while reducing operational complexity
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Computerized systems, methods, and computer-readable storage media storing code for implementing the methods are provided, in which camera information is used to calibrate one or more inference models used to generate a musculoskeletal representation. One such system includes at least one camera configured to capture at least one image, a plurality of neuromuscular sensors configured to sense and record a plurality of neuromuscular signals from a user, and at least one computer processor. The plurality of neuromuscular sensors are arranged on one or more wearable devices structured to be worn by the user to obtain the plurality of neuromuscular signals. The at least one computer processor is programmed to calibrate the one or more inference models by updating at least one parameter associated with the one or more inference models based, at least in part, on the plurality of neuromuscular signals and the at least one image.