Handstate Reconstruction Using Multimodal Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems that track human body movements using cameras are limited by occlusion issues, inability to determine force exerted by the user, and complexity in processing multiple users in environments with multiple cameras, while neuromuscular signal-based systems struggle with multidimensional estimates of hand position and movement.
Innovation Solution
A system that combines neuromuscular signals from wearable sensors with camera data to reconstruct handstate information, using statistical models to predict hand position and force, and dynamically adjusts signal weighting based on confidence levels to improve accuracy and robustness, even when the hand is out of the camera's field of view.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If camera-based systems are used to track hand movements, then visual representation of hand position is improved, but the system fails when hand is occluded or out of field of view
Solution Approach 1:
The patent combines camera-based visual tracking with inertial measurement unit (IMU) sensors and neuromuscular signal processing to create a hybrid tracking system. When the camera loses sight of the hand, the IMU sensors continue to track hand position and orientation using accelerometer and gyroscope data, while neuromuscular signals provide additional information about hand configuration and force exertion. This merging of multiple sensing modalities ensures continuous reliable tracking regardless of occlusion or field of view limitations.
Solution Approach 2:
The patent introduces an intermediary computational model that fuses data from camera, IMU, and neuromuscular sensors. This intermediary processing layer integrates visual information with inertial and physiological data, allowing the system to maintain accurate hand state reconstruction even when individual sensors fail or provide incomplete information. The intermediary model reconciles discrepancies between different sensor modalities and provides robust estimation of hand position, orientation, and configuration.
2Reliability
If neuromuscular signals are used to estimate hand position, then tracking is available when hand is out of view, but multidimensional estimates become complex and computationally intensive
Solution Approach 1:
The patent segments the complex neuromuscular signal processing into distinct functional components: muscle activation detection, force estimation, and configuration inference. By dividing the processing task into separate stages and using dedicated algorithms for each aspect, the system manages computational complexity more effectively while maintaining reliable tracking capability when the hand is out of camera view.
Solution Approach 2:
The patent develops a multi-functional processing framework that uses neuromuscular signals for multiple purposes simultaneously: hand position estimation, orientation determination, configuration analysis, and force exertion measurement. This universal approach consolidates multiple tracking functions into a single integrated system, reducing overall computational complexity compared to implementing separate systems for each function.
3Measurement precision
If multiple sensors are combined to improve tracking accuracy, then measurement precision is improved, but system complexity and calibration requirements increase
Solution Approach 1:
The patent performs preliminary calibration and synchronization of multiple sensors during an initial setup phase. The system establishes reference relationships between camera coordinates, IMU measurements, and neuromuscular signal patterns before actual tracking begins. This preliminary action includes synchronizing time stamps from different sensors, calibrating spatial transformations, and creating baseline models of user-specific neuromuscular patterns, which simplifies real-time processing and reduces operational complexity.
Solution Approach 2:
The patent dynamically adjusts processing parameters based on signal quality and contextual information. The system monitors the reliability of each sensor modality and automatically weights their contributions accordingly, changing parameters such as fusion weights, filtering thresholds, and model selection based on current operating conditions. This adaptive parameter adjustment optimizes measurement precision while managing system complexity by activating only necessary processing functions.
Data Source
AI summary
Methods and systems for dynamically reconstructing handstate information based on multiple inputs are described. The methods and systems use data from multiple inputs including a plurality of neuromuscular sensors arranged on one or more wearable devices and one or more cameras. The multimodal data is provided as input to a trained statistical model. The methods and systems determine, based on the data from the multiple inputs, an estimate and representation of the spatial relationship between two or more connected segments of the musculoskeletal representation and force information describing a force exerted by at least one segment of the musculoskeletal representation. The methods and systems further update the computerized musculoskeletal representation based, at least in part, on the position information and the force information.