Deep Motion Priors for Real-Time Sparse Sensor Motion Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional motion capture systems face challenges in accurately reconstructing motion data when using a small number of sensors or experiencing sensor dropout, particularly in live performance scenarios, and existing methods often require significant computational overhead or are limited to specific motion styles.
Innovation Solution
The use of deep learning techniques with motion priors, specifically deep motion priors in conjunction with inverse dynamics, to constrain pose and handle sparse sensor configurations and sensor dropouts, enabling improved motion estimation and reconstruction even with missing data points in space and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional motion capture mapping methods are used, then the system is simple to operate, but the motion reconstruction accuracy deteriorates when using sparse sensors or experiencing sensor dropout
Solution Approach 1:
The system performs preliminary training offline to learn statistical motion priors and build generative models before actual motion capture. This pre-computed knowledge enables the system to handle sparse sensors and sensor dropout during real-time capture without increasing operational complexity
Solution Approach 2:
The patent introduces statistical motion priors and generative models as intermediary components between sensors and the 3D model mapping process. These intermediaries fill in missing motion information when sensors dropout or are sparse, improving reconstruction accuracy without requiring additional sensors
2Reliability
If statistical motion estimators are used to estimate missing information, then the system can handle sensor dropout, but the computational overhead increases significantly for online estimation
Solution Approach 1:
The system performs computationally intensive statistical model training and motion prior learning during offline preprocessing. This shifts the computational burden from online real-time operation to offline preparation, enabling low-overhead runtime inference that can handle sensor dropout efficiently
Solution Approach 2:
The system computes only the essential motion priors and statistical models needed for handling missing data, rather than performing complete motion reconstruction. This partial computation approach reduces online computational overhead while maintaining robustness to sensor dropout
3Measurement precision
If dense sensor information from multiple depth cameras is used, then the motion capture accuracy is improved, but the device complexity and cost increase
Solution Approach 1:
The system creates a statistical copy or representation of dense motion capture data through learned motion priors. Instead of requiring actual dense sensors during capture, the system uses statistical models trained on dense data to reconstruct motion from sparse sensors, effectively copying the benefits of dense sensing without the hardware complexity
Data Source
AI summary
Training data from multiple types of sensors and captured in previous capture sessions can be fused within a physics-based tracking framework to train motion priors using different deep learning techniques, such as convolutional neural networks (CNN) and Recurrent Temporal Restricted Boltzmann Machines (RTRBMs). In embodiments employing one or more CNNs, two streams of filters can be used. In those embodiments, one stream of the filters can be used to learn the temporal information and the other stream of the filters can be used to learn spatial information. In embodiments employing one or more RTRBMs, all visible nodes of the RTRBMs can be clamped with values obtained from the training data or data synthesized from the training data. In cases where sensor data is unavailable, the input nodes may be unclamped and the one or more RTRBMs can generate the missing sensor data.