Marker Tracking for Real-Time Motion Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional motion capture systems face challenges in efficiently and automatically tracking marker movements associated with a user's limbs in real-time within VR or AR environments, particularly in film/television production, where marker re-assignment is tedious and often requires stopping filming.
Innovation Solution
A marker-based tracking system that uses a wearable item, such as a glove, equipped with markers that are tracked using a multi-camera imaging system and a neural network to generate depth maps and determine joint parameters, allowing for real-time hand motion capture without the need for specific poses during filming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional motion capture systems are used, then marker tracking can be performed, but the systems cannot efficiently and automatically map marker movements to avatar movements in real-time
Solution Approach 1:
The patent replaces conventional mechanical marker tracking systems with a neural network-based computer vision system. The neural network automatically detects markers in video frames, determines their 3D positions, and maps them to avatar joints in real-time, eliminating the need for manual marker assignment and achieving both real-time performance and full automation.
Solution Approach 2:
The system enables self-service marker tracking by using the neural network to automatically detect, track, and re-assign markers without human intervention. When markers become occluded or lost, the system automatically re-detects them and re-establishes tracking, allowing continuous operation during filming without requiring actors to pose or operators to intervene.
2Productivity
If automatic marker assignment is implemented, then re-assignment can occur without stopping filming, but the actor must assume a specific pose (e.g., T-pose)
Solution Approach 1:
The patent implements dynamic marker assignment that adapts to the actor's current pose. The neural network continuously learns and updates the mapping between markers and avatar joints based on the actor's movements, allowing automatic re-assignment in any pose rather than requiring static calibration poses like the T-pose. This dynamic adaptation enables seamless tracking throughout the filming process.
3Quantity of substance
If more markers are tracked, then more body parts can be monitored, but the complexity of the tracking system increases
Solution Approach 1:
The patent creates a universal neural network-based tracking system that can handle any number of markers and body parts through a single unified approach. The same neural network architecture and processing pipeline used for tracking a few markers also seamlessly handles tracking of many markers across multiple body parts, eliminating the need for different systems or increased complexity when scaling up the number of tracked elements.
Data Source
AI summary
A tracking system converts images to a set of points in 3D space. The images are of a wearable item that includes markers, and the set of points include representations of the markers. A view is selected from a plurality of views using the set of points, and the selected view includes one or more representations of the representations. A depth map is generated based on the selected view and the set of points, and the depth map includes the one or more representations. A neural network maps labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item. A joint parameter is determined using the mapped labels. The model is updated with the joint parameter, and content provided to a user of the wearable item is based in part on the updated model.