Mixed Reality Video from Skeletal Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Creating mixed-reality video that includes a moving human is labor-intensive and requires specialized, inflexible, and expensive sensors or markers, making it unsuitable for widespread adoption.
Innovation Solution
A system that estimates the positions of skeletal features in RGB video frames using a commodity camera, allowing for the creation of mixed-reality video without specialized hardware, by employing a video editor application with a skeletal tracker component to map estimated two-dimensional skeletal features to three-dimensional models, enabling animation based on human movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If specialized sensors or markers are used for motion capture, then motion capture accuracy is improved, but device cost and complexity increase
Solution Approach 1:
The patent uses a two-dimensional image copy from an RGB camera as a substitute for complex specialized sensors. Instead of using expensive depth sensors or motion capture cameras, the system captures a standard RGB image and processes it to extract skeletal information, creating a functional copy that achieves motion capture without requiring specialized hardware
Solution Approach 2:
The patent replaces expensive, specialized motion capture sensors with inexpensive, widely available RGB cameras. The system uses commodity devices that can be found in smartphones and webcams, making the technology accessible and eliminating the need for costly specialized equipment while maintaining functional capability
2Measurement precision
If specialized sensors are used for motion capture, then motion capture capability is improved, but flexibility and ease of use deteriorate
Solution Approach 1:
The patent makes the RGB camera universal by enabling it to perform multiple functions: standard color imaging and motion capture. The same camera that takes regular photos can now also capture skeletal information and drive animations, eliminating the need for separate specialized motion capture equipment and making the system more flexible and accessible
3Manufacturing precision
If manual frame-by-frame editing is used to create mixed reality video, then animation precision is improved, but productivity deteriorates
Solution Approach 1:
The system performs automatic skeletal tracking and animation generation without requiring manual frame-by-frame editing. The video processing application automatically detects skeletal features in each frame, tracks their movement, and generates animations autonomously, freeing the user from tedious manual work while maintaining precision through algorithmic consistency
4Measurement precision
If infrared markers and specialized cameras are used, then motion capture accuracy is improved, but adaptability and widespread adoption deteriorate
Solution Approach 1:
The patent creates a functional copy of motion capture capability using standard RGB cameras instead of specialized infrared systems. By processing regular color images to extract skeletal information, the system replicates motion capture functionality without requiring infrared markers or specialized cameras, thereby increasing adaptability and enabling widespread adoption across consumer devices
Data Source
AI summary
Technologies related to generating mixed reality video are described herein. Video frames capture a human, and positions of skeletal features of the human are estimated based solely upon the video frames. A mixed reality video is generated, wherein the mixed reality video includes the video combined with an animation, wherein the animation is based upon movement of the human in the video as represented by the estimated positions of the skeletal features of the human.