Face Tracking for Stable Virtual Environment Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing face detection technologies in virtual environments struggle to maintain a stable and centered video feed of a user's face across various positions and rotations, leading to perspective projection distortions and unstable video streams when using portable devices with front-facing cameras.
Innovation Solution
A method that utilizes face detection algorithms in conjunction with gyroscope data to apply perspective correction and crop the video feed, ensuring the user's face remains centered and stable within the video stream, even when the device or user's face moves, by calculating new position and rotation values based on UV coordinates and transmitting only the relevant frames over a network for display on 2D or 3D game objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If face detection algorithms are used to track user's face position, then the system can detect face position changes, but perspective projection distortions occur when the device or user moves
Solution Approach 1:
The patent applies perspective correction by transforming the detected face position parameters (UV coordinates) through mathematical calculations that compensate for device rotation and orientation. The gyroscope data is integrated to adjust the face position parameters, transforming distorted coordinates into corrected positions that maintain proper perspective regardless of device orientation.
2Productivity
If the video feed is transmitted in real-time over a network, then video chat functionality is enabled, but network bandwidth is consumed and latency occurs
Solution Approach 1:
The patent extracts and transmits only the essential data elements needed for video chat - specifically the corrected face position UV coordinates and relevant video frames - rather than transmitting the entire video stream. This selective extraction reduces network bandwidth consumption while maintaining the core functionality of real-time video communication.
3Ease of operation
If the user's face position is constantly updated to control game objects, then precise control is achieved, but computational processing increases
Solution Approach 1:
The patent performs perspective correction calculations and determines corrected face positions in advance, before the video transmission occurs. By pre-processing the face position data and applying corrections based on gyroscope orientation data beforehand, the system reduces the computational load during real-time video chat operations while maintaining precise game object control.
Data Source
AI summary
Methods for a mobile device with a front facing camera to use the detected position of the user's face as a game controller as well as a method of displaying the live streaming face within the video with audio within a virtual environment. Using a reference frame which serves as the origin, the subsequent frames of the live streaming video are used to calculate the change in position of the face detected within the video frame, and calculations of position, rotation, speed and distance are used to position a game object. Additionally this is a method for having a live video stream with live audio textured onto 2D/3D objects within a virtual environment with only the perspective cropped face displayed. This provides the appearance of the face as stable and unmoving even when the device's camera and/or user's face are moving relative to each other.