3D Face Tracking for Accurate Video Call Drawing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face difficulties in accurately drawing on moving faces during video calls, especially when drawing on another user's face, due to lack of control over the face's movement in displayed video frames.
Innovation Solution
A method and system that utilize a three-dimensional face tracker to map drawings onto points of a user's face, allowing the drawing to move with the face, incorporating a frame modification application that interacts with the communication client application to modify video frames and transmit the modified frames, enabling accurate and dynamic rendering of drawings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a user draws freehand on displayed video frames during a video call, then the drawing can be applied in real-time, but the drawing accuracy deteriorates when the face is moving
Solution Approach 1:
The patent transitions from 2D video frame coordinates to 3D face model coordinates. By reconstructing the face in three dimensions and mapping drawing coordinates through a transformation matrix that accounts for head pose (pitch, yaw, roll), the system enables accurate drawing placement even when the face moves. The 3D face model provides a stable reference framework that remains consistent despite changes in the 2D video frame appearance.
Solution Approach 2:
The patent replaces the mechanical interaction of direct freehand drawing on 2D video frames with an automated coordinate transformation system. Instead of manually adjusting drawing positions to track face movements, the system uses mathematical transformation matrices that automatically map drawing coordinates from the 2D video frame to the 3D face model, eliminating the need for manual tracking adjustments.
2Adaptability or versatility
If a user draws on another user's face during a video call, then interactive drawing is enabled, but control over face movement is lost
Solution Approach 1:
The system continuously receives feedback from the video stream about the other user's face position and orientation. By processing this feedback through the 3D face tracking and coordinate transformation system, the drawing automatically adapts to the face's movement without requiring the user to manually control or predict face position. The real-time transformation matrix updates ensure drawing accuracy is maintained despite the other user's facial movements.
3Speed
If 2D template tracking is used to track facial expressions, then real-time tracking is achieved, but three-dimensional drawing accuracy is insufficient
Solution Approach 1:
The patent enhances 2D template tracking by introducing a third dimension through 3D face model reconstruction. While the 2D templates provide real-time tracking speed by matching facial features in the video frame, the 3D face model adds depth information and spatial context, enabling accurate drawing placement in three-dimensional space. The transformation matrix bridges the 2D tracking data and 3D drawing coordinates.
Solution Approach 2:
The patent introduces a 3D face model as an intermediary between the 2D video tracking system and the drawing application. The 2D templates track facial features in the video frame, the 3D face model reconstructs the underlying three-dimensional geometry, and the transformation matrix maps between these representations. This intermediary 3D model enables accurate drawing placement while maintaining real-time performance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method comprising: displaying a UI for display of received video; detecting selection of a UI displayed button whilst a received video frame is displayed; in response, disabling the display of video frames received after the received video frame; determining a position of a face of a user in the received frame; receiving a plurality of drawing inputs whilst the button is selected, each drawing input defining image data to be applied at a position on said face; modifying the video frame in accordance with the drawing inputs by applying the image data to each of the positions; detecting a condition and in response, for each video frame received after the detection, determining a position of the face in the frame to determine the location of the positions in the frame, applying the image data to each of the positions, and displaying the modified video frame in the UI.