Real-time Video Object Proportion Transformation via Mesh Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies cannot efficiently process real-time video to change object proportions in a natural and unrecognizable manner, especially on mobile devices due to resource-intensiveness.
Innovation Solution
A computer-implemented method for real-time video processing that detects and tracks objects using feature reference points, creates a mesh to align and transform object proportions, and uses a square grid to avoid background distortion, implemented on mobile devices with GPU computation for performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional face tracking algorithms and filters are used to change object proportions in video streams, then some basic processing effects can be achieved, but the results are easily recognizable and do not appear natural
Solution Approach 1:
The face is segmented into multiple regions (eyes, nose, mouth, cheeks, jawline) with dedicated control points for each region. This allows independent manipulation of different facial features while maintaining overall facial coherence, resulting in more natural-looking proportion changes rather than uniform distortion.
Solution Approach 2:
Different regions of the face are treated with different transformation qualities and methods. For example, the eyes and mouth regions use different control point densities and transformation algorithms compared to the cheeks and jawline, allowing each region to be transformed in a way that preserves its natural appearance while achieving the desired proportion change.
2Manufacturing precision
If resource-intensive proportion changing algorithms are implemented, then high-quality natural-looking transformations can be achieved, but mobile devices cannot handle such operations in real time
Solution Approach 1:
The computationally intensive task of face transformation is segmented into independent regional operations. Each face region (eyes, nose, mouth, cheeks) is processed separately with its own set of control points and transformation parameters, allowing parallel processing and reducing overall computational burden while maintaining high transformation quality.
Solution Approach 2:
The system uses a limited number of strategically placed control points (e.g., 68 key facial landmarks) rather than processing every pixel or using overly complex algorithms. This partial action approach provides sufficient transformation quality for real-time mobile processing without the excessive computational cost of full-image or high-precision medical-grade algorithms.
3Manufacturing precision
If the entire video frame is transformed to change object proportions, then the object proportions are changed, but the background becomes distorted
Solution Approach 1:
The face (object of interest) is extracted from the video frame as a separate processing target. By isolating the face region and applying transformation only to this extracted area using control points, the background remains untouched and undistorted, while the face proportions are accurately modified.
Solution Approach 2:
The transformation is applied locally only to the face region rather than globally to the entire frame. This local processing approach ensures that the object (face) proportions are changed with high accuracy while the background maintains its original shape and integrity without any distortion.
Data Source
AI summary
Method involving: providing an object in the video that at least partially and at least occasionally is presented in frames of a video; detecting the object in the video, wherein said detection comprises detecting feature reference points of the object; tracking the detected object in the video, wherein the tracking comprises creating a mesh that is based on the detected feature reference points of the object and aligning the mesh to the object in each frame; generating a first set of node points on the created mesh based on a request for changing proportions; generating a second set of node points based on the first set of node points; and transforming the frames of the video in such way that the object's proportions are transformed in accordance with the second set of the node points using the mesh.