Parallel Makeup Rendering Pipeline for Video Conference VTO
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual try-on (VTO) technologies face challenges in face movement, lip movement, relative face size, face occlusion, and high frame rate requirements, leading to imperfections and reduced realism in video conferencing and teleconsultation applications.
Innovation Solution
An augmented reality makeup pipeline with parallel processing of effect rendering and object landmark detection, utilizing deep neural networks for face tracking and occlusion handling, and optical flow stabilization to enhance frame rate and stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential processing is used for face tracking and effect rendering, then processing simplicity is maintained, but frame rate and responsiveness deteriorate
Solution Approach 1:
The processing pipeline is segmented into independent parallel threads: face tracking/landmark detection operates in one thread while effect rendering operates in another thread. This segmentation allows both operations to execute simultaneously, doubling the effective processing throughput and achieving high frame rates without overwhelming a single processing core.
Solution Approach 2:
Face tracking and landmark detection are performed in advance on incoming video frames before the effect rendering begins. By preparing the face model and landmark positions beforehand, the system ensures that when rendering is needed, all necessary facial data is already processed and ready, enabling real-time responsive rendering at high frame rates.
2Speed
If high frame rate processing is implemented, then video smoothness is improved, but processing time and computational load increase
Solution Approach 1:
The system maintains continuous face tracking across all video frames without interruption, and continuously renders effects in parallel. This continuous parallel processing ensures that every frame is processed at full speed without idle waiting periods, maximizing the effective frame rate while utilizing processing resources efficiently to minimize overall processing time.
Solution Approach 2:
The processing system dynamically adjusts its operation mode based on detected events. During normal operation, full parallel processing maintains high frame rates. When face occlusion is detected, the system dynamically pauses effect rendering for occluded regions while continuing to track the face, optimizing processing time by avoiding wasted computation on obscured areas while maintaining smooth video flow.
3Reliability
If robust face tracking is used to handle movements and occlusions, then tracking reliability is improved, but processing complexity increases
Solution Approach 1:
The face is segmented into multiple landmark points distributed across key facial features (eyes, nose, mouth, contours). This segmentation allows the tracking system to monitor multiple independent reference points simultaneously, maintaining reliable face tracking even when parts of the face are occluded or moving, without requiring a single complex holistic tracking algorithm.
Solution Approach 2:
Facial landmarks serve as intermediaries between the raw video input and the effect rendering system. These landmark points mediate the tracking process by providing stable, easily detectable reference positions that bridge the gap between complex facial movements and the simpler effect application process, enhancing tracking reliability while keeping the overall system manageable.
4Productivity
If makeup effects are applied during occlusion, then rendering continuity is maintained, but realism and quality deteriorate
Solution Approach 1:
The effect rendering system dynamically responds to occlusion detection by adaptively pausing makeup application in occluded regions while continuing to render in visible regions. This dynamic adjustment maintains rendering continuity in visible areas and ensures effect application accuracy only where the face is actually visible, preventing unrealistic makeup placement on occluded areas while keeping the overall rendering process smooth and continuous.
Data Source
AI summary
There is provided device, system and method embodiments for streamlining the applying of an effect to an object appearing in a sequence of video frames of, for example, video chatting/conferencing or teleconsultation. In an embodiment, operations of i) effect rendering, and ii) object landmark determining are performed in parallel where effect rendering applies an effect in association with landmarks determined for the object to define a sequence of output video frames with the effect applied. Applications of the streamlined application of effects include virtual try on (VTO) of product effects such as makeup, and video chatting/conferencing with virtual try on, or teleconsultation. Embodiments and/or features of a user interface are also provided.