Virtual Makeup Rendering Pipeline With Parallel Face Tracking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing virtual try-on (VTO) technologies face challenges in face movement, lip movement, relative face size, face occlusion, and video frame rate with low latency, leading to imperfections that break the realism of makeup effects in applications like teleconsultation and video conferencing.

Innovation Solution

An augmented reality makeup pipeline with parallel processing of effect rendering and face tracking, optical flow stabilization, and occlusion handling to improve efficiency and realism, using deep neural networks for face detection and stabilization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If sequential processing is used for face tracking and effect rendering, then processing simplicity is maintained, but frame rate and smoothness deteriorate

Engineering Contradiction:

Improveframe rateVSAvoidprocessing pipeline complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The processing pipeline is segmented into two independent parallel threads: one for face tracking/landmark detection and another for effect rendering. This segmentation allows both operations to execute simultaneously without blocking each other, thereby increasing frame rate while maintaining manageable complexity through modular design.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts processing by using previously detected landmarks to initialize current frame tracking, and dynamically applies effects based on real-time landmark updates. This dynamic approach enables parallel processing to remain synchronized and effective without requiring complete re-processing of each frame.

Inventive Principle:

Principle #15Dynamics

2Reliability

If standard face tracking is used, then implementation simplicity is maintained, but stability during face movement and occlusion deteriorates

Engineering Contradiction:

Improveface tracking stabilityVSAvoidtracking algorithm complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system performs preliminary face detection and landmark identification in the first frame, then uses these preliminary results to guide subsequent tracking. This preliminary action establishes a stable baseline that helps maintain tracking reliability during movements and partial occlusions without requiring complex real-time adjustments.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms where detected landmarks from previous frames are used to predict and refine current frame landmarks. This feedback loop continuously adjusts tracking accuracy, maintaining stability during face movements and occlusions by leveraging historical tracking data.

Inventive Principle:

Principle #23Feedback

3Manufacturing precision

If full effect rendering is applied to all frames, then visual quality is maintained, but processing time and latency increase

Engineering Contradiction:

Improvemakeup effect qualityVSAvoidprocessing latency

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system maintains continuous effect rendering by applying makeup effects to every frame without interruption, ensuring visual quality and smoothness. Parallel processing enables this continuous rendering without increasing latency, as the effect application occurs simultaneously with face tracking rather than sequentially.

Inventive Principle:

Principle #20Continuity of useful action

4Speed

If high frame rate processing is implemented, then video smoothness is improved, but processing complexity and computational load increase

Engineering Contradiction:

Improvevideo frame rateVSAvoidprocessing system complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The high frame rate processing is achieved by segmenting the workload into parallel threads dedicated to face tracking and effect rendering. This segmentation distributes computational load across multiple processing units, enabling high frame rates without overwhelming a single processor with complexity.

Inventive Principle:

Principle #1Segmentation

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

Enhances frame rate by up to 2x and stabilizes makeup effects, ensuring smooth and realistic virtual try-on experiences in video conferencing and teleconsultation applications.

Implementation Method 1

an example of a double threaded effects pipeline 300 in which a face tracker 303B and a rendering component 303A execute in parallel to increase the frame rate

Methodology Applied

Scientific EffectOptical flow:

Implementation Method 2

skipping a detecting of the pixel locations for frame t responsive to the optical flow error and using pixel locations responsive to the optical flow function

Methodology Applied

Scientific EffectExponential moving average:

Data Source

PatentUS12632896B2Computing devices, computer program products, and methods for efficient rendering pipeline for makeup including VTO UI option configuration/selection and looks

Publication Date: 2026.05.19 LOREAL SA

AI summary

There is provided device, system and method embodiments for streamlining the applying of an effect (e.g. a virtual try on (“VTO”) effect) to an object appearing in a sequence of video frames such as for video chat, conferencing or teleconsultation applications. Embodiments and/or features of a user interface such as for video chatting/conferencing or teleconsultation are provided. UI features can include live and preview modes, recommendations using predefine looks, look refinement, look ordering, and access to e-commerce purchase interface. In an embodiment, operations of i) effect rendering, and ii) object landmark determining are performed in parallel where effect rendering applies an effect in association with landmarks determined for the object to define a sequence of output video frames with the effect applied.