Parallel Effects Rendering Pipeline for Real-Time Makeup Video

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing virtual makeup technologies face challenges in face movement, lip movement, relative face size, face occlusion, and high frame rate requirements, leading to imperfections that break the realism of the effect during applications like teleconsultation and video conferencing.

Innovation Solution

An augmented reality makeup pipeline that performs effect rendering and object landmark determination in parallel, using deep neural networks for face tracking, optical flow stabilization, and occlusion handling to improve efficiency and realism.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If sequential processing is used for effect rendering and object landmark determination, then processing order is simple to manage, but frame rate is reduced and latency increases

Engineering Contradiction:

Improveframe rateVSAvoidprocessing pipeline complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The processing pipeline is divided into separate parallel threads: one thread handles object landmark determination while another thread handles effect rendering. This segmentation allows both operations to execute simultaneously, doubling the frame rate while maintaining manageable complexity through clear thread separation and synchronization mechanisms.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Object landmarks are determined in advance in a separate thread before effect rendering begins. This preliminary action allows the rendering thread to start immediately with the landmark data already prepared, eliminating sequential waiting time and improving overall processing throughput.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If face tracking is performed on every frame to handle face movement and occlusion, then tracking accuracy is improved, but processing time increases and frame rate decreases

Engineering Contradiction:

Improvetracking accuracyVSAvoidframe rate

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system continuously performs object landmark determination on every video frame without interruption, ensuring uninterrupted tracking of face movement and occlusion. This continuous operation maintains high tracking reliability while the parallel processing architecture prevents frame rate degradation.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system uses detected occlusions and landmark positions as feedback to dynamically adjust rendering operations. When occlusion is detected, the system modifies or stops rendering in affected regions, improving tracking accuracy by adapting to changing conditions while maintaining efficient processing.

Inventive Principle:

Principle #23Feedback

3Area of stationary object

If makeup rendering is applied to occluded areas, then rendering coverage is maximized, but realism is broken and artifacts appear

Engineering Contradiction:

Improverendering coverageVSAvoidrealism

Core Design Contradiction:

Area of stationary objectVSReliability

Solution Approach 1:

The system detects occlusions and converts this potentially harmful condition into a benefit by using occlusion information to guide rendering decisions. Instead of rendering over occluded areas and creating artifacts, the system adapts rendering to match occlusion patterns, improving realism by making the virtual makeup appear naturally obscured rather than artificially applied.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

Different rendering strategies are applied to different regions of the face based on local occlusion conditions. Non-occluded areas receive full makeup rendering while occluded areas receive modified or no rendering, creating locally appropriate quality that maintains overall realism and avoids distracting artifacts.

Inventive Principle:

Principle #3Local quality

4Measurement precision

If deep neural networks are used for object landmark detection, then detection accuracy is improved, but processing time increases

Engineering Contradiction:

Improvelandmark detection accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system replaces traditional sequential mechanical processing with parallel computational processing using deep neural networks. By implementing DNN-based landmark detection in a separate parallel thread, the system achieves high detection accuracy while eliminating the time penalty through concurrent execution with rendering operations.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250278871A1Apparatus, systems and methods for an effects rendering pipeline, including makeup effects

Publication Date: 2025.09.04 LOREAL SA

AI summary

There is provided device, system and method embodiments for streamlining the applying of an effect to an object appearing in a sequence of video frames. In an embodiment, operations of i) effect rendering, and ii) object landmark determining are performed in parallel where effect rendering applies an effect in association with landmarks determined for the object to define a sequence of output video frames with the effect applied. Applications of the streamlined application of effects include virtual try on (VTO) of product effects such as makeup, and video chatting/conferencing with virtual try on, or teleconsultation. Embodiments and/or features of a user interface such as for video chatting/conferencing or teleconsultation are also provided.