Eye Gaze Refinement in Selfie Images via Facial Landmark Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Selfie photos often have the user's eyes not centered on the front-facing camera due to the user focusing on their own face on the display or eyes being inadvertently half-opened or closed, leading to suboptimal gaze direction.

Innovation Solution

A computing device with a front-facing camera applies facial landmark detection to identify eye regions, extracts attributes, determines an eye gaze score, and generates a modified eye region to ensure the eyes appear centered and open in the selfie image, using coefficients and warping operations to adjust eyelid curvature and pupil direction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If the user views themselves on the display while taking a selfie, then they can see their face clearly, but their eyes are not centered on the front-facing camera

Engineering Contradiction:

Improveselfie viewing convenienceVSAvoideye gaze alignment

Core Design Contradiction:

Ease of operationVSManufacturing precision

Solution Approach 1:

The patent replaces the mechanical/visual feedback mechanism (user viewing themselves on display) with an automated image processing system that detects eye regions, calculates gaze directions, and generates modified images with adjusted eye positions and orientations to achieve proper camera alignment

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system automatically detects eye regions, determines gaze directions, and generates modified images without requiring user intervention. The processing system serves itself by autonomously identifying and correcting eye gaze alignment issues in the selfie images

Inventive Principle:

Principle #25Self-service

2Ease of operation

If the user's eyes are half-opened or completely closed, then the user may be distracted or uncomfortable, but the eyes appear suboptimal in the selfie

Engineering Contradiction:

Improveuser comfort and attentionVSAvoideye appearance quality

Core Design Contradiction:

Ease of operationVSManufacturing precision

Solution Approach 1:

The patent replaces direct visual monitoring of eye state with an automated image analysis system that detects eye region attributes (openness, curvature) and generates modified images with corrected eye appearances while maintaining user comfort

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes parameters of the eye region in the generated image, such as adjusting eye openness, eyelid curvature, and pupil position, to transform suboptimal eye appearances into centered, open eyes while preserving natural appearance and user comfort

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11360555B2Systems and methods for automatic eye gaze refinement

Publication Date: 2022.06.14 CYBERLINK

AI summary

A computing device having a front-facing camera applies facial landmark detection and identifies eye regions in the digital image responsive to the front-facing camera capturing a digital image of an individual. For at least one of the eye regions, the computing device is further configured to extract attributes of the eye region, determine an eye gaze score based on the extracted attributes, generate a modified eye region based on the eye gaze score, and output a modified digital image with the modified eye region.