Multi-Task Deep Network for Real-Time Multi-Person Gaze Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current gaze direction estimation methods are inefficient when processing multiple faces in images, as they assume a single calibrated face, leading to reduced processing speed and inability to perform real-time video data processing.
Innovation Solution
A novel method for one-time calculation of multiple face regions' gaze directions using a deep network model with multi-task learning and a self-supervised loss function, enabling simultaneous prediction of gaze directions and face position information, and generating realistic labeled data for training deep learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing gaze direction estimation methods are used that assume a single calibrated face, then the method is simple to implement, but the processing speed decreases and real-time video data processing becomes impossible when multiple persons are present
Solution Approach 1:
The patent merges face detection and gaze direction estimation into a unified deep learning framework. The system processes multiple faces simultaneously through a single computational pipeline, combining face region detection, key point localization, and gaze angle calculation in one integrated model rather than separate sequential operations.
Solution Approach 2:
The deep learning model performs multiple functions simultaneously: it detects face regions, localizes facial key points (瞳孔中心，眉间点，鼻尖点), and estimates gaze directions for multiple persons in parallel. This multi-functional approach eliminates the need for separate processing steps and enables real-time performance.
2Adaptability or versatility
If existing gaze direction estimation methods are used that assume a single calibrated face, then the implementation is straightforward, but the ability to process multiple faces simultaneously is lost
Solution Approach 1:
The system segments the image into multiple face regions through detection, then processes each region independently through the same computational framework. Each face region is handled with its own set of key points and gaze calculation, allowing parallel processing of multiple persons while maintaining individual accuracy.
Solution Approach 2:
The patent introduces facial key points (瞳孔中心，眉间点，鼻尖点) as intermediary elements that bridge face detection and gaze estimation. These key points serve as intermediate representations that enable the system to transition from raw image data to gaze direction calculations in a structured manner.
3Productivity
If multiple faces are processed using existing methods, then comprehensive coverage is achieved, but real-time speed cannot be maintained
Solution Approach 1:
The system performs preliminary detection of face regions and localization of key points before gaze direction calculation. This preliminary processing organizes the data structure and identifies critical features in advance, enabling faster subsequent gaze estimation without sacrificing precision.
Solution Approach 2:
The patent changes the computational parameters by using relative coordinate systems and normalized calculations based on facial key point distances. Instead of absolute measurements, the system uses ratios and relative positions (e.g., distance between瞳孔中心 and 眉间点) that are computationally efficient and maintain accuracy across different face sizes and distances.
Data Source
AI summary
The embodiments of the present disclosure disclose a method for estimating the gaze directions of multiple persons in images, including: a novel method for estimating the gaze directions of multiple persons in images, being able to predict the gaze direction of a single or multiple face regions in an image accurately and in real-time; a novel multi-task learning network structure, which simultaneously predict the gaze directions of multiple face regions in the image through one-time calculation; a self-supervised loss function based on two-dimensional projection, which be used to supervise three-dimensional gaze direction estimation; a novel method for generating gaze direction replacement data of multiple face regions, being able to quickly generate a large amount of labeled realistic data, for training and testing of a deep learning model; the proposed network is trained end-to-end on the constructed data, then the model can predict the gaze direction in real-time during deployment testing.