Gaze-Guided GAN Training for Human-Realistic Image Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional generative adversarial networks (GANs) struggle to generate highly realistic images because they do not adequately account for human saliency, as humans focus more on certain parts of an image than others, and current models treat all image regions equally, leading to suboptimal performance in matching the target distribution.
Innovation Solution
Incorporating human-in-the-loop (HITL) training using eye and gaze tracking data to improve GANs, where user responses, including gaze saliency maps, are used to adjust the generator's weights and loss functions, emphasizing salient regions for better image generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional GANs treat all image regions equally, then the model structure remains simple, but the generated images lack human perceptual realism
Solution Approach 1:
The patent applies local quality by introducing gaze-weighted loss functions that assign different importance weights to different regions of the generated image based on predicted human gaze patterns. The loss function is modified to emphasize salient regions: L_gaze = L_pixelwise + λ * L_gaze_weighted, where L_gaze_weighted applies higher weights to regions expected to attract human attention. This resolves the contradiction by improving image realism through region-specific optimization without fundamentally changing the overall GAN architecture.
Solution Approach 2:
The patent introduces a gaze prediction model as an intermediary component that predicts human gaze patterns on generated images. This intermediary module processes the generated image and produces gaze maps that guide the training optimization. The gaze prediction model acts as a mediator between the generator and the loss function, enabling the system to incorporate human perceptual priorities without directly modifying the generator's core structure.
2Reliability
If GANs use standard loss functions, then training is computationally efficient, but the models fail to capture human attention patterns
Solution Approach 1:
The patent implements partial action by applying the gaze-weighted loss function selectively to specific image regions rather than uniformly across the entire image. The loss computation focuses computational resources on salient regions identified by the gaze prediction model, using a weighted approach: L_gaze_weighted = Σ w(i,j) * L(i,j) where w(i,j) represents the gaze weight at each pixel location. This reduces unnecessary computational expenditure on non-salient regions while maintaining alignment with human perception.
3Measurement precision
If all image regions are optimized equally, then training converges faster, but salient features are not emphasized enough
Solution Approach 1:
The patent ensures continuity of useful action by integrating the gaze prediction model and gaze-weighted loss computation into the existing GAN training loop. The gaze prediction operates continuously during training, providing real-time guidance on which regions to prioritize. This continuous feedback mechanism allows the model to progressively improve salient feature generation without requiring separate training phases or iterative recalibration, maintaining training momentum while improving saliency detection.
Data Source
AI summary
A system may display a set of images to a user, the set of images includes a plurality of synthetic images output by a generative adversarial network (GAN) includes a generator and a discriminator, and a plurality of non-synthetic images, detect a user response to the set of images, the user response includes at least a gaze of the user relative to the set of images, and train the GAN based at least on the user response, including tuning the generator based on the gaze of the user.