Keypoint Estimation Training With Filtered cGAN Image Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keypoint estimation networks face challenges in achieving high accuracy due to insufficient training data, which leads to overfitting, and current data augmentation techniques fail to generate synthetic images that adequately resemble real-world images.
Innovation Solution
A pre-trained conditional generative adversarial network (cGAN) is used to generate synthetic images conditioned on real-world keypoints, which are then filtered by the keypoint estimation network to ensure quality, iteratively training the network to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the training dataset is expanded using existing data augmentation techniques, then the quantity of training data increases, but the quality of synthetic images deteriorates (they do not resemble real-world images sufficiently)
Solution Approach 1:
The patent uses a pre-trained cGAN generator to copy and synthesize realistic hand images that closely resemble real-world images. The generator creates synthetic training samples by learning the distribution of real hand images and generating new samples that follow the same distribution, thereby increasing training data quantity while maintaining high quality visual resemblance
Solution Approach 2:
The patent employs parameter changes by using a pre-trained cGAN model that has learned optimal parameters for generating realistic hand images. The generator uses learned parameters from training on real images to produce synthetic images with appropriate visual characteristics, and the iterative refinement process adjusts parameters to improve keypoint alignment accuracy
2Quantity of substance
If more training data samples are collected from the real world, then the training dataset quantity increases, but the cost and difficulty of human annotation increases
Solution Approach 1:
Instead of manually annotating real-world images, the patent copies keypoint information from real images and applies it to synthetic images generated by the cGAN. This automated copying process eliminates the need for manual annotation of synthetic samples while maintaining accurate keypoint labels
Solution Approach 2:
The system performs self-service by using the pre-trained cGAN generator to automatically create annotated training data. The generator produces synthetic images with automatically assigned keypoint labels based on the conditional input, eliminating the need for external human annotators and enabling automated dataset generation
3Ease of manufacture
If the keypoint estimation network is trained with limited real-world data, then the training cost decreases, but the network performance deteriorates due to overfitting
Solution Approach 1:
The patent copies real-world data distribution characteristics through the pre-trained cGAN generator, which learns and reproduces the statistical properties of real hand images. This allows the network to train on synthetic copies that maintain the essential features and variability of real data, preventing overfitting while reducing reliance on large quantities of real annotated data
Solution Approach 2:
The patent applies preliminary action by pre-training the cGAN generator on real hand images before using it to generate training data for the keypoint estimation network. This preliminary training equips the generator with the ability to produce realistic synthetic images, which are then used to augment or replace real training data, improving network performance while controlling training costs
Data Source
AI summary
Method and devices for training a keypoint estimation network are described. In each training iteration, synthetic images are generated by a generator, each synthetic image being assigned respective assigned keypoints by the generator. Using a prior-iteration of the keypoint estimation network, a set of predicted keypoints is obtained for each synthetic image. Based on an error score between the predicted keypoints and the assigned keypoints, poor quality synthetic images are discarded. The remaining synthetic images, together with real world images, are used to train an updated keypoint estimation network. The performance of the updated keypoint estimation network is validated, and the training iterations are performed until a convergence criteria is satisfied.