Post-Training Generative Model for Spatial Image Structure Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning systems, particularly those involved in safety-critical tasks like autonomous driving, face challenges in generating diverse training data, especially when real image data for rare situations are not available.
Innovation Solution
The method involves post-training a generative probabilistic text-to-image diffusion model by duplicating its blocks into locked and trainable copies. Additional parameters are added to the trainable copy, and adjusted using a distribution-based evolutionary algorithm to improve the generation of images that match specified spatial image structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real image data are used for training, then the machine learning system can learn from authentic examples, but sufficient training data for rare safety-critical situations are not available
Solution Approach 1:
The patent uses a generative machine learning system to create synthetic image copies that mimic real images. These synthetic images serve as training data for rare safety-critical situations where real images are unavailable, thereby increasing the quantity of training data without requiring additional real-world captures
Solution Approach 2:
The generative model is pre-trained on available real images before being used to generate synthetic training data. This preliminary training enables the model to learn authentic image characteristics and subsequently generate realistic synthetic images for rare scenarios in advance of actual deployment
2Manufacturing precision
If a ControlNet is used to control image structure, then spatial image structure can be precisely controlled, but the complexity of the machine learning system increases
Solution Approach 1:
The machine learning system is segmented into distinct functional components: a base generative model for image synthesis and a ControlNet module for spatial structure control. This segmentation allows each component to specialize in specific tasks, achieving precise spatial control without requiring complete redesign of the entire system
Solution Approach 2:
The ControlNet acts as an intermediary component between the input conditions and the base generative model. It processes spatial structure specifications and translates them into controlled modifications of the generated images, thereby achieving precise spatial control while maintaining the integrity of the original generative model
3Adaptability or versatility
If all parameters are adjusted during post-training, then the model can adapt to new tasks, but the training time and computational resources increase significantly
Solution Approach 1:
The patent extracts and freezes the parameters of the base generative model that were learned during pre-training. Only the parameters of the ControlNet module are adjusted during post-training, thereby reducing the total number of trainable parameters and significantly decreasing training time while maintaining adaptability to new spatial structure requirements
Solution Approach 2:
Different parts of the model are assigned different training regimes: the base model parameters are frozen (local quality of stability) while the ControlNet parameters are trainable (local quality of adaptability). This local differentiation allows the system to adapt to new tasks efficiently without requiring extensive retraining of the entire model
4Adaptability or versatility
If synthetic images are generated to expand training data, then data diversity increases, but the images may not accurately represent real-world scenarios
Solution Approach 1:
The system incorporates feedback mechanisms where the ControlNet module receives spatial structure specifications as conditional inputs and adjusts the generated images accordingly. This feedback loop ensures that synthetic images maintain accurate spatial structure representations by continuously comparing generated output against target spatial configurations
Solution Approach 2:
The patent changes key parameters of the generative model during post-training, specifically adding and training the ControlNet module with spatial structure control parameters. This parameter modification enables the model to generate synthetic images with precisely controlled spatial structures while maintaining the diversity benefits of synthetic data generation
Data Source
AI summary
A computer-implemented method for post-training a machine learning system. The machine learning system is configured to generate digital images under the specification of a spatial image structure. The method includes: receiving training data, which each contain specifications of a spatial image structure, by means of the machine learning system; adding further parameters to the machine learning system; generating an image for each training datum by means of the machine learning system; ascertaining the spatial image structure of an image generated by the machine learning system, by means of a second machine learning system; and adjusting the further parameters by using a loss function, which measures a similarity between the ascertained spatial image structure of a generated image and the spatial image structure specified by the associated training datum.