Predefined-Layout Diffusion Image Generation for Data Augmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of training machine learning systems for generating images from predefined layouts is exacerbated by the scarcity of large-scale data, particularly in safety-critical applications like autonomous driving and automated inspection, where data of defective samples is hard to acquire.
Innovation Solution
A diffusion probabilistic model with an encoder and decoder structure is trained using a U-net architecture, incorporating layout-aware affine transformations and mask attention layers to generate synthetic images from predefined bounding boxes and labels, ensuring realistic and controlled image generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real data is collected from safety-critical applications like autonomous driving or automated inspection, then the data quality and reliability are improved, but the quantity of available data deteriorates due to scarcity of defective samples
Solution Approach 1:
The patent uses a diffusion probabilistic model to generate synthetic images that copy the statistical properties and visual characteristics of real images. The model learns from a small set of real images and generates additional synthetic samples that augment the training data, effectively creating copies of real data patterns without requiring additional physical samples.
Solution Approach 2:
The patent applies parameter changes by modifying the latent space representations during the diffusion process. By adjusting parameters such as noise schedules, guidance scales, and transformation parameters, the system generates diverse synthetic images while maintaining consistency with the underlying data distribution, thus increasing data quantity without compromising reliability.
2Quantity of substance
If synthetic images are generated to augment training data, then the data quantity is improved, but the manufacturing complexity of the generation system deteriorates
Solution Approach 1:
The patent segments the image generation process into distinct modular components: an encoder that extracts features from input images, a diffusion model that performs probabilistic transformation, and a decoder that reconstructs images. This segmentation allows each component to be independently trained and optimized, reducing overall system complexity while maintaining generation capability.
Solution Approach 2:
The patent introduces latent space representations as an intermediary between the encoder and decoder. Instead of directly transforming input images to output images, the system operates in a compressed latent space, which simplifies the transformation process and reduces computational complexity while enabling efficient generation of diverse synthetic images.
3Ease of operation
If layout information is used to guide image generation, then the control over generated image structure is improved, but the difficulty of detecting and measuring layout parameters deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-processing layout information into standardized representations before feeding it to the diffusion model. Bounding boxes and semantic labels are prepared in advance and encoded into a format that the model can easily process, making the generation process more controllable while simplifying the detection and measurement of layout parameters.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method of training a machine learning system for generating images from a predefined layout. The machine learning system is a diffusion probabilistic model with an encoder part and a decoder part, wherein a normalization layer in a residual block of the decoder part comprises weighted layout-aware affine transformation parameters γ and β, determined from layout-aware affine transformation parameters γ' and β' by multiplication with a weighted semantic map, wherein the weighted semantic map comprises a sum of two contributions, wherein the first contribution comprises a non-overlapping semantic map computed from determined and size-ranked object probabilistic masks and wherein the second contribution comprises an edge-aware semantic map, determined from extended object probabilistic masks, wherein an extended object probabilistic mask is extended with respect to the corresponding object probabilistic mask by one pixel along the borders.