Scene-Based Transformer for Controllable Text-to-Image Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional text-to-image generation methods lack controllability, human perception alignment, and resolution limitations, resulting in random output and suboptimal image quality.

Innovation Solution

A novel method using a scene-based transformer with implicit conditioning and explicit losses targeting human-preferred regions, combined with classifier-free guidance, to enhance controllability and quality, enabling scene editing and storytelling.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional text-to-image generation methods are used, then the generation process is simple, but the output lacks controllability and structural consistency

Engineering Contradiction:

Improvegeneration process simplicityVSAvoidoutput controllability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent segments the image generation process into multiple controllable components: text encoding, scene graph generation, and image synthesis. By introducing scene graphs as an intermediate representation that decomposes images into objects, relationships, and attributes, the system achieves fine-grained controllability over generated images while maintaining a relatively simple overall workflow.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces scene graphs as an intermediary between text prompts and final images. This intermediate representation layer enables precise control by allowing users to specify objects, their attributes, and spatial relationships in a structured format, which then guides the image generation process to produce structurally consistent and controllable outputs.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If conventional text-to-image models are used, then the model structure is simple, but the resolution and image fidelity are limited

Engineering Contradiction:

Improvemodel structure complexityVSAvoidimage fidelity

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The patent employs a dynamic and modular model architecture that can adapt to different resolution requirements. The scene graph-based approach allows the system to generate images at varying resolutions by controlling the level of detail in object representations and relationships, enabling high-fidelity output without requiring a completely different model structure for each resolution level.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent adds a new dimensional layer to the generation process by introducing scene graphs as an intermediate representation. This additional dimension allows the model to capture fine-grained semantic information and spatial relationships, which translates to improved image fidelity and detail without simply increasing model size or complexity in the traditional sense.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Device complexity

If conventional text-to-image generation is used, then the input processing is simple, but the alignment with human perception is suboptimal

Engineering Contradiction:

Improveinput processing complexityVSAvoidhuman perception alignment

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent applies local quality by focusing computational resources on semantically important regions identified through scene graph analysis. By decomposing images into objects and relationships, the system can prioritize the accurate generation of key elements that are most important to human perception, rather than treating all regions uniformly.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent performs preliminary analysis of text prompts to generate scene graphs before actual image synthesis. This preliminary action allows the system to pre-identify important objects, attributes, and spatial relationships, enabling better alignment with human perception during the subsequent image generation phase by focusing on semantically critical elements.

Inventive Principle:

Principle #10Preliminary action

4Quantity of substance

If conventional text-to-image models are used, then the training data requirements are standard, but the ability to handle out-of-distribution prompts is limited

Engineering Contradiction:

Improvetraining data quantityVSAvoidout-of-distribution prompt handling

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent creates a universal scene graph representation that can accommodate diverse and out-of-distribution text prompts. By using a flexible scene graph structure that can represent various objects, relationships, and attributes in a unified framework, the system can handle prompts from different domains and distributions without requiring domain-specific training data, enhancing its adaptability and versatility.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12387388B2Scene-based text-to-image generation with human priors

Publication Date: 2025.08.12 META PLATFORMS INC

AI summary

In one embodiment, a method includes accessing a text input and a scene input corresponding to the text input, wherein the scene input comprises semantic segmentations, generating text tokens for the text input and scene tokens for the scene input by machine-learning models, generating predicted image tokens based on the text tokens and the scene tokens by the machine-learning models, and generating an image corresponding to the text input and the scene input based on the predicted image tokens by the machine-learning models.