Text-and-Sketch Image Generation for Spatial Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-to-image generation technologies face limitations in describing complex shapes, positions, or postures of objects, making it difficult to generate ideal images based on user descriptions, especially for non-experts.
Innovation Solution
An image generation method that utilizes a fusion vector derived from both text and sketch input, denoising a noise image to produce a target image that matches the user's intent, using models like diffusion models and attention mechanisms to integrate spatial and textual information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If text-to-image generation technology is used, then image generation can be achieved based on user descriptions, but the ability to accurately represent complex shapes, positions, or postures is insufficient
Solution Approach 1:
The patent combines text descriptions with sketch images as dual inputs for image generation. The text encoder processes natural language descriptions while the image encoder processes sketch images, and their features are fused through a cross-attention mechanism to create comprehensive guidance for the diffusion model, preserving both semantic and spatial information.
Solution Approach 2:
The patent introduces a sketch image as an intermediary representation that bridges the gap between text descriptions and final image generation. The sketch serves as a spatial scaffold that guides the diffusion process, helping to maintain accurate shapes, positions, and postures while the text provides semantic context.
2Manufacturing precision
If only text descriptions are used for image generation, then the system is simple to operate, but the ability to meet user needs for complex spatial arrangements is limited
Solution Approach 1:
The patent segments the input system into two independent but complementary components: text input for semantic description and sketch input for spatial structure. This segmentation allows users to provide information in their strength (text for non-experts, sketches for spatial relationships) while the system integrates both through feature fusion and cross-attention mechanisms.
3Ease of operation
If text descriptions are used for image generation, then the input process is simple, but the description of complex shapes and postures becomes difficult
Solution Approach 1:
The patent adds a visual dimension (sketch images) to complement the textual dimension of input. Instead of relying solely on language descriptions, users can provide simple sketches that capture spatial relationships, shapes, and postures visually. The system processes both dimensions through dual encoders and fuses their features, allowing users to leverage their natural strengths in both modalities.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The present disclosure provides an image generation method and an apparatus, and an electronic device. The image generation method comprises: acquiring a target text, a sketch image, and a noise image; obtaining a fusion vector on the basis of the target text and the sketch image; and performing denoising processing on the noise image on the basis of the fusion vector, to obtain a target image matching the target text and the sketch image. According to embodiments of the present disclosure, the image generation effect is improved, and the objective of generating an image according to the willingness of a user is achieved.