Scene Text Image Generation with Character-Level Prompt Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional generative models often fail to generate images with legible scene text due to the use of byte-pair encoding (BPE), which obfuscates the meaning of individual characters, leading to misshaped or misspelled text in generated images.
Innovation Solution
Utilizing a character-level encoder in conjunction with a prompt encoder to encode scene text at the character level, combining these encodings to condition image generation, thereby enhancing the generation of images with legible scene text while maintaining reduced compute power and model size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If byte-pair encoding (BPE) is used to encode text prompts, then the model can process text efficiently, but the individual character meanings are obfuscated leading to illegible scene text in generated images
Solution Approach 1:
The patent segments the text encoding process into two distinct parts: (1) BPE encoding for the overall prompt to maintain processing efficiency, and (2) character-level encoding specifically for scene text to preserve character legibility. This segmentation allows each encoding method to serve its optimal function without compromising the other.
Solution Approach 2:
The patent applies different encoding qualities to different parts of the text input. Standard BPE encoding is applied to general prompt text where semantic meaning is sufficient, while character-level encoding is applied specifically to scene text portions where precise character reproduction is critical for legibility.
2Manufacturing precision
If character-level encoding is used for scene text, then legible text can be generated, but computational resources and model size increase
Solution Approach 1:
The patent applies computationally intensive character-level encoding only to the specific scene text portions of the prompt rather than the entire prompt. This localized application maintains text legibility while minimizing the additional computational overhead by limiting it to only where necessary.
Solution Approach 2:
The patent segments the text processing pipeline to identify and separately encode only the scene text portions using character-level encoding, while processing the rest of the prompt with efficient BPE encoding. This reduces overall computational resource consumption compared to applying character-level encoding to the entire prompt.
3Manufacturing precision
If character-level encoding is used for scene text, then the model size increases, but accurate character recreation is achieved
Solution Approach 1:
The patent segments the encoding architecture into two parallel pathways: a BPE encoding pathway for general prompt processing and a character-level encoding pathway for scene text. This segmentation allows the model to maintain smaller overall size by using efficient BPE for most text while only invoking the more complex character-level encoding where absolutely necessary for scene text reproduction.
Solution Approach 2:
The patent applies high-fidelity character-level encoding only to scene text portions where precise character reproduction is critical, while using more compact BPE representations for the rest of the prompt. This local application of high-quality encoding achieves accurate character recreation while minimizing the increase in overall model size.
Data Source
AI summary
Systems and methods for generating images with legible scene text are described. Embodiments are configured to obtain a prompt describing a scene, where the prompt includes scene text indicating text that is intended to be shown in a generated image; encode, using a prompt encoder, the prompt to generate a prompt embedding; encode, using a character-level encoder, the scene text to generate a character-level embedding; and generate, using an image generation network, an image that includes the scene text based on the prompt embedding and the character-level embedding.