Graphical Style Adapters for Consistent Sketch-to-Image Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI service technologies are large in size, require significant memory and processing power, pose privacy concerns due to transmission of private information, struggle with interpreting hand-drawn sketches, and face challenges in handling inputs in different styles and quality while delivering inconsistent outputs.
Innovation Solution
Implementing a local generative AI service on a computing device using graphical style adapters to optimize model size, preprocess inputs for consistency, and utilize adapters to enhance the generative AI service's versatility, along with applications that provide intuitive interfaces and system prompts to improve user input extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If generative AI service technologies are deployed locally on resource-constrained devices, then privacy is enhanced and accessibility is improved, but model size and computational requirements become excessive for local deployment
Solution Approach 1:
The patent segments the generative AI system into two parts: a compact local model that handles private data processing, and a larger remote model that provides comprehensive generative capabilities. The local model is trained on distilled knowledge from the remote model, enabling it to perform key functions with minimal model size while maintaining privacy benefits of local deployment.
2Weight of stationary object
If the generative AI service is made compact for local deployment, then resource requirements are reduced, but quality and versatility of image generation deteriorate
Solution Approach 1:
The system performs preliminary action by training the local model in advance using knowledge distillation from a large remote model. This pre-training process transfers essential generative knowledge to the compact local model, enabling it to deliver acceptable image generation quality despite its small size, thus resolving the trade-off between model compactness and generation quality.
3Adaptability or versatility
If generative AI services process inputs in different styles directly, then versatility is improved, but output consistency deteriorates
Solution Approach 1:
The patent introduces a style transfer module as an intermediary between the diverse input images and the generative AI model. This module standardizes different input styles (sketches, photos, paintings) into a unified representation that the local model can process consistently, thereby maintaining output style consistency while preserving versatility in handling various input types.
Data Source
AI summary
The present technology attempts to provide a generative AI service to run locally on a computing device where the generative AI service can receive a rough sketch input as a prompt and generate a higher-quality output. The present technology utilizes a common generative AI service for a variety of use cases and supplements the common generative AI service with a variety of graphical style adapters. The graphical style adapters are also configured to receive sketches as inputs and condition them for use by the generative AI service. Some conditioning of sketches can include determining a sketch complexity metric and taking steps to acknowledge that sketches might be an outline of any object without much fill coloring but that the outline might not reflect the intention of the user that a sketched object is to be created with or without fill and texture.