Reduced-Precision Generative Graphics Models Using Saliency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative models for image, video, and 3D content generation using machine learning techniques require high precision formats like fp32, bfloat16, and int8, which can be computationally expensive and inefficient, especially for human-perceivable content.
Innovation Solution
Implementing a hybrid approach using 1-bit quantized networks for imperceptible regions and higher precision when needed, leveraging saliency and attention models with training losses to optimize performance, merging low and high-bit generation pipelines for improved computational efficiency and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision formats (fp32, bfloat16, int8) are used for generative models, then image quality and model accuracy are improved, but computational cost and processing time increase
Solution Approach 1:
The patent applies different precision levels to different regions of the generated content based on human visual perception characteristics. High precision is used only for visually critical regions, while lower precision is used for imperceptible regions, thereby maintaining image quality where needed while improving overall computational efficiency
Solution Approach 2:
The patent dynamically adjusts precision parameters during the generative process, transitioning between different precision levels (fp32, bfloat16, int8, and lower) based on the specific computational stage and region being processed, optimizing the balance between quality and efficiency
2Reliability
If high precision formats are used for generative models, then model accuracy is improved, but memory usage and bandwidth requirements increase
Solution Approach 1:
The patent implements precision allocation based on visual importance, storing and processing high precision data only for regions that contribute significantly to perceived image quality, while using compressed lower precision representations for other regions, thereby reducing overall memory consumption
Solution Approach 2:
Instead of uniformly using high precision throughout the model, the patent inverts the approach by defaulting to lower precision and selectively applying high precision only where necessary for maintaining model accuracy, thus reducing memory requirements
3Productivity
If 1-bit quantized networks are used for imperceptible regions, then computational efficiency is improved, but image quality may deteriorate
Solution Approach 1:
The patent strategically assigns 1-bit quantization specifically to imperceptible regions identified through saliency and attention models, while preserving higher precision in visually critical regions, thus achieving speed improvements without perceptible quality loss
Solution Approach 2:
The patent introduces saliency models and attention mechanisms as intermediary components that identify which regions require high precision and which can tolerate quantization, mediating between the conflicting requirements of speed and quality
Data Source
Figure 1
Figure 2A
Figure 2B~2C
AI summary
Systems and methods are for reduced precision models for generative graphics are provided. In one example, information is received that is indicative of saliency of one or more portions of an image that is to be generated. The image is then generated by, for each portion of the one or more portions of the image: (i) based on the information indicative of saliency, selecting a generative model from among multiple pre-trained generative models (e.g., diffusion models) quantized at different precision levels (e.g., using various weights of between 32-bits and 1-bit, inclusive); and (ii) applying the selected generative model to pixels associated with the portion.