GPU Image Filter Pipeline Using Low-Resolution Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning algorithms for image filtering and enhancement, particularly in high dynamic range (HDR) applications, are computationally intensive and slow, leading to output delays on mobile devices, which frustrates users by hindering real-time image processing.
Innovation Solution
An integrated pipeline on a graphics processing unit (GPU) converges computer graphics rendering with machine learning, utilizing a convolutional neural network (CNN) trained via a generative adversarial network (GAN) with a self-regularized attention mechanism, allowing for real-time image enhancement operations by performing most learning and inferencing on low-resolution images and upsampling to high resolution, thus reducing processing time and output delay.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If machine learning algorithms are used for image filtering and enhancement, then image quality is improved, but processing time increases and output delay occurs
Solution Approach 1:
The patent segments the image processing task by separating training and inferencing operations. Training is performed offline on a server using bracketed images, while inferencing is executed in real-time on the mobile device GPU during camera operation. This segmentation allows complex machine learning processing to occur without blocking real-time image capture and display.
Solution Approach 2:
The patent applies preliminary action by pre-training the neural network model offline using a dataset of bracketed images before deployment. The model is trained in advance to learn the mapping between differently exposed images, so that during real-time operation, only the lightweight inferencing step is required, enabling real-time performance without sacrificing image quality enhancement capabilities.
2Manufacturing precision
If machine learning algorithms are used for image filtering and enhancement, then image quality is improved, but computational resources are consumed
Solution Approach 1:
The patent segments computational workload between server and mobile device. The computationally intensive training phase is performed on a server with abundant resources, while the mobile device only performs lightweight inferencing during camera operation. This segmentation allows high-quality image enhancement while conserving mobile device computational resources and energy.
Solution Approach 2:
The patent uses a pre-trained neural network model that is copied from the training environment to the mobile device. Instead of performing full training on the resource-constrained mobile device, the trained model weights and architecture are copied and deployed for efficient real-time inferencing, significantly reducing computational resource consumption on the mobile device while maintaining image quality improvement.
Data Source
AI summary
Various aspects of the subject technology relate to systems, methods, and machine-readable media for outputting filtered visual media content items. Various aspects may include receiving an input frame of a visual media content item. Aspects may also include training a machine learning algorithm based on a dataset of bracketed images. Aspects may include configuring a neural network based on image filtering of the input frame and via a shader component of a graphics processing unit. Aspects may include determining portions of the input frame that are associated with an extent of darkness. Aspects may include performing an image enhancement operation to the portions of the input frame. Aspects may include providing instructions to display an output frame changed by the image enhancement operation.