On-Device Image Generation Using Neural Radiance Fields
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image processing techniques are limited in generating new images with desirable characteristics, as they can only select from prestored images and consume significant memory, making them unsuitable for real-time operation on constrained-resource devices.
Innovation Solution
A method that computes scores and estimates camera poses for a set of images to determine a new camera pose, allowing the generation of a new image with improved characteristics using a lightweight, on-device image assessment network and Neural Radiance Fields, efficiently traversing scene space to find optimal image scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional image processing techniques are used to select images from prestored sets, then image selection can be performed automatically, but memory consumption is high and real-time operation on constrained devices is not feasible
Solution Approach 1:
The patent extracts only the essential components needed for image assessment (scoring function and camera pose estimation) from the complex prestored image models, enabling the system to operate with minimal memory on constrained devices while maintaining automatic image selection capability
Solution Approach 2:
The patent replaces the mechanical system of storing and searching through large prestored image databases with a computational approach using neural radiance fields and scoring functions, which can generate and evaluate images on-demand with much lower memory requirements
2Measurement precision
If large models are used to select images from prestored sets, then image selection accuracy can be maintained, but the system cannot run in real-time on constrained-resource devices
Solution Approach 1:
The patent uses lightweight, temporary computational representations (scoring functions and pose estimates) instead of large persistent models, enabling accurate image assessment on constrained devices by creating and discarding simple computational objects rather than loading heavy models
Solution Approach 2:
The patent changes the parameters of the image assessment system by using differentiable scoring functions and continuous pose representations instead of discrete model-based approaches, enabling real-time operation on constrained devices while maintaining assessment accuracy
3Loss of time
If prestored images are used for image suggestion, then processing time is reduced, but the system cannot generate new images with desirable characteristics when no suitable prestored image exists
Solution Approach 1:
The patent performs preliminary actions by pre-training the neural radiance field with input images and their camera poses, so that when image suggestion is needed, the system can quickly generate and evaluate new images using the pre-processed scene understanding without time-consuming processing
Data Source
AI summary
According to an embodiment of the disclosure, a method performed by an apparatus may include obtaining a plurality of images, each of the plurality of images comprising a view of a scene. The method may include computing respective scores for each of the plurality of images. The method may include estimating respective camera poses of each of the plurality of images. The method may include using the computed scores and the estimated camera poses to determine a new camera pose useable for generating a new image comprising a view of the scene and having a score greater than a first threshold score. The method may include generating the new image using the new camera pose.