Latent Space Smoothing Using Optical Flow for Stable AR Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models in augmented reality applications suffer from high latency, computational inefficiencies, and visual artifacts like flickering and jittering due to noise in the latent space, which degrade the user experience and resource utilization, especially on mobile devices.
Innovation Solution
The disclosed techniques smooth the latent space of neural network encoders/decoders using optical flow information from consecutive video frames, reducing noise and improving the realism of AR elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional machine learning models are used in AR applications, then the system can process images and videos, but the output contains noise and visual artifacts like flickering and jittering
Solution Approach 1:
The patent introduces optical flow as an intermediary mechanism between consecutive video frames. By computing optical flow vectors that describe pixel movement between frames and using these to guide latent space smoothing, the system eliminates noise and visual artifacts without requiring changes to the core ML model architecture. The optical flow acts as a mediator that transfers motion information to smooth the latent space representations.
Solution Approach 2:
The patent implements a feedback mechanism where optical flow information from consecutive frames is continuously fed back into the latent space smoothing process. The system uses the optical flow computed from frame t-1 and t to smooth the latent space at time t, creating a feedback loop that progressively reduces noise and visual artifacts while maintaining temporal consistency in the AR output.
2Productivity
If machine learning models process video frames in real-time, then AR applications can run on mobile devices, but computational resources are consumed and latency increases
Solution Approach 1:
The patent segments the computational task by separating the heavy ML model processing from the optical flow computation. Instead of processing every pixel and feature in the original image through the complete ML pipeline, the system computes optical flow at a lower computational cost and then applies smoothing only to the latent space representations, reducing overall computational burden while maintaining real-time performance.
Solution Approach 2:
The patent transitions from processing data in the original image dimension to processing in the latent space dimension. By computing optical flow in the latent space rather than manipulating original images, and by smoothing operations that operate on compressed latent representations, the system reduces computational complexity while preserving the essential information needed for AR rendering.
3Reliability
If the latent space is smoothed using optical flow information, then visual artifacts are reduced, but additional processing steps are required
Solution Approach 1:
The patent merges the optical flow computation with the existing ML model processing pipeline by integrating the smoothing operation into the forward pass. Rather than adding a separate post-processing stage, the optical flow-based smoothing is combined with the latent space transformation, allowing both operations to execute together in a unified computational graph that reduces overall system complexity.
Data Source
AI summary
Methods and systems are disclosed for using machine learning models to perform smoothing in latent space using optical flow information. The methods and systems access a first frame of a video depicting an object and a second frame of the video, the second frame corresponding to a later time period in the video than the first frame. The methods and systems generate optical flow information based on the first frame and the second frame, the optical flow information describing movement of the object from the first frame to the second frame. The methods and systems smooth a latent space generated by one or more neural network encoders of a machine learning model using the optical flow information and process the smoothed latent space by one or more neural network decoders to generate a result of the machine learning model.