Deep Equilibrium Optical Flow Estimation With Lower Memory Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing optical flow estimation methods in computer vision, particularly those based on recurrent neural networks, suffer from high computational and memory overheads, poor convergence, and instability, despite achieving superior performance.
Innovation Solution
The implementation of a deep equilibrium (DEQ) neural network model that directly solves for a fixed-point representation of optical flow, using a black-box solver and inexact gradients, allowing for efficient training and inference with reduced memory and computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If recurrent neural networks are used for optical flow estimation, then performance is improved, but computational overhead and memory requirements increase
Solution Approach 1:
The patent replaces the traditional recurrent neural network mechanical structure with a deep equilibrium model that uses implicit differentiation and fixed-point iteration. This substitution eliminates the need for sequential recurrent computations while maintaining estimation accuracy, thereby reducing computational overhead and memory requirements.
Solution Approach 2:
The patent changes the fundamental parameters of the neural network model by transitioning from explicit recurrent updates to implicit equilibrium solutions. By using inexact gradients and fixed-point iteration, the model achieves the same performance with different computational characteristics that reduce overhead.
2Measurement precision
If recurrent neural networks are used for optical flow estimation, then performance is improved, but memory overhead increases
Solution Approach 1:
The patent substitutes the memory-intensive recurrent computation mechanism with a deep equilibrium model that computes solutions iteratively without storing intermediate recurrent states. This replacement dramatically reduces memory requirements while preserving estimation accuracy through fixed-point convergence.
Solution Approach 2:
The patent extracts and removes the problematic recurrent state storage requirement from the model architecture. By using implicit differentiation, the model computes gradients without needing to store the entire sequence of intermediate activations, thereby extracting the memory burden from the system.
3Measurement precision
If recurrent neural networks are used for optical flow estimation, then performance is improved, but convergence and stability deteriorate
Solution Approach 1:
The patent replaces the unstable recurrent update mechanism with a deep equilibrium model based on fixed-point iteration and implicit differentiation. This substitution provides inherent stability through the mathematical properties of equilibrium solutions and enables reliable convergence even with inexact gradients.
Data Source
AI summary
A computer-implemented method for a machine learning (ML) system includes receiving a first image frame and a second frame from a sensor, wherein the first and second image frames are time series data, determining a first flow state and a first latent state of the first image frame, determining a Deep Equilibrium Model (DEQ) based fix point solution via a root finding method based on the first flow state, the first latent state, and a layer function to obtain an estimated flow and latent state, receiving a third image frame, wherein the second and third image frames are time series data, determining the fix point solution via the root finding method based on the estimated flow state, the estimated latent state, and layer function to obtain an updated flow state and updated latent state, and outputting the updated flow state.