Image Key Point Smoothing Using Optical Flow Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for stabilizing key points in consecutive images, such as those used in augmented reality, suffer from poor smoothing effects due to delays or jitters caused by improper weight adjustments in temporal smoothing, which are not suitable for scenarios requiring high stability.
Innovation Solution
A method involving optical flow processing to determine position offset information, compensating predicted position information using this offset, and performing temporal smoothing with adjusted weights to improve stability, reducing delays and jitters in key point stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If temporal smoothing uses large weight for previous frame predicted position information, then key point stability is improved, but key point delay occurs
Solution Approach 1:
The patent performs optical flow processing in advance to calculate position offset information between frames. This pre-calculated offset information is then used to compensate the predicted position information before temporal smoothing, allowing the system to anticipate and correct potential delays rather than reacting to them after they occur.
Solution Approach 2:
The patent dynamically adjusts the weight parameter in temporal smoothing based on the calculated position offset information. When position changes are significant, the weight for previous frame information is reduced to avoid delay. When position changes are minimal, the weight is increased to enhance stability. This adaptive parameter adjustment resolves the contradiction between stability and delay.
2Loss of time
If temporal smoothing uses small weight for previous frame predicted position information, then key point delay is reduced, but key point jitter increases
Solution Approach 1:
The patent dynamically adjusts the weight parameter in temporal smoothing based on the calculated position offset information. When position changes are significant, the weight for previous frame information is reduced to avoid delay. When position changes are minimal, the weight is increased to enhance stability. This adaptive parameter adjustment resolves the contradiction between stability and delay.
3Device complexity
If deep learning model does not consider temporality during training, then model training simplicity is maintained, but key point jitter in consecutive images occurs
Solution Approach 1:
The patent introduces optical flow processing as an intermediary mechanism between frame processing steps. This intermediary calculates position offset information that bridges temporal relationships between frames, allowing the deep learning model to maintain its simple training structure while still achieving temporal stability through the mediating optical flow compensation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method effectively avoids delays and jitters in key points, enhancing the smoothing effect and stability of key points in consecutive images, particularly in augmented reality applications.
Implementation Method 1
performing optical flow processing on the key points according to the first image region and the second image region to determine position offset information of the key points
Data Source
AI summary
A method and apparatus for processing key points in an image are provided. The method includes: obtaining a first image region to which key points belongs from an (N−1)th frame image; obtaining a second image region to which the key points belong from an Nth frame image; analyzing optical flow of the key points between the first image region and the second image region, to determine position offset information of the key points; compensating first predicted position information of the key points in the (N−1)th frame image by using the position offset information to obtain compensated predicted position information of the key points; and performing temporal smoothing on second predicted position information of the key points in the Nth frame image using the compensated predicted position information to obtain smoothed predicted position information of the key points in the Nth frame image.