CRNN Training With Optical Flow State Warping for Video Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for semantic video segmentation struggle with temporal inconsistency and flickering effects due to the lack of effective integration of temporal information, leading to unsatisfactory performance in real-world applications like autonomous driving and augmented reality.
Innovation Solution
A method for training a Convolutional Recurrent Neural Network (CRNN) that incorporates optical flow-based warping of internal states between frames to adapt to pixel motion, using a recurrent module such as FlowingGRU or FlowingLSTM, enhancing temporal consistency in semantic segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If image-level semantic segmentation is performed independently on each frame, then processing speed is maintained, but temporal consistency deteriorates causing flickering effects
Solution Approach 1:
The patent applies continuity of useful action by maintaining temporal continuity in semantic segmentation through recurrent neural networks. The RNN processes video frames sequentially, preserving temporal dependencies and ensuring consistent segmentation across frames, thereby eliminating flickering effects while maintaining reasonable processing speed through efficient recurrent computation.
Solution Approach 2:
The patent implements dynamics by adapting the segmentation model to dynamic video content. The recurrent module dynamically adjusts segmentation based on temporal context from previous frames, allowing the system to handle moving objects and changing scenes while maintaining temporal consistency, thus resolving the contradiction between speed and reliability.
2Reliability
If temporal information is integrated using recurrent modules, then temporal consistency is improved, but computational complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the computational task into distinct modular components: a base segmentation network for spatial feature extraction and a recurrent module for temporal integration. This modular segmentation allows efficient computation at each stage while achieving overall temporal consistency, reducing the burden of computational complexity.
Solution Approach 2:
The patent implements partial action by applying recurrent temporal modeling selectively at key processing stages rather than throughout the entire network. The base segmentation network processes frames efficiently, and the recurrent module adds temporal consistency only where needed, avoiding excessive computation while maintaining reliability.
3Measurement precision
If optical flow warping is applied to adapt internal states to pixel motion, then segmentation accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-computing optical flow fields and preparing warped feature representations in advance within the recurrent module. This preliminary preparation of motion-compensated features enables accurate segmentation without adding significant processing time during inference, as the computationally intensive warping operations are efficiently integrated into the recurrent processing pipeline.
Data Source
Figure 1a
Figure 1b
Figure 1c
AI summary
The present invention relates to a method for training a convolutional recurrent neural network, CRNN, for semantic segmentation in videos; the method being characterized in that it comprises the implementation, by a data processor (11a) of a first server (1a), of steps of: (a) Training from a base of training images already semantically segmented, a first convolutional neural network, CNN; (b) Training from a base of training videos already semantically segmented, a recurrent convolutional neural network, CRNN, corresponding to the first CNN wherein a convolutional layer has been replaced by a recurrent module having a hidden state; said training comprising, for each pair of successive frames (t - 1, t ∈ 1; T 2) of a video of said base of training videos already semantically segmented: (b1) Warping the internal state of the recurrent layer according to an estimated optical flow between the frames of the pair, so as to adapt the internal state to the motion of pixels between the frames of the pair; (b2) learning parameters of at least the recurrent module A method for semantic segmentation of an inputted video is further proposed.