Video Semantic Segmentation via Bidirectional Feature Propagation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing video semantic segmentation methods face challenges in achieving high segmentation accuracy and temporal coherence while incurring high computational costs, particularly due to inaccuracies in optical flow estimation.

Innovation Solution

The proposed method employs bidirectional feature propagation and attention-based feature correction using learned occlusion maps to refine semantic segmentation in video frames, leveraging key frames in both forward and backward directions to improve segmentation accuracy and temporal consistency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Stability of the object's composition

If optical flow estimation is used for feature propagation in video semantic segmentation, then temporal coherence is improved, but segmentation accuracy deteriorates due to inaccuracies in optical flow estimation

Engineering Contradiction:

Improvetemporal coherenceVSAvoidsegmentation accuracy

Core Design Contradiction:

Stability of the object's compositionVSMeasurement precision

Solution Approach 1:

The patent introduces an occlusion map as an intermediary element that mediates between the optical flow estimation and the final segmentation result. The occlusion map identifies regions where optical flow estimation is unreliable and applies attention-based weighting to correct distortions in propagated features, thereby resolving the contradiction between temporal coherence and segmentation accuracy

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements a feedback mechanism where the occlusion detection module continuously monitors the quality of optical flow estimation and adjusts the propagation process accordingly. By computing occlusion maps from optical flows and using them to correct propagated features, the system creates a closed-loop control that improves segmentation accuracy while maintaining temporal coherence

Inventive Principle:

Principle #23Feedback

2Measurement precision

If per-frame image semantic segmentation is used, then segmentation accuracy is maintained, but computational cost increases and temporal coherence is not adequately handled

Engineering Contradiction:

Improvesegmentation accuracyVSAvoidcomputational cost

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent performs preliminary feature extraction and propagation on key frames only, then uses occlusion-aware attention mechanisms to correct and propagate these features to intermediate frames. This preliminary action on selected key frames reduces computational cost while maintaining segmentation accuracy through the corrective feedback mechanism

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the video processing task into key frame processing and intermediate frame processing. By applying full semantic segmentation only to key frames and using corrected feature propagation for intermediate frames, the system reduces overall computational cost while maintaining accuracy through the occlusion map guidance

Inventive Principle:

Principle #1Segmentation

3Productivity

If feature propagation is performed from key frames, then computational efficiency is improved, but segmentation accuracy deteriorates due to distortions from inaccurate optical flow

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidsegmentation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The occlusion map serves as an intermediary that corrects the distortions introduced during key frame to intermediate frame propagation. By identifying occluded regions and applying attention-based weighting, the system maintains computational efficiency while correcting accuracy degradation

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent converts the harmful effect of optical flow inaccuracies into a beneficial correction process. By using the occlusion map to identify where distortions occur, the system applies targeted corrections that improve segmentation accuracy in precisely those regions where key frame propagation caused problems

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Data Source

PatentUS12340586B2Video semantic segmentation with bidirectional feature propagation and attention-based feature correction

Publication Date: 2025.06.24 SAMSUNG ELECTRONICS CO LTD

AI summary

A system and a method that provide video semantic segmentation are disclosed herein. A first frame of a sequence of video frames is semantically segmented to obtain at least one first semantic feature of the first frame. A second frame of the sequence is semantically segmented to obtain at least one second semantic feature of the second frame in which the second frame is subsequent to the first frame. A third frame of the sequence is semantically segmented to obtain at least one third semantic feature in which the third frame is subsequent to the second frame and also being subsequent to the first frame by a first predetermined number of consecutive frames. The at least one first semantic feature, the at least one second semantic feature and the at least one third semantic feature are combined to form at least one fourth semantic feature for the second frame.