Bilateral Attention Transformer for Accurate Video Object Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing video object segmentation (VOS) methods struggle to accurately distinguish foreground objects from backgrounds, particularly when multiple objects have similar visual appearances and are proximal, and optical flow-based methods suffer from noise and error accumulation.

Innovation Solution

A bilateral attention transformer is employed in a motion-appearance neighboring space to compute local attention based on spatial distance, appearance similarity, and optical flow smoothness, using a bilateral space that considers both motion and appearance information to generate object masks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If optical flow-based methods are used for video object segmentation, then motion information can be captured, but noise and error accumulation occur leading to reduced segmentation accuracy

Engineering Contradiction:

Improvesegmentation accuracyVSAvoidnoise and error accumulation

Core Design Contradiction:

ReliabilityVSObject-generated harmful factors

Solution Approach 1:

The patent converts the harmful noise and error accumulation in optical flow by using bilateral attention to identify and emphasize only the most reliable correspondence relationships. The bilateral attention mechanism selectively weights optical flow information based on appearance similarity and spatial consistency, transforming noisy optical flow data into reliable guidance for segmentation by focusing on high-confidence matches and suppressing erroneous ones.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The patent introduces bilateral attention as an intermediary mechanism between optical flow computation and segmentation generation. This intermediary selectively processes optical flow information through appearance similarity weighting and spatial consistency constraints, acting as a filter that mediates between raw optical flow data and final segmentation results, thereby reducing noise and error accumulation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If deep neural networks are used for video object segmentation, then segmentation can be performed, but computational complexity increases

Engineering Contradiction:

Improvesegmentation accuracyVSAvoidcomputational complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent applies local quality by computing bilateral attention only in local neighborhoods rather than performing global optimization. The bilateral attention mechanism operates locally by considering only neighboring pixels and their corresponding regions in reference frames, weighted by appearance similarity and spatial distance. This local approach maintains segmentation accuracy while significantly reducing computational complexity compared to global deep neural network methods.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent uses partial action by computing bilateral attention selectively only where needed - in regions with high appearance similarity and spatial consistency - rather than processing the entire image uniformly. This partial computation approach reduces overall computational complexity while maintaining segmentation accuracy in critical regions.

Inventive Principle:

Principle #16Partial or excessive action

3Loss of information

If global attention mechanisms are used to establish correspondence between frames, then comprehensive information can be processed, but distinction between target objects and background fails when multiple similar objects are present

Engineering Contradiction:

Improveinformation processing completenessVSAvoidobject-background distinction accuracy

Core Design Contradiction:

Loss of informationVSReliability

Solution Approach 1:

The patent resolves this contradiction by implementing local quality through spatially-aware bilateral attention. Instead of uniform global attention, the mechanism adapts attention weights based on spatial distance and appearance similarity locally. This allows comprehensive information processing in relevant local regions while automatically suppressing attention to irrelevant background areas, enabling accurate object-background distinction even when multiple similar objects are present.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent applies segmentation by dividing the global attention mechanism into local attention units that operate independently on different spatial regions. Each local unit processes information based on its specific spatial context and appearance characteristics, allowing the system to distinguish between multiple similar objects by processing them through separate local attention channels that preserve spatial relationships.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12505560B2Bilateral attention transformer in motion-appearance neighboring space for video object segmentation

Publication Date: 2025.12.23 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Example solutions for video object segmentation (VOS) use a bilateral attention transformer in motion-appearance neighboring space, and perform a process that includes: receiving a video stream comprising a plurality of video frames in a sequence; receiving a first object mask for an initial video frame of the plurality of video frames; selecting a video frame of the plurality of video frames as a current query frame, the current query frame following, in the sequence, a reference frame of a reference frame set, wherein each reference frame has a corresponding object mask; using the current query frame and a video frame in the reference frame set, determining a bilateral attention; and using the bilateral attention, generating an object mask for the current query frame.