HDR Video Tone Mapping Using Spatiotemporal Optical Flow Filtering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current tone mapping techniques for high dynamic range (HDR) video fail to effectively address temporal artifacts and halo effects, leading to undesirable results when reproducing HDR moving images on consumer-level displays.

Innovation Solution

The method involves applying spatiotemporal filters based on forward and backward optical flows to separate HDR images into a base layer and a detail layer, followed by tone mapping to reduce dynamic range while preserving local details, using iterative filtering to minimize halo artifacts and enhance temporal coherence.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If tone mapping techniques from still imagery are applied to HDR video, then dynamic range is reduced to accommodate consumer displays, but temporal artifacts are introduced due to lack of temporal coherence

Engineering Contradiction:

Improveadaptability to consumer display capabilitiesVSAvoidtemporal coherence

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent segments the HDR video processing into multiple components: base layer extraction, detail layer extraction, and separate tone mapping processing. This segmentation allows temporal coherence to be maintained in the base layer while preserving spatial details in the detail layer, resolving the contradiction between adapting to display capabilities and maintaining temporal reliability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies dynamic tone mapping that adapts to temporal variations in the video sequence. By using optical flow to track motion and applying temporal coherence constraints, the system dynamically adjusts tone mapping parameters across frames, maintaining both display adaptability and temporal reliability

Inventive Principle:

Principle #15Dynamics

2Productivity

If HDR images are separated into base layer and detail layer for compression, then dynamic range is managed, but halo effects are introduced at boundaries

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidhalo effects

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The patent applies local quality by processing different regions of the image with different characteristics. The base layer captures global illumination and temporal coherence, while the detail layer captures local spatial frequencies. This local differentiation allows efficient compression while minimizing halo effects at boundaries through careful blending

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent uses feedback mechanisms where the detail layer is refined based on the base layer results. By iteratively adjusting the detail layer extraction and blending parameters, the system reduces halo effects while maintaining processing efficiency

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If aggressive tone mapping is applied to reduce dynamic range, then consumer display compatibility is improved, but local contrast and detail are lost

Engineering Contradiction:

Improvedisplay compatibilityVSAvoidlocal contrast preservation

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The patent segments tone mapping into base layer and detail layer processing, applying different tone mapping strategies to each. The base layer uses aggressive tone mapping for display compatibility, while the detail layer preserves local contrast and edges, resolving the contradiction between display adaptation and detail preservation

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates a composite image by combining the tone-mapped base layer with the detail layer. This composite approach allows the aggressive tone mapping of the base layer to improve display compatibility while the detail layer restores local contrast and spatial frequencies

Inventive Principle:

Principle #40Composite materials

Data Source

PatentUS9378543B2Temporally coherent local tone mapping of high dynamic range video

Publication Date: 2016.06.28 DISNEY ENTERPRISES INC

AI summary

Approaches are described for tone-mapping an image frame in a sequence of image frames. A tone-mapping pipeline applies a spatiotemporal filter to each pixel of the image frame, based on a forward optical flow and a backward optical flow, to produce a base layer image. The tone-mapping pipeline applies a temporal filter to each pixel of the image frame, based on the forward and backward optical flows, to produce a temporally filtered frame. The tone-mapping pipeline produces a detail layer image based on the base layer image and the temporally filtered frame. The tone-mapping pipeline produces a detail layer image based on the base layer image and the temporally filtered frame. The tone-mapping pipeline applies a tone curve to the base and detail layer images to produce a tone-mapped base and detail layer image, respectively, and combines the tone-mapped base and detail layer images to produce a tone-mapped image frame.