Light Field Video Super-rays Temporal Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for processing light field videos lack temporal consistency when applying super-rays representation, requiring significant memory and being unsuitable for GPU implementation, especially in cases with large data volumes and angular inconsistencies.

Innovation Solution

A method for processing light field videos by determining a first super-rays representation based on centroids, tracking their displacement between frames using minimization or deep matching techniques, and applying these displacements to maintain temporal consistency without requiring the entire video sequence in memory.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If the super-rays representation is applied to each frame of light field video independently, then the editing capability is improved, but temporal consistency is lost

Engineering Contradiction:

Improveediting capabilityVSAvoidtemporal consistency

Core Design Contradiction:

Ease of operationVSStability of the object's composition

Solution Approach 1:

The patent applies dynamics by making the super-rays representation adaptive across frames. Instead of independently processing each frame, the method dynamically updates super-ray centroids and assignments based on temporal information from previous frames, allowing the representation to evolve consistently with scene changes while maintaining editing capability.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent implements feedback mechanisms where the super-rays representation from one frame serves as input for processing subsequent frames. The centroid positions and ray assignments from previous frames provide feedback that guides the current frame processing, ensuring temporal consistency while preserving the ability to perform editing operations.

Inventive Principle:

Principle #23Feedback

2Stability of the object's composition

If the entire video sequence is loaded into memory for processing, then temporal consistency is improved, but memory requirements become prohibitive

Engineering Contradiction:

Improvetemporal consistencyVSAvoidmemory requirements

Core Design Contradiction:

Stability of the object's compositionVSQuantity of substance

Solution Approach 1:

The patent segments the video processing into manageable units by maintaining super-rays representations for only a limited number of recent frames rather than the entire sequence. This segmentation allows temporal consistency to be maintained through frame-based processing while significantly reducing memory requirements by discarding older frame data that is no longer needed for consistency maintenance.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-computing and storing super-rays representations for reference frames that will be needed for future processing. This allows the system to maintain temporal consistency without needing to load the entire video sequence into memory, as the necessary reference data is prepared in advance and stored efficiently.

Inventive Principle:

Principle #10Preliminary action

3Stability of the object's composition

If super-pixels approach is used with dense flow propagation, then temporal consistency is improved, but angular consistency is lost

Engineering Contradiction:

Improvetemporal consistencyVSAvoidangular consistency

Core Design Contradiction:

Stability of the object's compositionVSManufacturing precision

Solution Approach 1:

The patent applies local quality by maintaining different levels of detail and precision for different aspects of the super-rays representation. The method ensures angular consistency is preserved in the ray direction information while allowing temporal consistency in the centroid positions and assignments, achieving both requirements simultaneously through differentiated processing of different representation components.

Inventive Principle:

Principle #3Local quality

4Device complexity

If the light field video is processed frame-by-frame, then processing simplicity is improved, but data volume handling becomes inefficient

Engineering Contradiction:

Improveprocessing simplicityVSAvoiddata volume handling efficiency

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent merges the processing of multiple frames by utilizing the super-rays representation from previous frames as input for current frame processing. This combining approach allows efficient handling of large data volumes through reusable intermediate representations, improving productivity while maintaining processing simplicity through a unified algorithmic framework.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11195288B2Method for processing a light field video based on the use of a super-rays representation

Publication Date: 2021.12.07 INTERDIGITAL MADISON PATENT HLDG

AI summary

A method and device for processing a light field video is described. The light field video includes a set of image views per unit of time, the light field video being associated with a scene without cuts. In the method a first super-rays representation of reference image views at a given time is determined based on centroids. A second super-rays representation associated with corresponding views of a subsequent set of image views is next determined based on de-projection and re-projection of centroids. The displacement of centroids between the first and second super-rays is determined and then the determined displacement is applied to centroids of the second super-rays representation.