Video Frame Rate Control Using Depth Maps

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current video processing technologies lack the ability to automatically generate videos with different frame rates for specific entities or regions of interest within a video stream, particularly in devices with limited processing power such as smartphones or tablets, which hinders the creation of enhanced visual effects and user experiences.

Innovation Solution

A system and method that uses a digital camera, interface, and processor to mark entities in a video stream, determine frame rate ratios, and generate output streams where entities of interest or regions of interest are played at different frame rates, utilizing depth maps for spatial and temporal alignment and occlusion handling.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If automated entity-based frame rate modification is implemented, then visual effects and user experience are improved, but processing power requirements increase

Engineering Contradiction:

Improvevisual effectsVSAvoidprocessing power

Core Design Contradiction:

Adaptability or versatilityVSPower

Solution Approach 1:

The video frame is segmented into multiple regions based on depth information, with each region assigned a different frame rate. Entities are identified and isolated as separate segments, allowing independent frame rate control for each entity while processing only relevant portions at higher rates, thus reducing overall processing requirements.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different frame rates are applied to different spatial regions of the video based on entity importance and depth. High frame rates are applied locally to specific entities of interest, while other regions maintain standard frame rates, optimizing processing power allocation based on local quality requirements rather than uniform processing.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If entity-specific frame rate control is implemented, then selective highlighting of objects is achieved, but system complexity increases

Engineering Contradiction:

Improveselective highlightingVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Depth information from depth maps is introduced as an additional dimension for entity identification and separation. This depth dimension enables automatic entity segmentation and frame rate assignment without requiring complex 2D image analysis algorithms, simplifying the system while achieving selective highlighting.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The system uses automatically generated depth maps and object detection algorithms to self-identify entities and determine appropriate frame rates without requiring manual annotation or complex user input. The depth information inherently provides the segmentation needed for selective processing.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11659135B2Slow or fast motion video using depth information

Publication Date: 2023.05.23 COREPHOTONICS

AI summary

Systems comprising a digital camera, an interface operable to mark a first entity in a frame of an input video stream and to determine a frame rate ratio FR1/FR2 between a first frame rate FR1 and a second frame rate FR2, a processor configurable to generate an output video stream of the digital camera, wherein the output video stream includes a first entity played at FR1 and a second entity played at FR2, and methods of using and providing same.