Content-Aware Metadata for Dynamic Video Cropping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional media playback systems lack the ability to dynamically adjust video content based on the importance of visual objects, leading to low-quality displays with wasted screen space and discarded important visual data due to agnostic cropping and resizing methods.

Innovation Solution

The implementation of content-aware metadata that identifies and prioritizes visually important objects within a video file, allowing for dynamic adjustments such as cropping, resizing, and watermarking to optimize playback on various devices without discarding critical content.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional agnostic cropping and resizing methods are used, then the video can be played back on various devices, but important visual data is discarded and screen space is wasted

Engineering Contradiction:

Improvedevice compatibilityVSAvoidvisual data loss

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The system performs preliminary analysis of the video content to identify important visual objects and regions before playback. Metadata is generated in advance that describes the spatial and temporal characteristics of these important elements, enabling the playback device to make informed decisions about cropping and resizing without discarding critical content.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different quality preservation strategies to different regions of the video frame based on their importance. High-importance regions containing key visual objects are preserved with higher fidelity and prioritized during cropping, while less important background regions can be compressed or cropped more aggressively.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If conventional agnostic cropping and resizing methods are used, then the video can be adapted to different screen sizes, but screen real estate is wasted

Engineering Contradiction:

Improvescreen adaptationVSAvoidscreen real estate utilization

Core Design Contradiction:

Adaptability or versatilityVSArea of stationary object

Solution Approach 1:

The system dynamically adjusts the cropping and resizing parameters based on the importance map of visual objects in each frame. Rather than applying a fixed cropping strategy, the system adapts the crop region in real-time to follow and prioritize important visual elements, maximizing the utilization of screen real estate for meaningful content.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system allocates screen space differentially based on the importance of visual content in different regions. Important objects are positioned and sized to occupy optimal screen real estate, while less important regions are compressed or cropped, thereby maximizing the effective use of available display area.

Inventive Principle:

Principle #3Local quality

3Manufacturing precision

If content-aware metadata is generated and used, then playback quality is improved by maintaining important visual elements, but the system complexity increases

Engineering Contradiction:

Improveplayback qualityVSAvoidsystem complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The system introduces content-aware metadata as an intermediary layer between the original video content and the playback rendering. This metadata, which describes the spatial and temporal characteristics of important visual objects, enables the playback device to make intelligent adjustments without requiring complex real-time analysis algorithms, thereby improving playback quality while managing system complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Loss of information

If content-aware metadata is generated and used, then important visual elements are maintained during playback, but processing and storage requirements increase

Engineering Contradiction:

Improvevisual data preservationVSAvoiddata volume

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The system represents important visual content not by storing additional full-resolution image data, but by encoding parametric descriptions of object characteristics such as bounding boxes, motion vectors, and importance scores. This parametric representation preserves visual information while occupying minimal storage space compared to storing actual image pixels.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11557121B2System, device, and method for generating and utilizing content-aware metadata

Publication Date: 2023.01.17 CLOUDINARY LTD

AI summary

System, device, and method for generating and utilizing content-aware metadata, particularly for playback of video and other content items. A method includes: receiving a video file, and receiving content-aware metadata about visual objects that are depicted in said video file; and dynamically adjusting or modifying playback of that video file, on a video playback device, based on the content-aware metadata. The modifications include content-aware cropping, summarizing, watermarking, overlaying of other content elements, modifying playback speed, adding user-selectable indicators or areas around or near visual objects to cause a pre-defined action upon user selection, or other adjustments or modification. Optionally, a modified and content-aware version of the video file is automatically generated or stored. Optionally, the content-aware metadata is stored internally or integrally within the video file, in its header or as a private channel; or is stored in an accompanying file.