Neural Network Media Captioning for Surveillance Video Summarization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems for reviewing and navigating long media files, such as surveillance videos, are inefficient due to the lack of automatic caption generation and summarization capabilities, requiring manual scanning through hours of content to find specific events.

Innovation Solution

A method using a neural network to calculate importance scores for media segments based on content features, select relevant segments, generate captions, and create summaries, allowing for targeted captioning and improved navigation of media files.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual scanning is used to review long media files, then no automatic processing is required, but the time required to find specific events is excessive

Engineering Contradiction:

Improveevent finding speedVSAvoidreview time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent divides long media files into multiple segments and processes them in parallel using multiple processing units. This segmentation allows simultaneous analysis of different portions of the media file, dramatically reducing the total time required to find specific events while maintaining comprehensive coverage of the content.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary processing by generating captions and extracting key information from media segments before the user needs to search for specific events. This advance preparation creates an indexed structure that enables rapid event location without requiring manual scanning of the entire media file.

Inventive Principle:

Principle #10Preliminary action

2Ease of operation

If rapid scanning is used to navigate media files, then navigation speed improves, but automatic caption generation and summarization are not achieved

Engineering Contradiction:

Improvenavigation easeVSAvoidautomatic processing level

Core Design Contradiction:

Ease of operationVSExtent of automation

Solution Approach 1:

The system automatically generates captions and summaries without requiring user intervention. Multiple processing units independently analyze media segments, extract content features, and generate captions autonomously. The system self-manages the entire process from raw media input to structured summary output, maximizing automation while improving navigation ease.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual navigation and analysis mechanisms with automated neural network-based processing. Instead of mechanical scanning by users, the system uses intelligent algorithms to automatically understand, caption, and summarize media content, substituting human cognitive effort with automated intelligent processing.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If all media segments are processed in detail, then comprehensive analysis is achieved, but the complexity and time required increase significantly

Engineering Contradiction:

Improveanalysis completenessVSAvoidprocessing complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system processes media segments selectively rather than uniformly. Multiple processing units focus on different segments in parallel, and the system generates captions for all segments while prioritizing detailed analysis of key portions. This partial processing approach maintains comprehensive coverage while managing complexity through distributed parallel processing.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent combines multiple processing units that work in parallel to analyze different media segments simultaneously. By merging their individual results, the system achieves comprehensive analysis of the entire media file without requiring a single complex processor to handle everything sequentially, thereby reducing overall system complexity while maintaining analysis completeness.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS10592751B2Method and system to generate targeted captions and summarize long, continuous media files

Publication Date: 2020.03.17 FUJIFILM BUSINESS INNOVATION CORP

AI summary

A method of generating a summary of a media file that comprises a plurality of media segments is provided. The method includes calculating, by a neural network, respective importance scores for each of the media segments, based on content features associated with each of the media segments and a targeting approach, selecting a media segment from the media segments, based on the calculated importance scores, generating a caption for the selected media segment based on the content features associated with the selected media segment, and generating a summary of the media file based on the caption.