Zero-Exemplar Event Detection via Joint Embedding Space

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Zero-exemplar event detection in unconstrained consumer videos is challenging due to the lack of prior exemplars, making it difficult to retrieve videos semantically relevant to novel queries, as there are no direct annotations for novel queries in the text description.

Innovation Solution

A method involving a neural network that propagates text blocks and training videos into a joint space to reduce the distance between textual and visual representations, using a deep convolutional network to embed visual and textual modalities, allowing for the ranking of videos based on their similarity to a novel query without pre-defined queries.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If no exemplar is given a priori for novel queries, then the system can handle any textual description freely, but it becomes difficult to retrieve videos semantically relevant to the query

Engineering Contradiction:

Improvequery flexibilityVSAvoidsemantic similarity measurement

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent introduces a joint embedding space as an intermediary that bridges textual queries and video content. This shared representation space enables semantic similarity measurement between novel queries and videos without requiring pre-defined annotations, thus resolving the contradiction between query flexibility and measurement precision

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system transforms both textual queries and video content into a common parameter space (joint embedding space) with unified dimensions. By changing the representation parameters of both modalities to be compatible, the system enables direct semantic comparison while maintaining query flexibility

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If a joint embedding space is learned to improve video retrieval accuracy, then semantic similarity can be measured effectively, but the system complexity increases

Engineering Contradiction:

Improvesemantic similarity measurementVSAvoidneural network structure

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent merges the textual embedding process and video embedding process into a unified joint embedding space. By combining these functions into a single neural network architecture with shared parameters, the system achieves effective semantic measurement while controlling overall system complexity through parameter sharing

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS10496885B2Unified embedding with metric learning for zero-exemplar event detection

Publication Date: 2019.12.03 QUALCOMM INC

AI summary

A method, a computer-readable medium, and an apparatus for zero-exemplar event detection are provided. The apparatus may receive a plurality of text blocks, each of which may describe one of a plurality of pre-defined events. The apparatus may receive a plurality of training videos, each of which may be associated with one of the plurality of text blocks. The apparatus may propagate each text block through a neural network to obtain a textual representation in a joint space of textual and video representations. The apparatus may propagate each training video through the neural network to obtain a visual representation in the joint space. The apparatus may adjust parameters of the neural network to reduce, for each pair of associated text block and training video, the distance in the joint space between the textual representation of the associated text block and the visual representation of the associated training video.