Memory Video Creation Using Semantic Media Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing and navigating large media libraries efficiently to retrieve contextually-relevant content is cumbersome and inefficient, often requiring extensive user interaction and device resources.
Innovation Solution
A method utilizing machine-learning models to generate a video from media assets based on a natural language description of a memory, involving multiple models for query token generation, trait determination, and story outlining to select and arrange media items, with metadata and embedding searches for efficient content retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If media libraries are organized in chronological order, then media items are systematically arranged, but users must manually navigate through large numbers of directories to find contextually-relevant content, wasting time and device resources
Solution Approach 1:
The system automatically analyzes user queries and retrieves relevant media content without requiring manual navigation. The machine learning model processes the query, identifies relevant media items across the entire library, and presents results directly, allowing the system to serve itself rather than requiring user intervention to browse through chronological directories
Solution Approach 2:
The patent replaces the mechanical navigation process (manual browsing through directories) with an intelligent information retrieval system. Instead of physically moving through folder structures, the system uses natural language processing and machine learning to directly locate and present relevant content based on semantic understanding of the user's query
2Productivity
If existing techniques are used to retrieve contextually-relevant content, then content can be found, but the process requires multiple key presses and complex user interface interactions that waste user time and device energy
Solution Approach 1:
The system performs preliminary processing by pre-analyzing media items and organizing them in a structure optimized for semantic search. When a user query is received, the system can quickly retrieve relevant content without requiring multiple interface interactions, as the heavy lifting of content organization and relationship mapping has already been done in advance
Solution Approach 2:
The patent extracts the essential function of content retrieval from the complex user interface layer and implements it through a dedicated machine learning-based search system. This separates the intelligence required for content understanding from the user interaction layer, allowing fast, energy-efficient retrieval without requiring users to navigate complex interfaces
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
The present disclosure generally relates to generating a video corresponding to a memory (e.g., an event or context) from media assets on a device. In some embodiments, the device receives user inputs requesting a video based on a natural language description of a memory. The device sends information of the natural language description to a first machine-learning (ML) model, and receives query tokens, which are used to find media items on the device that match the query tokens. The device sends information representing the found media items to another ML model that determines traits from the media items. These traits are sent to a third ML model to generate a story outline, and the video is generated by comparing the descriptions of shots in the story outline to visual embeddings of the found media assets to curate and arrange them into the video consistent with the story outline.