Video-to-Music Model With Contrastive Training for Event Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of video sharing platforms often struggle to find background music that matches their video's desired mood and aligns with its events, as existing techniques using symbolic music annotations lack expressivity and fail to capture nuances like timbre, articulation, and rhythm, and have limited genre diversity.
Innovation Solution
A video-to-music machine learning model that generates background music using a video encoder and autoregressive decoder, trained with a video-music contrastive loss term and autoregressive loss term to align music events with video events, utilizing a training dataset to enhance music generation fidelity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If symbolic music annotations (e.g., MIDI) are used for video-based music generation, then the generation process can be performed, but the expressivity and fidelity of the generated music is limited due to inability to capture nuances like timbre, articulation, and rhythm
Solution Approach 1:
The patent changes the fundamental parameter representation from symbolic MIDI annotations to waveform audio data. This parameter transformation enables the model to capture fine-grained musical nuances including timbre, articulation, and rhythm that are lost in symbolic representations, thereby simultaneously improving both fidelity and expressivity of generated music
Solution Approach 2:
The patent replaces the mechanical symbolic annotation system (MIDI) with a neural network-based waveform generation system. This substitution allows the model to directly learn and generate continuous audio waveforms from video inputs, capturing temporal and spectral characteristics that symbolic systems cannot represent, thus resolving the contradiction between fidelity and expressivity
2Productivity
If pre-produced music is used for videos, then users can find music quickly, but the music may not match the user's intentions or align with video events
Solution Approach 1:
The patent implements a self-service system where the model automatically generates custom background music directly from video inputs without requiring users to manually search through pre-produced music libraries. The system extracts visual features from the video and autonomously generates music that matches the video's mood, tone, and events, thereby simultaneously improving selection efficiency and alignment precision
Solution Approach 2:
The patent introduces a video-to-music machine learning model as an intermediary between video content and music generation. This intermediary processes video frames to extract relevant features and transforms them into corresponding musical characteristics, enabling precise alignment between video events and music patterns while eliminating the need for manual music selection
Data Source
AI summary
A computing system including one or more processing devices configured to receive an input video. At a video-to-music machine learning model including a video encoder and an autoregressive decoder, the one or more processing devices compute video feature tensors at the video encoder based at least in part on the input video. The one or more processing devices autoregressively generate music tokens at the autoregressive decoder based at least in part on the video feature tensors. The video-to-music machine learning model has been trained using a training data set including training input pairs that each include a training input video and training background music. The training also uses a loss function including a video-music contrastive loss term and an autoregressive loss term. The one or more processing devices convert the music tokens into background music associated with the input video. The one or more processing devices output the background music.