Action Clip Segmentation Using Audio Novelty Curves
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for editing video and audio clips, especially action footage, lack efficiency in automatically detecting action segments and integrating music seamlessly, often resulting in false positives and inefficient editing processes.
Innovation Solution
The method involves converting audio into an auditory spectrogram, calculating self-similarity and novelty curves to segment action clips, and using song folding techniques to create a structured representation of music for synchronized video sequencing, employing algorithms like K-Dijkstra for path optimization and stochastic logic for clip rhythm determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated action detection algorithms are used to segment video clips, then editing efficiency is improved, but false positives increase leading to inaccurate action segment detection
Solution Approach 1:
The patent segments the video clip into multiple portions based on detected action segments, then applies different processing operations to different segments. This allows automated editing to focus computational resources on action portions while maintaining accuracy by manually verifying specific segmented regions rather than processing the entire clip uniformly.
Solution Approach 2:
The patent introduces an intermediary manual review step where the user selects and confirms action segments detected by the automated algorithm. This intermediary process filters out false positives by requiring human verification, thereby maintaining high editing efficiency while improving detection accuracy through a hybrid automated-manual approach.
2Measurement precision
If manual review of action segments is performed to reduce false positives, then detection accuracy is improved, but editing time increases
Solution Approach 1:
The patent divides the video into action segments and non-action segments, allowing the operator to review only the detected action portions rather than the entire video. This segmented review approach maintains high detection accuracy while significantly reducing the time required for manual verification compared to reviewing the complete video clip.
Solution Approach 2:
The patent performs preliminary automated detection and segmentation of action segments before the manual review stage. This preliminary action pre-processes the video to identify potential action portions, so that subsequent manual review focuses only on confirming or rejecting these pre-identified segments, thereby reducing overall editing time while maintaining accuracy.
3Manufacturing precision
If complex song folding algorithms are used for music integration, then music-video synchronization quality is improved, but computational complexity increases
Solution Approach 1:
The patent segments both the video and music into corresponding portions based on action detection and musical structure analysis. By folding and repeating specific music segments to match video action segments, the system achieves high synchronization quality through structured segmentation rather than complex continuous transformation, reducing computational requirements.
Solution Approach 2:
The patent implements song folding where music segments are nested and repeated within the final soundtrack structure. Specific music portions are folded back and nested multiple times to match the duration and rhythm of action segments, creating high-quality synchronization through iterative nesting of musical segments rather than complex algorithmic composition.
Data Source
AI summary
In accordance with an embodiment, a method is disclosed for detecting action in a video clip. An audio clip is extracted from the video clip. The audio clip is converted to an auditory spectrogram. The auditory spectrogram is used to construct a self-similarity matrix. The self-similarity matrix is then used to calculate a novelty curve. The clip is then segmented into segments according to peaks in the novelty curve. Each of segments is scored, and then classified as an action clip if the score is above or below a predetermined threshold. Related methods for folding a digitized song into a shorter version of itself and for sequencing a set of user-supplied video and photo clips to a user-supplied song are further disclosed.