Video Editing Using Copied Unet Models and Self-Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning technologies for video editing, particularly those using text-to-image models, struggle to meet higher requirements in actual applications, such as reducing calculation costs and improving editing effects.
Innovation Solution
The proposed solution involves using a pre-trained text-to-image model, specifically a Unet model, to perform video editing by determining noised codes for video frames and a text code for guiding editing. Denoising processing is done using multiple copied Unet models with self-attention layers, and decoding processing generates target images for an edited video.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional machine learning models are used for video editing, then basic editing functionality is achieved, but calculation costs are high and editing effects are limited
Solution Approach 1:
The video editing task is segmented into multiple independent Unet models, each handling specific video frames. This allows parallel processing of different frame segments, reducing overall calculation cost while maintaining editing quality through distributed computation across multiple specialized models
Solution Approach 2:
The Unet models are pre-trained on image data before being applied to video editing tasks. This preliminary training action enables the models to already possess relevant feature extraction capabilities, reducing the need for extensive video-specific training and thereby lowering calculation costs while preserving editing effectiveness
2Productivity
If multiple Unet models are used for parallel processing, then processing speed improves, but device complexity increases
Solution Approach 1:
Multiple Unet models are created as copies of a base model architecture. This copying approach enables parallel processing of multiple video frames simultaneously, improving processing speed while maintaining manageable complexity through standardized, reusable model templates that can be deployed efficiently
3Reliability
If self-attention layers are added to Unet models, then editing coherence improves, but calculation cost increases
Solution Approach 1:
Self-attention layers are selectively integrated into specific portions of the Unet model architecture rather than uniformly across all layers. This localized application of self-attention mechanisms improves editing coherence in critical regions while minimizing the overall calculation cost increase by avoiding redundant attention computations in less critical areas
Data Source
AI summary
A computer-implemented method includes determination of n noised codes corresponding to n video frames of an original video. A text code corresponding to a description text guiding video editing is determined. Denoising processing, using n Unet models obtained by using the text code and copying a Unet model, is performed on the n noised codes, where a pre-trained text-to-image model includes the Unet model, which includes a self-attention layer connected after a target network layer, and where the denoising processing includes performing, in a self-attention layer of any ith Unet model, attention calculation based on an output of a target network layer of the ith Unet model and an output of a target network layer in a predetermined target Unet model. Decoding processing is separately performed on the n denoised codes by an image decoder to obtain n target images to form an edited target video.