Audio-Visual Subject Removal for Seamless Video Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing technologies fail to effectively remove unwanted subjects from moving images with audio without degrading the audio quality, leading to a sense of incongruity during playback.
Innovation Solution
An image processing system and method that detects and deletes specific subjects from moving images, accompanied by corresponding audio components, using machine learning models to identify and match subject types in both visual and audio data, ensuring seamless playback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a subject that emits audio is deleted from a moving image with audio, then the unwanted subject is removed from the visual content, but the audio quality deteriorates due to residual audio components remaining in the playback
Solution Approach 1:
The audio data is segmented into multiple audio components, each corresponding to a different sound source. The system identifies and separates the audio component generated by the deleted subject from other audio components, allowing selective removal of only the problematic audio while preserving other sounds in the audio track.
Solution Approach 2:
The system uses detection results from the subject detection unit as feedback to guide the audio component separation and deletion process. By continuously monitoring the relationship between visual subject detection and audio components, the system adjusts the audio processing to ensure complete removal of the target subject's audio while maintaining overall audio quality.
2Device complexity
If subject detection and audio component detection are performed separately, then the processing complexity is reduced, but the synchronization between visual and audio deletion becomes difficult
Solution Approach 1:
The system merges the subject detection process with the audio component detection process by establishing correspondence relationships between detected subjects and audio components. This integration allows the system to maintain separate detection functions while ensuring they work together cohesively to achieve synchronized audio-visual processing.
Data Source
AI summary
There is provided an image processing system. A subject detection unit detects, among a plurality of frames of moving image data accompanied by audio data, a subject of a first type. A subject deletion unit deletes the subject of the first type from one or more frames, among the plurality of frames, in which the subject of the first type is detected. An audio detection unit detects, in the audio data, an audio component corresponding to the subject of the first type. An audio deletion unit deletes the audio component corresponding to the subject of the first type from the audio data.