Parallel Object Extraction and Masking for Real-Time Video Clipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processing technologies fail to meet the real-time requirements for object extraction and clipping processing in video and audio communication, leading to delays and hindered smooth communication in scenarios like remote concerts.
Innovation Solution
A video processing system utilizing a software processing unit for advanced object detection and contour extraction, combined with a hardware processing unit for generating mask information, performing these processes in parallel to reduce processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If object extraction and clipping processing is performed using classical methods or deep learning, then object extraction accuracy is improved, but processing time increases to several tens of milliseconds or more
Solution Approach 1:
The video processing system divides the object extraction and clipping processing into separate functional modules: object detection unit, contour extraction unit, and clipping processing unit. Each module handles a specific task independently, allowing parallel processing and reducing overall processing time while maintaining extraction accuracy through specialized processing for each stage.
Solution Approach 2:
The system performs object detection and contour extraction in advance before the actual clipping processing. By pre-processing and identifying object boundaries beforehand, the system prepares mask information and contour data that can be quickly applied during clipping operations, reducing real-time processing delays.
2Speed
If object extraction and clipping processing is performed within the video and audio processing time, then real-time communication requirements are improved, but the processing time available for object extraction is reduced to several milliseconds or less
Solution Approach 1:
The system performs object detection, contour extraction, and clipping processing continuously across multiple video frames without interruption. By maintaining continuous processing pipelines and using frame-by-frame or block-based processing, the system ensures that object extraction operations are always ready when needed, meeting real-time communication requirements with only several milliseconds of latency.
3Productivity
If data reception time for 60 frames per second video is 16.7 milliseconds, then video frame rate is improved, but additional data processing time is required beyond this reception time
Solution Approach 1:
The system processes video data in periodic intervals corresponding to the 60 frames per second rate, using fixed time slots for object detection, contour extraction, and clipping operations. By synchronizing processing cycles with the video frame rate and using periodic processing batches, the system minimizes additional processing time beyond the 16.7 milliseconds required for data reception.
Data Source
AI summary
An object of the present disclosure is to reduce a time required for object extraction and clipping processing. The present disclosure provides a video processing system including a software processing unit configured to detect an object included in at least some of input images included in an input video and extract a contour of the object, and a hardware processing unit configured to generate mask information for clipping out the object from the input images included in the input video by using the contour extracted by the software processing unit, in which the software processing unit and the hardware processing unit perform processing independently in parallel.