Real-time Video Overlaying Using Depth-Based Binary Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for video overlaying and superimposing are time-consuming and not suitable for real-time applications, especially in live streaming where flexibility and ease of use are essential, particularly when performed on mobile devices or wearable devices with multiple cameras.
Innovation Solution
A real-time video overlaying system and method utilizing a user device with multiple cameras and a depth sensor, where an image processor synchronizes video data, generates a binary mask, smooths edges, transforms the mask for drag-and-zoom features, and overlays the mask onto a second video stream, enabling real-time processing and flexible interaction during live streaming.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional techniques are used for video overlaying and superimposing, then image processing can be achieved, but the process is time-consuming and not suitable for real-time applications
Solution Approach 1:
The system segments the video processing task into distinct modules: depth data acquisition, binary mask generation, edge smoothing, and video merging. Each module processes specific aspects independently, enabling parallel computation and reducing overall processing time for real-time overlaying
Solution Approach 2:
The system performs preliminary actions by pre-processing depth data to generate binary masks and pre-smoothing edges before the actual video overlaying occurs. This preparation work is done in advance so that when real-time overlaying is needed, the computationally intensive tasks have already been completed or are in progress
2Adaptability or versatility
If conventional overlaying techniques are used on mobile devices, then video content can be created, but the techniques are outdated and require personal computers
Solution Approach 1:
The system is designed to be universal by implementing video overlaying functionality directly on mobile devices with multiple cameras, eliminating the need for personal computers. The same device performs both video capture and overlaying processing, making the technology accessible and easy to operate for live streamers anywhere
Solution Approach 2:
The mobile device serves itself by performing all video processing operations internally using its own processors and sensors. The device captures video from multiple cameras, processes depth data, generates masks, and merges videos all within itself, making the system self-contained and eliminating dependency on external computers
3Adaptability or versatility
If multiple cameras are used for live streaming, then flexible video content creation is enabled, but real-time overlaying becomes more complex
Solution Approach 1:
The system manages multiple camera inputs by segmenting the processing workflow into distinct stages: synchronizing video feeds from multiple cameras, independently generating binary masks for each, smoothing edges separately, and finally merging all videos with their respective masks. This modular approach reduces complexity by breaking down the challenging multi-camera overlaying task into manageable, sequential steps
Data Source
AI summary
Technologies relating to system and method of real-time video overlaying or superimposing display from multiple mutually synchronous cameras are disclosed. An example method of real-time video overlaying includes the steps of: synchronizing frame rates of a depth data, a face metadata, and a video data of a first camera video output captured by a first camera; determining a first depth between a user face and the first camera; using a cutoff depth to determine a user body contour; generating a binary mask of the user body contour based on the first depth and the cutoff depth; smoothing an edge of the binary mask; merging the binary mask with the first camera video output and generating a merged first camera video output; and overlaying the merged first camera video output onto a second camera video output.