Mobile Video Face Stabilization for Bandwidth Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In mobile video conferencing, conventional techniques for detecting moving objects fail due to non-stationary backgrounds, leading to reduced clarity of facial expressions caused by changes in position, size, orientation, lighting, and color, and inefficient bandwidth usage.
Innovation Solution
A method that identifies and processes face data to maintain the face's area, light direction, and color constant, using feature templates and low-pass filtering to stabilize the image, thereby enhancing expression clarity and optimizing bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional background subtraction and frame differencing techniques are used to detect moving objects, then object detection can be achieved, but the technique fails when the background is non-stationary (user is moving)
Solution Approach 1:
The patent creates a virtual stationary background by copying and transforming the background from a reference frame. The background is extracted, transformed to match the current frame's perspective, and composited with the foreground to create a synthetic background that accounts for camera motion, enabling accurate moving object detection in mobile conferencing scenarios
Solution Approach 2:
The patent introduces a virtual background as an intermediary between the actual moving background and the object detection process. This virtual background serves as a mediator that reconciles the conflict between camera motion and the stationary background assumption required by conventional detection techniques
2Loss of energy
If raw video data is encoded to meet low maximum bit rate requirements, then bandwidth efficiency is improved, but the clarity of facial expressions diminishes due to variations in lighting and position
Solution Approach 1:
The patent applies parameter changes by adjusting lighting parameters (illumination direction, intensity) and position parameters (face location, orientation) in the video data. By normalizing these parameters across frames, the patent reduces the variation that would otherwise require high bit rates to encode, thereby improving bandwidth efficiency while preserving facial expression clarity
Solution Approach 2:
The patent performs preliminary processing of the video data by detecting and correcting lighting variations and position changes before encoding. This preliminary action removes unnecessary variations that would consume bandwidth, allowing the encoder to focus bit rate on preserving actual facial expressions rather than compensating for environmental variations
3Illumination intensity
If white balance compensation is applied to adjust RGB channel gains, then overall image brightness is balanced, but the face appears green or blue due to over-correction
Solution Approach 1:
The patent applies local quality by treating the face region differently from the rest of the image. While white balance compensation is applied to the overall image to correct brightness balance, the patent selectively adjusts or limits the correction in the face region to prevent over-correction that would cause unnatural green or blue coloration, thereby maintaining both overall balance and local color accuracy
Data Source
AI summary
Video data comprising a plurality of sets of frame data is captured by mobile video data capture device. The video data is processed by a method which comprises: (a) finding a face in each frame of the video, and (b) processing a corresponding set of frame data to: (i) maintain the area of the image occupied by the face substantially constant; and (ii) maintain the apparent direction of light incident upon the face substantially constant; and/or (iii) maintain the apparent color of the face substantially constant.