Mobile CNN Hair Segmentation for Real-Time Video Color Editing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Image processing, particularly for real-time video processing on mobile devices, is resource-intensive and challenging due to the complexity of tasks like hair segmentation and color augmentation, especially when limited computational power is available.
Innovation Solution
A convolutional neural network (CNN) is adapted for mobile devices using the MobileNet architecture, incorporating skip connections and a mask-image gradient consistency loss to achieve accurate hair segmentation and color change in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning neural networks are used for image processing tasks like hair segmentation and color augmentation, then the quality and accuracy of image processing results are improved, but the computational resource consumption and processing time increase significantly
Solution Approach 1:
The patent segments the deep learning network into a pre-training phase (performed on powerful servers) and an inference phase (performed on mobile devices). The network architecture is divided into feature extraction components trained offline and lightweight components executed in real-time on mobile devices, enabling high accuracy without continuous high computational resource consumption.
Solution Approach 2:
The patent performs preliminary training of the deep learning network on powerful servers before deployment to mobile devices. The network is pre-trained on large datasets and then fine-tuned with device-specific characteristics, so that when deployed on mobile devices, it requires minimal computational resources while maintaining high segmentation accuracy.
2Speed
If real-time video processing is performed on mobile devices, then the processing speed and responsiveness are improved, but the computational complexity and energy consumption increase
Solution Approach 1:
The patent applies local quality by optimizing the network architecture specifically for mobile device constraints. Different parts of the network are designed with different computational characteristics - feature extraction layers are lightweight for real-time processing, while more complex operations are performed offline or on server-side, balancing real-time performance with computational complexity.
3Manufacturing precision
If accurate hair segmentation is achieved through deep learning, then the quality of color augmentation is improved, but the training time and data requirements increase
Solution Approach 1:
The patent performs comprehensive network training and optimization in advance on powerful servers using large datasets. The pre-trained network captures general hair segmentation patterns, reducing the need for extensive training on mobile devices while maintaining high segmentation quality for color augmentation tasks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method implement deep learning on a mobile device to provide a convolutional neural network (CNN) for real time processing of video, for example, to color hair. Images are processed using the CNN to define a respective hair matte of hair pixels. The respective object mattes may be used to determine which pixels to adjust when adjusting pixel values such as to change color, lighting, texture, etc. The CNN may comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. The CNN may be trained for image segmentation (e.g. using coarse segmentation data) to minimize a mask-image gradient consistency loss. The CNN may further use skip connections between corresponding layers of an encoder stage and a decoder stage where shallower layers in the encoder, which contain high-res but weak features are combined with low resolution but powerful features from deeper decoder layers.