Real-time Style Transfer via Object Masking and Segmentation

The patent segments the style transfer process into two distinct stages: a training phase where heavy machine learning models process static style reference images to create style maps, and an inference phase where lightweight convolutional neural networks process video frames in real-time. This segmentation allows computationally intensive operations to be performed offline during training, while enabling real-time performance during actual style transfer execution.

The patent applies preliminary action by pre-processing style reference images to generate style maps and pre-training convolutional neural networks before the actual real-time style transfer operation. The heavy computational workload of model training and style map generation is completed in advance, leaving only lightweight inference operations for real-time execution during video processing.

The patent applies local quality by generating style maps at the object level using segmentation algorithms that identify and process different regions of the image separately. The convolutional neural network then applies style transfer locally to each segmented region, allowing different parts of the video to receive appropriate style transformations based on their specific content characteristics, thereby reducing visual artifacts and improving overall quality.

The patent segments the video processing into distinct object-level regions using segmentation algorithms, processing each segment independently with the convolutional neural network. This segmentation approach allows the model to maintain temporal consistency across frames by processing similar objects in the same manner across multiple frames, eliminating visual artifacts and ensuring stable style transfer.