Dynamic Neural Network Style Transfer for Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing algorithms for artistic style transfer, such as Gatys' neural algorithm, are computationally intensive and time-consuming, making it impractical to apply artistic styles to images or videos in real-time on portable devices due to thermal and processing constraints, and they often result in undesirable 'jitter' or 'flicker' in video sequences.
Innovation Solution
The techniques involve extracting artistic styles from source images and storing them as layers in a neural network, using optimization methods like scalar control and dynamically adjustable Deep Neural Networks (DNNs) to apply these styles efficiently, while employing temporal consistency constraints to maintain smoothness in video sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If Gatys' neural algorithm is used for artistic style transfer, then artistic style can be applied to images, but processing time and computational power requirements increase significantly
Solution Approach 1:
The patent divides the style transfer process into multiple stages: first extracting artistic style from source images, then applying it to target images. The neural network processes images at different resolution levels, segmenting the computational task to reduce overall processing time while maintaining quality.
Solution Approach 2:
The patent employs dynamically adjustable Deep Neural Networks that can adapt their complexity based on system performance parameters. The network structure can be modified during operation to balance between processing speed and output quality, allowing real-time adjustments to computational requirements.
2Ease of manufacture
If Gatys' neural algorithm is applied to portable devices, then artistic style transfer can be performed, but thermal and processing constraints are exceeded
Solution Approach 1:
The patent uses dynamically adjustable neural networks that can modify their computational complexity based on thermal conditions. When temperature increases, the network automatically reduces its operational intensity to prevent overheating, while maintaining acceptable style transfer quality.
Solution Approach 2:
The system changes key parameters such as resolution and network complexity based on thermal constraints. By adjusting these parameters dynamically, the system can perform style transfer on portable devices without exceeding thermal limits, balancing performance and thermal management.
3Productivity
If artistic style transfer is applied to each image in a video sequence, then style transfer can be performed on all frames, but random jitter or flicker appears in the assembled video
Solution Approach 1:
The patent extracts artistic style from source images beforehand and stores it in the neural network before processing target images. This preliminary style extraction ensures that the same style representation is consistently applied across all video frames, preventing jitter and flicker in the final assembled video sequence.
Solution Approach 2:
The system uses feedback from temporal consistency constraints to maintain stability across video frames. By comparing and constraining the style transfer results across sequential frames, the system eliminates random variations and ensures consistent appearance throughout the video sequence.
4Measurement precision
If high-resolution images are processed through style transfer, then image quality is maintained, but processing power and time requirements increase
Solution Approach 1:
The patent processes images at multiple resolution levels, applying style transfer first at lower resolutions to reduce computational load, then progressively refining the result at higher resolutions. This segmentation of the processing task maintains image quality while significantly reducing overall processing power requirements.
Solution Approach 2:
The dynamically adjustable neural network can change its computational complexity based on the input image resolution and system capabilities. When processing high-resolution images, the network automatically adjusts its operational parameters to balance between maintaining quality and reducing processing power consumption.
Data Source
AI summary
Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.