Mobile Neural Network Style Transfer via Offline Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image manipulation techniques, such as neural network-based image style transfer, are computationally complex and impractical for implementation on mobile devices due to high processing requirements, making real-time application on smartphones or in live video impractical without server support, which may not always be available and can compromise user data privacy.
Innovation Solution
A style transfer system processes a portion of an image using a trained neural network, adjusts the color spectrum and shape to match the original, and integrates the modified portion seamlessly, allowing for efficient complex image schemes on mobile devices without server support, enabling real-time processing of images and videos.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural network-based image style transfer is implemented, then image manipulation capability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the image processing task by training the neural network offline to learn style transfer mappings, then only applying the pre-trained model during runtime. This separates the computationally intensive training phase from the lightweight inference phase, enabling complex image manipulation without requiring high computational resources during actual use.
Solution Approach 2:
The patent performs preliminary training of the neural network model before deployment, storing the learned style transfer parameters in the device. This preliminary action transfers the computational burden from runtime processing to offline preparation, allowing the mobile device to execute simple forward propagation rather than full training during actual image style transfer operations.
2Productivity
If complex image manipulation is performed in real-time, then processing speed is improved, but processing requirements increase
Solution Approach 1:
The patent divides the image processing workflow into offline training (performed once) and online inference (performed repeatedly in real-time). The offline training phase captures all computationally intensive operations, while the online phase only requires lightweight forward propagation through the trained network, enabling real-time processing with minimal power consumption.
Solution Approach 2:
The patent changes the operational parameters of the neural network by pre-computing and storing optimal transformation parameters during training. During real-time operation, the system only needs to retrieve and apply these pre-determined parameters rather than performing complex calculations, significantly reducing processing requirements while maintaining real-time performance.
3Power
If server support is used for image processing, then computational capability is improved, but data privacy is worsened
Solution Approach 1:
The patent extracts the neural network model from the server environment and embeds it directly into the mobile device. This extraction transfers the computational capability from remote server to local device, eliminating the need to upload images to external servers and thereby protecting user data privacy while maintaining advanced image processing capabilities.
Solution Approach 2:
The patent enables the mobile device to perform image style transfer independently using the embedded neural network model. The device serves itself by locally processing images without requiring external server support, thus maintaining computational capability while ensuring data remains private and never leaves the user's device.
Data Source
AI summary
A mobile device can implement a neural network-based style transfer scheme to modify an image in a first style to a second style. The style transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The style transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.