Local Convolutional Neural Network for Mobile Image Processing

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Mobile client devices with limited computing resources struggle to perform complex image processing tasks efficiently, leading to long processing times and increased power consumption, which negatively impacts user experience and battery life.

Innovation Solution

Implementing a local convolutional neural network on mobile devices to perform real-time or near-real-time image and video stream modifications, such as face manipulation, by using efficient convolutional kernel approximations and tensor projections, allowing for fast inference speed, compact model size, and low energy consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If complex image processing is performed on mobile client devices, then image modification capability is improved, but power consumption increases and processing time extends

Engineering Contradiction:
Improveimage modification capabilityVSAvoidpower consumption
Core Design Contradiction:
Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent replaces traditional mechanical image processing methods with neural network-based processing. The neural network model, trained on server-side computational resources, is deployed to mobile devices to perform complex image modifications such as face manipulation, style transfer, and object detection. This substitution enables mobile devices to execute sophisticated image processing tasks that would otherwise be computationally infeasible on such hardware, directly improving image modification capability while managing power consumption through efficient model design.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent employs parameter optimization techniques to adjust the neural network model for mobile deployment. By modifying model parameters such as reducing layer depth, decreasing filter counts, and optimizing activation functions, the system achieves a balance between processing capability and power consumption. These parameter changes enable the neural network to perform complex image modifications on mobile devices with limited computational resources and battery capacity.

Inventive Principle:
Principle #35Parameter changes

2Adaptability or versatility

If complex image processing is performed on mobile client devices, then image modification capability is improved, but processing time increases

Engineering Contradiction:
Improveimage modification capabilityVSAvoidprocessing time
Core Design Contradiction:
Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-training the neural network model on extensive image datasets using powerful server-side computational resources before deploying it to mobile devices. The model is pre-processed and optimized during the training phase, allowing it to perform complex image modifications efficiently on mobile hardware without requiring real-time training. This preliminary preparation enables fast inference speed on resource-constrained devices.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent replaces traditional mechanical image processing algorithms with a neural network-based system that has been optimized for mobile deployment. By substituting conventional processing methods with a pre-trained neural network, the system achieves faster processing times for complex tasks such as face manipulation and style transfer, as the neural network can leverage learned patterns rather than computing from scratch.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

3Productivity

If image processing is passed to networked computing devices, then processing capability is improved, but device complexity and network dependency increase

Engineering Contradiction:
Improveprocessing capabilityVSAvoidsystem architecture complexity
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The patent extracts the essential image processing functionality from complex server-side systems and encapsulates it in a standalone neural network model that can run independently on mobile devices. By taking out the core processing capability and packaging it as a portable model, the system reduces network dependency and simplifies the overall architecture while maintaining high processing capability. The mobile device becomes self-sufficient for image modifications without requiring constant network communication.

Inventive Principle:
Principle #2Taking out (Extraction)

Solution Approach 2:

The patent creates a universal neural network model that can perform multiple image processing functions on mobile devices, including face detection, manipulation, style transfer, and object detection. This multi-functional model replaces the need for multiple specialized processing systems, reducing device complexity and eliminating network dependency while maintaining comprehensive processing capability across various image modification tasks.

Inventive Principle:
Principle #6Universality (Multi-functionality)

Data Source

PatentUS20230384918A1Device-based image modification of depicted objects
Publication Date: 2023.11.30 SNAP INC
  • US20230384918A1 patent drawing
  • US20230384918A1 patent drawing
  • US20230384918A1 patent drawing

AI summary

A system of machine learning schemes can be configured to efficiently perform image processing tasks on a user device, such as a mobile phone. The system can selectively detect and transform individual regions within each frame of a live streaming video. The system can selectively partition and toggle image effects within the live streaming video.