On-Device Object Recognition for Augmented Reality

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing augmented reality applications on mobile devices rely on remote server processing for object recognition, leading to latency and privacy concerns due to data transmission.

Innovation Solution

Implementing on-device machine learning models, such as convolutional neural networks, to perform real-time object recognition directly on the mobile device, reducing latency and enhancing privacy by minimizing data transmission.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If remote server processing is used for object recognition, then processing power and model complexity can be increased, but latency increases and privacy concerns arise due to data transmission

Engineering Contradiction:

Improveobject recognition accuracyVSAvoidprocessing latency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent extracts the machine learning model from the remote server and places it directly on the mobile device. This allows the device to perform object recognition locally without transmitting image data to external servers, thereby reducing latency while maintaining recognition accuracy through on-device processing

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent introduces an on-device machine learning model as an intermediary between the camera feed and the application logic. This local model processes images directly on the device, eliminating the need for data transmission to remote servers while still providing accurate object recognition results

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If remote server processing is used for object recognition, then complex models can be deployed, but privacy concerns increase due to data transmission

Engineering Contradiction:

Improveobject recognition accuracyVSAvoidprivacy concerns

Core Design Contradiction:

Measurement precisionVSObject-affected harmful factors

Solution Approach 1:

The patent extracts the processing function from remote servers and implements it locally on the mobile device. This eliminates the transmission of personal image data to external servers, addressing privacy concerns while maintaining accurate object recognition through on-device machine learning models

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The mobile device performs object recognition independently using locally deployed machine learning models. The device serves its own processing needs without requiring external server assistance, thereby preventing data transmission and associated privacy risks while maintaining recognition accuracy

Inventive Principle:

Principle #25Self-service

3Loss of time

If on-device machine learning models are implemented, then latency is reduced and privacy is enhanced, but device complexity and computational requirements increase

Engineering Contradiction:

Improveprocessing latencyVSAvoidmodel deployment complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent implements a universal machine learning framework that can run on standard mobile devices with existing hardware capabilities. The system leverages common mobile device components (CPU, GPU, neural processing units) to execute complex models, making the solution broadly applicable without requiring specialized hardware

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Productivity

If on-device machine learning models are implemented, then real-time processing is achieved, but energy consumption increases

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoiddevice energy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent implements periodic or event-triggered object recognition processing rather than continuous analysis of all camera frames. The system processes images at optimized intervals or only when specific conditions are met, reducing overall energy consumption while maintaining real-time processing capability for critical detections

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The system performs partial processing by focusing computational resources on detecting specific object types or regions of interest rather than analyzing entire images exhaustively. This selective approach reduces energy consumption while maintaining real-time detection capability for priority objects

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250044912A1Object based content recommendation

Publication Date: 2025.02.06 SNAP INC

AI summary

Systems and methods for object-based content recommendation are described. A camera feed comprising a plurality of image frames is caused to be displayed at a client device. An object is detected within an image frame from the camera feed, the object corresponding with an object category. Responsive to detecting the object, an icon associated with the object category is selected and displayed at a position upon the camera feed. The icon corresponds with a media collection related to the object category. An input is received selecting the icon. Responsive to the input, a presentation of media items from the media collection is displayed at the client device. By detecting real-world objects and surfacing relevant virtual icons that link to associated media, an augmented reality experience is provided allowing virtual content to be overlaid and anchored to objects in reality.