Operating System Visual Search with On-Device ML Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face difficulties in obtaining additional information associated with visually presented content across different applications and media files due to the challenges of constructing effective search queries, as text-based searching is often inadequate and screenshot methods yield irrelevant results.
Innovation Solution
A visual search interface in the operating system leverages on-device machine-learned models to process display data, perform object detection, optical character recognition, and segmentation, providing application suggestions and additional information without relying on server computing systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If text-based searching is used to find information, then the search process is simple, but the search results are inadequate and irrelevant
Solution Approach 1:
The patent replaces text-based search (mechanical keyword matching) with visual search using machine learning models. The system captures display content, processes it through object detection and OCR models to extract visual features, and performs image-based similarity search to find relevant information, thereby substituting the inadequate text-searching mechanism with a more effective visual recognition system.
Solution Approach 2:
The patent changes the search parameter from text keywords to visual image data. Instead of searching based on user-typed words, the system converts display content into image representations and uses image similarity metrics (cosine similarity, Euclidean distance) to retrieve relevant information, fundamentally changing the search parameter space to improve result relevance.
2Loss of information
If screenshot capture is used as a search query, then visual content can be searched, but the search leads to irrelevant results and requires multiple user inputs
Solution Approach 1:
The patent implements self-service by enabling the system to automatically capture display content and perform visual search without requiring manual screenshot capture or cropping by the user. The visual search interface monitors and captures relevant display areas autonomously, processes the images through ML models, and retrieves results, eliminating the need for multiple user inputs while maintaining visual search capability.
3Loss of information
If visual search processing is performed on server computing systems, then comprehensive search can be achieved, but computational costs and data transmission increase
Solution Approach 1:
The patent segments the visual search processing into two parts: lightweight on-device processing using machine learning models for initial image analysis and feature extraction, and server-side processing for comprehensive search execution. This segmentation allows essential processing to occur locally (reducing data transmission) while maintaining access to comprehensive search capabilities on the server, thereby balancing search completeness with energy efficiency.
Data Source
AI summary
Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.