OS Search Using Screenshot Embeddings for Transient Content
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing operating system (OS) search technologies are limited to returning exact matches, failing to capture and retrieve transient content such as web interactions and dynamic in-game content, leading to mechanical and limited search results.
Innovation Solution
Capture screenshots intermittently and index them using machine learning models to generate embeddings and full text indexing, enabling semantic and full text searches to retrieve a wide range of content, including visual and textual elements, and integrate results with relevance scoring and constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional exact match search is used, then search precision is maintained, but search versatility and ability to retrieve transient content deteriorates
Solution Approach 1:
The search system is segmented into multiple independent search paths: exact match search, semantic search using embeddings, and full-text search. Each path handles different types of queries independently, allowing the system to maintain precision for exact matches while gaining versatility through alternative search methods for transient and visual content
Solution Approach 2:
Embeddings serve as an intermediary representation that bridges exact text matching and semantic understanding. By converting both query and screenshot text to embedding vectors, the system enables approximate matching while maintaining a structured search space, thus achieving both precision and versatility
2Adaptability or versatility
If screenshots are captured and indexed for semantic search, then search versatility improves, but device complexity increases
Solution Approach 1:
Screenshots are captured and processed in advance, with text extracted and embeddings generated before user queries. This preliminary action transforms transient visual content into searchable data structures, enabling versatile search without adding complexity to the query processing path
Solution Approach 2:
Instead of storing and processing actual screenshot images for search, the system creates text-based copies through OCR extraction and embedding representations. This copying approach enables semantic search while avoiding the complexity of image processing and storage
3Adaptability or versatility
If multiple search indices are used, then search versatility improves, but processing time increases
Solution Approach 1:
The system performs partial searches across multiple indices in parallel, returning results from the most relevant index first. Rather than exhaustively searching all indices, it uses heuristics to determine which index is most likely to yield relevant results, reducing overall processing time while maintaining versatility
Solution Approach 2:
The search space is segmented into distinct indices (exact match, semantic embeddings, full-text) that can be queried independently and in parallel. This segmentation allows the system to distribute search operations across multiple data structures without sequential overhead, reducing total search time
Data Source
AI summary
The disclosed techniques provide improved methods of operating system (OS) search. Users are enabled to search for documents, emails, presentations, content they entered into a web form, meetings they participated in, and other interactions they had with their computing device. To accomplish this, screenshots are periodically captured and indexed. Machine learning models are used to infer embeddings for visual elements of the screenshots and/or text extracted from the screenshots. A full text index of a relational database may also be populated with text extracted from the screenshots. The embeddings and full text index may then be used to retrieve screenshots in response to a user history query. For example, screenshots of embeddings within a defined distance of an embedding of the user history query may be selected. Query results from different embedding indices and relational databases may be ordered by applying different weights to different kinds of search scores.