Screenshot Retrieval Using Graph Neural Network Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of manually retrieving and organizing captured screenshots is cumbersome, time-consuming, and resource-intensive, especially when a large number of screenshots are involved.
Innovation Solution
A graph-forming process using a graph neural network (GNN) is employed to generate a graph representing screenshots, where machine-trained models determine features and edges, and a retrieval process uses the GNN to efficiently match query embeddings with target embeddings for screenshot retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If screenshots are manually organized and retrieved, then retrieval accuracy can be maintained, but the process becomes cumbersome and time-consuming
Solution Approach 1:
The system performs preliminary actions by automatically generating embeddings for all screenshots in advance and organizing them in a graph structure with pre-computed relationships. This allows the retrieval process to simply query pre-processed data rather than manually searching through raw screenshots, thus maintaining accuracy while dramatically reducing retrieval time.
Solution Approach 2:
The patent introduces embeddings as an intermediary representation between the original screenshots and the retrieval process. These embeddings serve as a mediator that captures the essential features and relationships of screenshots in a compressed vector form, enabling efficient similarity search without requiring manual inspection of actual screenshot content.
2Quantity of substance
If a large number of screenshots are stored locally for reference, then retrieval completeness is improved, but memory and processing resources are significantly consumed
Solution Approach 1:
The system extracts only the essential features from screenshots by converting them into embeddings, separating the critical information (visual and textual features) from the full screenshot data. This extraction allows the system to work with compressed representations that consume minimal memory and processing resources while still enabling effective retrieval of large screenshot collections.
Solution Approach 2:
The patent transforms screenshots from their original high-dimensional image format into low-dimensional embedding vectors, fundamentally changing the parameter representation. This parameter transformation reduces the computational complexity from processing entire images to comparing compact vectors, enabling the system to handle large numbers of screenshots with limited resources.
3Measurement precision
If comprehensive features are extracted from screenshots using multiple machine-trained models, then retrieval precision is improved, but device complexity increases
Solution Approach 1:
The system segments the feature extraction process by using multiple specialized machine-trained models, each responsible for extracting specific types of features (visual features, textual features, etc.). This segmentation allows each model to focus on a particular aspect of screenshot analysis, improving overall retrieval precision while organizing the complexity into manageable, modular components.
Data Source
AI summary
A graph-forming process generates a graph having nodes that represent a plurality of previously captured screenshots. The graph-forming process relies on a plurality of machine-trained models to identify edges between pairs of the nodes. The edges represent relationships among the screenshots. The graph-forming process then trains a graph neural network (GNN) based on the graph. The training produces a plurality of target embeddings associated with respective nodes in the graph. A retrieval process retrieves a previously captured screenshot using the plurality of target embeddings. The retrieval process involves adding a new node to the graph that represents the query and using the GNN to produce a query embedding associated with the new node. The retrieval process then finds at least one target embedding that matches the query embedding and retrieves a screenshot associated with the matching target embedding.