Screenshot Graph Neural Network Retrieval Using Query Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of manually retrieving and organizing captured screenshots is cumbersome, time-consuming, and resource-intensive, especially when a large number of screenshots are involved.
Innovation Solution
A graph-forming process using a graph neural network (GNN) is employed to generate a graph representing screenshots, where machine-trained models determine features and edges, and a retrieval process uses the GNN to efficiently match query embeddings with target embeddings for screenshot retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual retrieval of screenshots is used, then simplicity of implementation is maintained, but time consumption and resource consumption increase significantly
Solution Approach 1:
The patent replaces manual mechanical search processes with an automated neural network-based information retrieval system. The neural network automatically indexes, stores, and retrieves screenshot information based on query embeddings, eliminating the need for manual browsing and searching through captured screenshots.
Solution Approach 2:
The patent introduces an intermediary neural network system that mediates between the user's screenshot retrieval needs and the stored screenshot database. The neural network processes queries, generates embeddings, and retrieves relevant screenshots automatically, acting as an intelligent intermediary that reduces manual intervention and time consumption.
2Productivity
If manual searching through screenshots is used, then device simplicity is maintained, but memory and processing resource consumption increase
Solution Approach 1:
The patent performs preliminary action by pre-processing and indexing all captured screenshots into the neural network database before retrieval is needed. The system pre-generates embeddings and organizes screenshot metadata, so that during actual retrieval operations, the system can quickly query and retrieve relevant screenshots without performing resource-intensive analysis in real-time.
Solution Approach 2:
The patent substitutes manual mechanical search processes with an automated neural network-based information retrieval system. The neural network automatically indexes, stores, and retrieves screenshot information based on query embeddings, eliminating the need for manual browsing and searching through captured screenshots.
3Productivity
If a graph neural network is used to represent screenshots, then retrieval efficiency is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary neural network system that mediates between the user's screenshot retrieval needs and the stored screenshot database. The neural network processes queries, generates embeddings, and retrieves relevant screenshots automatically, acting as an intelligent intermediary that reduces manual intervention and time consumption.
Solution Approach 2:
The patent applies universality by designing a neural network system that performs multiple functions: it processes various types of input data (images, text, video frames), generates embeddings, retrieves relevant information, and can handle different query types. This multi-functional approach consolidates what would otherwise require multiple separate systems into a single unified neural network architecture.
Data Source
Figure 1
Figure 2~4
Figure 5~6
AI summary
A graph-forming process generates a graph having nodes that represent a plurality of previously captured screenshots. The graph-forming process relies on a plurality of machine-trained models to identify edges between pairs of the nodes. The edges represent relationships among the screenshots. The graph-forming process then trains a graph neural network (GNN) based on the graph. The training produces a plurality of target embeddings associated with respective nodes in the graph. A retrieval process retrieves a previously captured screenshot using the plurality of target embeddings. The retrieval process involves adding a new node to the graph that represents the query and using the GNN to produce a query embedding associated with the new node. The retrieval process then finds at least one target embedding that matches the query embedding and retrieves a screenshot associated with the matching target embedding.