Head-Wearable AI Assistant for Low-Latency Screen Text Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for screen-text recognition and multimodal large language models require sending large images to remote servers, leading to increased latency, computational resource usage, and decreased accuracy, which negatively impacts user experience.
Innovation Solution
Implementing an artificially intelligent assistant on wearable devices with on-device scene text recognition models and lightweight multimodal large language models to perform operations locally, reducing latency and computational requirements while maintaining high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large images are sent to remote server for screen-text recognition and multimodal large language model processing, then recognition accuracy is improved, but latency and computational resource usage increase
Solution Approach 1:
The patent divides the image processing task into two segments: on-device scene text recognition that extracts text and locations from images, and server-side multimodal large language model processing that generates responses. This segmentation allows accurate text extraction locally while minimizing data transmission, thereby reducing latency while maintaining recognition accuracy.
Solution Approach 2:
The patent extracts only the essential elements (text and text locations) from the image using on-device scene text recognition, rather than transmitting the entire image to the server. This extraction principle reduces the amount of data transmitted and processed remotely, decreasing latency and computational resource usage while preserving the accuracy needed for generating accurate responses.
2Measurement precision
If large images are sent to remote server for screen-text recognition and multimodal large language model processing, then recognition accuracy is improved, but computational resource usage increases
Solution Approach 1:
The patent segments computational tasks between on-device and server components. The computationally intensive scene text recognition is performed locally on the wearable device using optimized models, while the server handles only the response generation based on extracted text. This division reduces the computational burden on the wearable device's battery while maintaining high recognition accuracy.
Solution Approach 2:
The patent extracts only necessary information (text and locations) from images using efficient on-device models, avoiding the need to process entire large images on resource-constrained wearable devices. This extraction approach maintains recognition accuracy while significantly reducing computational resource consumption and energy usage on the wearable device.
3Loss of time
If smaller images are sent to remote server for screen-text recognition and multimodal large language model processing, then latency and computational resources are reduced, but recognition accuracy decreases
Solution Approach 1:
The patent segments the processing pipeline so that critical scene text recognition occurs on-device with full image quality, ensuring high accuracy. Only the extracted text results are transmitted to the server, combining the benefits of local accurate processing with reduced communication overhead, thereby achieving both low latency and high accuracy.
Solution Approach 2:
The patent performs preliminary scene text recognition and extraction on the wearable device before transmitting data to the server. This preliminary action ensures that accurate text extraction is completed locally using full-resolution images, and only the essential text results are sent remotely, maintaining accuracy while minimizing transmission time and resources.
Data Source
AI summary
System and method including an artificially intelligent assistant are described. An example method includes, in response to initiation of an artificially intelligent assistant at a head-wearable device, capturing contextual data. The contextual data includes one or more of image data, audio data, and/or sensor data. The method includes determining, based on the contextual data, a contextual cue, and providing a portion of the contextual data and a portion of the contextual cue to the artificially intelligent assistant. The method includes determining, by the artificially intelligent assistant, a user request based on the portion of the contextual data and the contextual cue, and receiving a response to the user request. The response is generated using a machine learning model. The method further includes causing the head-wearable device to present the response.