Wearable AI Assistant Using On-Device Scene Text Recognition for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for screen-text recognition and multimodal large language models on wearable devices require sending full-resolution images to remote servers, leading to increased latency, computational resource usage, and decreased accuracy.
Innovation Solution
Implementing an artificially intelligent assistant on wearable devices with on-device scene text recognition models and lightweight multimodal large language models to process contextual data locally, reducing latency and computational requirements while maintaining high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-resolution images are sent to remote servers for screen-text recognition and multimodal large language model processing, then processing capability and accuracy are improved, but latency increases and computational resources are consumed
Solution Approach 1:
The processing pipeline is segmented into two stages: (1) a lightweight on-device scene text recognition model processes full-resolution images locally to extract text content and locations, and (2) a lightweight multimodal large language model processes only the extracted text information. This segmentation allows the system to maintain high accuracy for screen-text recognition while significantly reducing latency by avoiding the need to send full-resolution images to remote servers for all processing steps.
Solution Approach 2:
The patent extracts only the essential text information and its locations from the full-resolution images using on-device processing, rather than transmitting the complete image data. This extraction approach preserves the measurement precision needed for accurate text recognition while minimizing the data transfer requirements and associated latency.
2Measurement precision
If full-resolution images are sent to remote servers, then processing capability is improved, but computational resources and energy consumption increase
Solution Approach 1:
The computational workload is segmented between on-device processing and cloud processing. The on-device scene text recognition model handles the computationally intensive task of processing full-resolution images locally, extracting only the necessary text information. The subsequent multimodal large language model processing requires minimal computational resources since it operates on extracted text rather than full images. This segmentation maintains measurement precision while significantly reducing overall energy consumption.
Solution Approach 2:
The wearable device performs self-service processing by executing the scene text recognition model locally on its own hardware resources. This eliminates the need to rely solely on remote servers for image processing, reducing the computational burden on cloud infrastructure and lowering overall energy consumption while maintaining the accuracy needed for effective text recognition.
3Loss of time
If smaller images are sent to remote servers, then latency is decreased, but accuracy decreases
Solution Approach 1:
The processing is segmented such that full-resolution images are processed locally on the wearable device using an optimized scene text recognition model, while only the extracted text content and locations are transmitted to the cloud for further processing. This approach maintains measurement precision by preserving full-resolution image quality for local analysis while achieving low latency by minimizing the amount of data transmitted to remote servers.
Solution Approach 2:
The patent applies local quality processing by executing the scene text recognition model directly on the wearable device's local hardware, utilizing the device's computational capabilities to process full-resolution images without relying on remote servers. This local processing approach maintains high measurement precision while enabling fast response times, as the processing occurs immediately on the device without network transmission delays.
4Loss of time
If on-device scene text recognition models are used, then latency is reduced, but device complexity increases
Solution Approach 1:
The patent applies parameter changes by optimizing the scene text recognition model for on-device execution, adjusting parameters such as model size, precision, and computational architecture to fit within the wearable device's hardware constraints. This optimization enables the device to process full-resolution images locally with reduced latency while managing computational requirements through careful model parameter selection and hardware-aware design.
Data Source
AI summary
System and method including an artificially intelligent assistant are described. An example method includes, in response to a user input initiating an artificially intelligent (AI) assistant, capturing contextual data including one or more of image data and audio data. The method includes generating, based on the contextual data, user query data including a user query and a portion of the contextual data. The method includes determining, using an AI assistant model that receives the user query data, a user prompt based on, at least the user query and the portion of the contextual data, and generating, by the AI assistant model, a response to the user prompt. The method further includes causing presentation of the response to the user prompt at a head-wearable device.