Head-Mounted Text Summary via OCR and LLM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing head-mounted devices with video passthrough or see-through displays do not effectively provide summaries of text in a user's physical environment, limiting the ability to quickly absorb information from text-based sources.
Innovation Solution
An electronic device equipped with cameras, displays, processors, and memory that captures images of text in the physical environment, performs optical character recognition, and sends the text information to a trained model to generate a summary, which is then presented on the device's displays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If users read text directly through video passthrough or see-through displays, then they can view the original text, but it takes time to absorb information and reading efficiency is low
Solution Approach 1:
The system captures text from the physical environment using cameras, performs optical character recognition to create a digital copy of the text, generates a summary of this copied text, and displays it on the see-through display. This copying approach allows users to quickly access summarized information without reading the entire original text, thereby improving reading efficiency while maintaining access to the source material.
2Loss of information
If a text summary is displayed on the see-through display, then information absorption is enhanced, but the original text may be obscured
Solution Approach 1:
The system positions the text summary in the virtual space at a different spatial dimension relative to the original text. The summary can be displayed adjacent to, above, or in proximity to the captured text region, allowing users to access both the original text and its summary simultaneously without one obscuring the other. This spatial separation resolves the conflict between information absorption and text visibility.
3Productivity
If the device processes and displays text summaries in real-time, then reading efficiency improves, but device complexity increases
Solution Approach 1:
The system performs optical character recognition and text capture in advance before the user needs to read. By pre-processing the text extraction from images and preparing it for summary generation, the system reduces the computational burden during the actual reading interaction. This preliminary action approach maintains improved reading efficiency while managing device complexity through staged processing.
Data Source
AI summary
A head-mounted device may include one or more cameras that detect text in a physical environment surrounding the head-mounted device. The head-mounted device may send information regarding the text in the physical environment, contextual information, response length parameters, and/or user questions associated with the text in the physical environment to a trained model. The trained model may be a large language model. The head-mounted device may receive a text summary from the trained model that is based on the information regarding the text, contextual information, response length parameters, and user questions. The head-mounted device may present the text summary on one or more displays.