Multi-Stage Image Querying With Attention-Based Audio Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Images are inaccessible to individuals with visual impairments, particularly those who are blind or have low vision, due to their inherently visual nature.
Innovation Solution
Utilizing computational models, such as neural networks, to analyze images and provide non-visual access by determining feature information and attention information, refining query text, and generating audio outputs based on image and query correlations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If images are presented in their original visual form, then image information is preserved accurately, but accessibility for visually impaired individuals is lost
Solution Approach 1:
The patent introduces computational models including convolutional computational models and network computational models as intermediaries between the image and the user. These models process the visual image data and transform it into accessible formats such as text descriptions and audio outputs, enabling visually impaired users to access image content without direct visual perception
Solution Approach 2:
The patent replaces the mechanical/visual system of direct image perception with a computational processing system. Instead of relying on visual mechanisms, the system uses computational models to analyze image features, determine attention information, and generate accessible representations through text-to-speech or other non-visual output mechanisms
2Ease of operation
If computational models process images to provide accessibility, then accessibility is improved, but processing complexity increases
Solution Approach 1:
The patent segments the image processing task into distinct computational stages: a convolutional computational model stage for extracting image features, and network computational model stages for processing query text and determining attention information. This segmentation allows each component to specialize in specific functions, managing overall system complexity through modular architecture
Solution Approach 2:
The patent transitions the problem from the visual dimension to the computational and auditory dimensions. By processing images through computational models and outputting results in text or audio form, the system operates in different dimensional spaces, reducing the complexity burden on any single component while maintaining comprehensive image analysis capability
3Measurement precision
If multi-stage computational processing is applied, then accuracy of image analysis is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by using the convolutional computational model to extract and process image features before the network computational models perform their analysis. This preliminary feature extraction prepares the data in advance, allowing subsequent stages to focus on specific aspects like query matching and attention determination, thereby improving overall accuracy without excessive time penalty
Solution Approach 2:
The patent maintains continuity of useful action through the multi-stage processing pipeline where each stage builds upon the previous stage's output. The convolutional model continuously provides feature information to the network models, which continuously refine the analysis through attention mechanisms, ensuring that processing is efficient and avoids redundant operations while maintaining high accuracy
Data Source
AI summary
In some examples, a computing device refines feature information of query text. The device repeatedly determines attention information based at least in part on feature information of the image and the feature information of the query text, and modifies the feature information of the query text based at least in part on the attention information. The device selects at least one of a predetermined plurality of outputs based at least in part on the refined feature information of the query text. In some examples, the device operates a convolutional computational model to determine feature information of the image. The device network computational models (NCMs) to determine feature information of the query and to determine attention information based at least in part on the feature information of the image and the feature information of the query. Examples include a microphone to detect audio corresponding to the query text.