Screen Reader Markup Probe for Visual Control Access

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users with visual impairments face limited access to visual controls in applications like visual integrated development environments (IDEs) due to screen readers' inability to navigate and activate all screen parts, making it difficult for them to create and modify graphical user interface elements.

Innovation Solution

A screen reader system that includes a markup code probe to locate visual controls, a user menu factory to create accessible menu items, a visual control extraction engine to extract properties, and a menu controller to simulate actions at the screen location corresponding to visual controls, enabling visually impaired users to access and interact with visual controls through arrow key input or other input methods.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If screen readers use traditional navigation methods (arrow keys, tab key), then users can navigate accessible elements, but visually impaired users cannot reach or access visual controls in visual IDEs

Engineering Contradiction:

ImproveAccessibility of visual controlsVSAvoidScreen reader navigation capability

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary system consisting of a visual control extraction engine, menu item factory, and menu controller that bridges the gap between screen reader users and visual controls. This intermediary extracts control information from the visual interface, creates accessible menu items, and simulates user actions to enable navigation and interaction with visual controls that would otherwise be inaccessible.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the traditional mechanical navigation system (arrow keys, tab key) with a new system that uses markup code analysis and simulated mouse actions. The visual control extraction engine analyzes the underlying markup code to locate controls, and the menu controller simulates mouse clicks and hovers to activate controls, substituting keyboard-based navigation with a code-analysis-based approach.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If visual IDEs provide full visual interface access, then sighted users can interact with all controls, but visually impaired users cannot access or generate visual controls

Engineering Contradiction:

ImproveAccessibility for visually impaired usersVSAvoidAbility to navigate visual development canvas

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent creates a universal access mechanism that works across different visual controls in the IDE. The visual control extraction engine identifies controls based on their markup code properties, and the menu item factory generates appropriate menu items for each control type. This universal approach enables visually impaired users to access various visual controls (message flow controls, connection controls, etc.) through a consistent interaction pattern.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent performs preliminary analysis of the markup code to extract visual control information before the user needs to interact with the controls. The visual control extraction engine proactively identifies controls, their locations, and their actions, preparing accessible menu items in advance. This preliminary action eliminates the need for users to discover controls through trial and error, providing immediate accessibility.

Inventive Principle:

Principle #10Preliminary action

3Loss of information

If screen readers analyze markup code to locate visual controls, then accessibility is improved, but the system complexity increases

Engineering Contradiction:

ImproveInformation about visual control locationsVSAvoidMarkup code analysis capability

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent extracts essential information from the markup code through the visual control extraction engine. It identifies visual controls, their screen locations, and their associated actions by analyzing the markup structure. This extraction process separates the necessary control information from the overall visual interface, making it accessible to screen readers without requiring the entire visual interface to be re-engineered.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10452231B2Usability improvements for visual interfaces

Publication Date: 2019.10.22 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

One or more aspects relate to a screen reader. Markup code for a visual interface containing at least one visual control is located. A screen reader menu for the visual interface is created. A visual control name and screen location, both corresponding to the at least one visual control, are extracted from the markup code. A menu item is generated in the screen reader menu corresponding to the at least one visual control. The menu item is set to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.