Handwriting Gesture Recognition for Screen Reader Accessibility

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing accessibility interfaces on devices with touch-sensitive surfaces are cumbersome and inefficient for users with impaired vision, leading to a significant cognitive burden and increased energy consumption in battery-operated devices.

Innovation Solution

The integration of handwriting gesture recognition with a screen reader on touch-sensitive surfaces allows users to unlock devices, select applications, navigate web pages, and input characters through single finger gestures that correspond to character paths, reducing the need for visual interaction and enhancing accessibility.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If traditional touch-based navigation methods are used for accessibility interfaces, then basic functionality is provided, but the interface becomes cumbersome and inefficient for users with impaired vision

Engineering Contradiction:

Improveease of operationVSAvoidinterface complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent replaces traditional mechanical touch-based navigation with voice recognition technology. Users speak commands to navigate and interact with the interface, substituting the mechanical finger-tapping system with an acoustic input system. This reduces the cognitive burden and physical effort required for users with impaired vision while maintaining full functionality.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces voice commands as an intermediary between the user and the graphical user interface. Instead of directly manipulating visual elements, users speak natural language commands that are processed by the system, which then translates them into appropriate interface actions. This intermediary layer simplifies the interaction model for users with impaired vision.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If existing accessibility methods are used, then users can interact with the device, but it takes longer than necessary, wasting energy

Engineering Contradiction:

Improveinteraction speedVSAvoidenergy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent implements periodic voice command processing where the system continuously listens for voice inputs at optimized intervals rather than requiring continuous visual attention and manual interaction. This periodic acoustic sampling enables faster task completion while reducing overall energy consumption by keeping the display and processor in lower-power states between voice interactions.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

By replacing slow, deliberate touch-based navigation with faster voice recognition processing, the system achieves quicker interaction speeds. Voice commands can be processed and executed more rapidly than manual navigation through multiple screen layers, thereby reducing the time tasks take to complete and lowering energy consumption accordingly.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If touch-sensitive surfaces require visual interaction for navigation, then precise control is achieved, but cognitive burden increases significantly for users with impaired vision

Engineering Contradiction:

Improvecontrol precisionVSAvoidcognitive burden

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent substitutes visual-manual control with acoustic-vocal control. Voice recognition maintains precision through voice command interpretation and confirmation protocols, while eliminating the need for users to visually locate and precisely tap interface elements. This reduces cognitive burden by allowing users to interact through familiar speech patterns rather than learning complex touch-gesture vocabularies.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system implements immediate audio feedback to confirm voice command recognition and interpretation. Users hear their commands repeated or confirmed by the system, providing real-time verification without requiring visual feedback. This feedback loop maintains control precision while keeping the interaction model simple and intuitive for users with impaired vision.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10013162B2Device, method, and graphical user interface for integrating recognition of handwriting gestures with a screen reader

Publication Date: 2018.07.03 APPLE INC

AI summary

While an electronic device with a display and a touch-sensitive surface is in a screen reader accessibility mode, the device displays a character input area and a keyboard, the keyboard including a plurality of key icons. The device detects a sequence of one or more gestures on the touch-sensitive surface that correspond to one or more characters. A respective gesture of the one or more gestures that corresponds to a respective character is a single finger gesture that moves across the touch-sensitive surface along a respective path that corresponds to the respective character. The respective path traverses one or more locations on the touch-sensitive surface that correspond to one or more key icons of the plurality of key icons without activating the one or more key icons. In response to detecting the respective gesture, the device enters the corresponding respective character in the character input area of the display.