Automated Accessibility Fixing for Mobile UI Widgets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing accessibility testing approaches for mobile applications are inadequate as they fail to dynamically capture the behavior of UI elements during execution and require tedious manual intervention to fix inaccessible elements, particularly for Android and iOS systems where features like Talkback and VoiceOver do not automatically make elements like ImageView and UIImageView accessible without explicit settings.
Innovation Solution
Automated techniques are implemented to identify and fix inaccessible UI widgets by using associated label mapping (ALM) and image-based content retrieval (IBCR) to dynamically adjust accessibility properties during runtime, setting properties like android:contentDescription for Android and accessibilityLabel for iOS, ensuring UI elements are accessible to users with sensory impairments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual accessibility testing and fixing is performed, then accessibility issues can be identified and corrected, but the process is tedious and time-consuming
Solution Approach 1:
The system performs self-testing and self-fixing of accessibility issues. The automated accessibility testing system identifies inaccessible UI elements and automatically generates fixes by modifying accessibility properties, eliminating the need for manual intervention in both testing and fixing processes.
Solution Approach 2:
The patent replaces manual mechanical processes (human testers manually checking and fixing accessibility issues) with an automated computational system that uses algorithms to identify inaccessible elements and programmatically apply accessibility fixes.
2Ease of manufacture
If static UI layout file scanning is used for accessibility testing, then initial accessibility issues can be detected, but dynamic behavior of elements during execution is not captured
Solution Approach 1:
The system transitions from static accessibility testing (analyzing UI layout files) to dynamic accessibility testing (monitoring UI elements during actual application execution). This allows the system to capture the real-time behavior and state changes of UI elements, providing comprehensive accessibility coverage.
Solution Approach 2:
The system performs preliminary static analysis of UI layout files to identify potential accessibility issues before execution, then complements this with dynamic monitoring during runtime to capture actual behavior, combining both approaches for thorough testing.
3Productivity
If accessibility properties are not explicitly set for certain UI elements, then development is simpler and faster, but elements like ImageView and UIImageView remain inaccessible to users with sensory impairments
Solution Approach 1:
The system implements automated feedback mechanisms that monitor UI elements during execution and identify those lacking proper accessibility properties. The system then provides feedback to the development process by automatically generating and applying fixes for missing accessibility attributes.
Solution Approach 2:
The automated accessibility testing system acts as an intermediary between the UI development process and accessibility compliance requirements. It bridges the gap by automatically detecting elements that need accessibility properties and applying the necessary fixes without requiring developers to manually set each property.
Data Source
AI summary
Methods, systems, and articles of manufacture for automatically fixing inaccessible widgets during mobile application execution are provided herein. A method includes identifying an image element from a set of one or more user interface elements of an application that are inaccessible to a given set of users; generating a text description associated with the image element; extracting a portion of the text description related to an accessibility property of the image element; and adjusting the accessibility property of the image element to render the image element accessible to the given set of users.