AI Model Training Through Accessibility APIs for Cross-Application Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI models are limited in scope, accuracy, and efficiency due to their inability to adapt to changing contexts and user interactions across multiple applications, as they are trained using backend data without considering user device context and can only record actions within specific application environments.
Innovation Solution
An electronic device uses an accessibility API to gather user interface data, trains an AI model based on this data to identify user actions, and determines recommended actions across varying contexts, enabling adaptive task performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional AI models are trained using backend data from troubleshooting trees, then the training process is simple and data organization is straightforward, but the scope, accuracy, and efficiency of tasks are limited
Solution Approach 1:
The patent introduces an accessibility API as an intermediary mechanism to bridge the gap between the AI model and the user interface. This API serves as a mediator that collects accessibility data from UI elements and events, enabling the model to understand user interactions without directly accessing complex application internals. The accessibility API standardizes the data collection process, making it easier to gather comprehensive interaction data across different applications while maintaining system simplicity.
Solution Approach 2:
The patent implements a universal data collection approach using the accessibility API that works across multiple applications and UI types. Instead of building separate data collection systems for each application, the system uses a single universal interface (accessibility API) to gather data from various sources including UI elements, events, and user interactions. This multi-functional approach enables the AI model to learn from diverse contexts without requiring application-specific training pipelines.
2Adaptability or versatility
If action recording is done per application basis, then the recording mechanism is simple and application-specific, but actions cannot be recorded across multiple different applications
Solution Approach 1:
The accessibility API provides a universal recording mechanism that functions across all applications regardless of their specific implementations. The system collects accessibility data from UI elements and events in a standardized manner that works consistently across different application types. This universal approach enables cross-application action recording without requiring separate recording systems for each application, as the same accessibility API interface handles all data collection needs.
Solution Approach 2:
The system creates a standardized representation (copy) of user interactions through the accessibility API. Instead of directly manipulating complex application-specific event streams, the system accesses standardized accessibility data structures that represent UI elements and events in a uniform format. This copying approach simplifies the recording mechanism by working with standardized data representations rather than application-specific internals, enabling cross-application functionality.
3Adaptability or versatility
If recorded actions are played back with same window elements active, then the playback is accurate for that specific context, but the actions cannot adapt to changes in context such as different dates, times, or applications
Solution Approach 1:
The system implements feedback mechanisms by continuously collecting accessibility data from the current UI state and user interactions. The AI model processes this feedback data to understand the current context and adapts its responses accordingly. The system monitors UI element states, user actions, and temporal information, using this feedback to dynamically adjust task performance rather than relying on static pre-recorded actions. This feedback loop enables the model to adapt to changing contexts while maintaining accuracy.
Solution Approach 2:
The system performs preliminary data collection and analysis by gathering accessibility data from UI elements and events before executing tasks. The AI model analyzes this preliminary information to understand the current context, including active applications, visible UI elements, and user interaction patterns. This preliminary action enables the model to prepare appropriate responses that adapt to the specific context rather than executing fixed pre-recorded actions, ensuring contextual accuracy before task execution.
Data Source
AI summary
Examples provide an electronic device including at least one electronic processor configured to receive, via an accessibility application programming interface (“API”), first accessibility data associated with a first user interface (“UI”) displayed by a user device and including information identifying one or more UI elements and/or UI events in the first UI; identify, based on the first accessibility data, a first action performed by a user of the user device and defining a first set of operations; train, based on the first accessibility data and the first set of operations, an artificial intelligence (“AI”) model; receive, via the accessibility API, second accessibility data associated with a second UI displayed by the user device; determine, based on the second accessibility data and the AI model, a recommended action defining a second set of operations; and output a command to perform the recommended action on the user device.