Application Fault Detection via Screenshot Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Host systems embedding client applications lack visibility into the native code of client devices, making it difficult to detect issues like a hung loading screen, leading to user frustration and challenges in identifying and correcting problems.
Innovation Solution
A debug tool captures periodic screenshots of the application and applies a classifier to determine its state, using machine learning models trained on labeled data to differentiate between normal and fault states, prompting users with alerts when issues are detected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the host system does not have access to the native code of the client application, then the client application can operate with full native performance and security, but the host system cannot detect application faults or problems
Solution Approach 1:
The system creates visual copies (screenshots) of the application's display state and feeds them to a machine learning classifier to infer the application's internal state. This allows fault detection without accessing the actual native code, resolving the contradiction between detection capability and code accessibility requirements.
Solution Approach 2:
The patent introduces an intermediary layer consisting of screenshots and machine learning classifiers that mediate between the host system and the native application code. This intermediary enables indirect observation of application state without requiring direct code access, maintaining both reliability and security.
2Productivity
If the host system cannot detect when problems occur in the client application, then the application can maintain full native execution without restrictions, but the host system cannot identify the source of problems or provide timely feedback
Solution Approach 1:
The system implements feedback by continuously capturing screenshots, classifying them through machine learning models, and providing real-time information about application state to the host system. This enables timely problem identification and feedback without compromising native execution performance.
Solution Approach 2:
By creating visual copies of the application display and analyzing them through machine learning classifiers, the system gains information about application state without interfering with the actual execution. This copying approach enables problem identification while maintaining full native performance.
3Ease of repair
If the host system lacks visibility into the source code, then the application enjoys maximum performance and security, but the host system cannot trace the root cause of faults even when they occur
Solution Approach 1:
The patent uses screenshots and machine learning classifiers as intermediaries to bridge the information gap between the host system and the application's internal state. This intermediary layer enables fault analysis without requiring source code access, maintaining security while improving repair capabilities.
Solution Approach 2:
The continuous feedback loop of screenshot capture, machine learning classification, and state reporting provides the host system with information about application faults and their temporal context. This feedback mechanism enables root cause analysis by tracking when and under what conditions faults occur, even without code visibility.
Data Source
AI summary
Systems and methods are disclosed herein for using screen shots of an application to evaluate whether the application is in a fault state. In an embodiment, a processor captures screen shots of a screen of an application during operation of the application. The processor applies a classifier to the captured screen shots, and determines, from output of the classifier, a state of the application. The processor evaluates whether the determined state is a fault state of the application. Responsive to determining that the determined state is a fault state of the application, the processor prompts a user with an alert indicating the fault state and a corresponding point during the operation that exhibited the fault state.