Autonomous UI Testing via AI Recognition and Dynamic Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for testing website or application user interfaces require human intervention, leading to inaccuracies and increased time and cost due to the complexity of platforms and the need for frequent test program modifications, and often result in false error reporting.
Innovation Solution
A system and method for autonomous user interface testing, comprising a testable action recognizer, a test action generator, an error recognizer, and an external service integrator, which recognizes UI elements, generates test actions, and verifies their accuracy without human intervention, using machine learning and natural language processing to automate the testing process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human testing is used to test platform functions, then testing accuracy can be maintained through human judgment, but testing time and manpower costs increase significantly
Solution Approach 1:
The system enables autonomous testing where the testing system automatically performs test execution, result analysis, and error detection without human intervention. The AI model independently completes the entire testing workflow, eliminating the need for human testers to manually execute test cases and analyze results, thereby reducing testing time while maintaining accuracy through automated judgment algorithms
Solution Approach 2:
The patent replaces human mechanical testing operations with an AI-based autonomous testing system. The system uses machine learning models to automatically recognize UI elements, generate test actions, execute tests, and detect errors, substituting human physical and cognitive operations with automated computational processes that operate faster and more consistently
2Loss of time
If test programs are developed to conduct automated testing, then testing time is reduced, but the test program needs frequent modification when platform changes occur, increasing development complexity
Solution Approach 1:
The autonomous testing system dynamically adapts to platform changes by using AI models that can automatically learn and recognize new UI elements and structures. When the platform changes, the system automatically updates its understanding through continuous learning from new screenshots and UI hierarchies, eliminating the need for manual test program modifications while handling dynamic platform evolution
Solution Approach 2:
The system performs self-updating when platform changes occur. The AI model automatically learns new UI patterns and element structures from updated platform versions without requiring external intervention to modify test programs. The system independently maintains its testing capabilities across platform iterations through continuous autonomous adaptation
3Productivity
If traditional automated testing is implemented, then manpower costs are reduced, but false error reporting occurs due to wrong implementation, reducing reliability
Solution Approach 1:
The system implements feedback mechanisms where test results are continuously analyzed and used to improve future testing. The AI model learns from test outcomes, correcting false positives and negatives over time. The feedback loop enables the system to refine its error detection accuracy by comparing expected versus actual results and adjusting its judgment algorithms accordingly
Solution Approach 2:
The patent replaces rule-based automated testing with AI-driven autonomous testing that uses machine learning models for judgment. Instead of rigid predefined rules that cause false errors, the system uses intelligent models that can understand context and make accurate error determinations, substituting mechanical automation with intelligent automation that reduces false reporting
Data Source
AI summary
Provided is a system and method capable of automatically testing a website or application for an error in a user interface without human intervention. As an example, in a system for testing an error in a user interface of an application or website, a user interface system that includes a testable action recognizer that obtains a screenshot of a screen of the application or website, manages a layout and a test action based on user interface (UI) configuration and text recognition information from the screenshot, and a test action generator that receives the layout, selects a test scenario corresponding the layout, and performs a test action according to the test scenario, and in which the testable action recognizer manages whether or not a test is progressed for each screen layout according to the test scenario, is disclosed.