UI Test Script Generation From Video-Based User Interaction Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional application testing methods require skilled engineers for manual test authoring and recording, which can be slow and inaccurate, and may not capture real user interactions effectively.

Innovation Solution

Utilize machine learning algorithms to analyze video frames of user interactions with an application under test, identifying objects and actions to generate automated test scripts without instrumentation, using AI object detection engines to determine user intentions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual test design is used, then test accuracy can be maintained, but the process becomes slow and requires highly skilled engineers

Engineering Contradiction:

Improvetest accuracyVSAvoidtest creation speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent uses video recording to capture user interactions and automatically converts them into test scripts. This copying approach allows real user behavior to be replicated and analyzed, generating accurate tests without requiring manual scripting while significantly improving productivity.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces the mechanical process of manual test design with an automated system using machine learning and computer vision. The system automatically analyzes video frames, detects user actions, and generates test scripts, eliminating the need for skilled manual intervention while maintaining or improving test accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If recording capability is activated to capture user interactions, then comprehensive test data can be obtained, but the hooking process impacts application performance and may miss actions

Engineering Contradiction:

Improvetest data completenessVSAvoidapplication performance impact

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

The patent introduces video recording as an intermediary layer between the user and the application. Instead of hooking into the application directly, the system captures the visual output and user interactions through video, then analyzes this external representation to generate tests. This approach eliminates performance impact while maintaining comprehensive test data capture.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system creates a visual copy of user interactions through video recording and analyzes this copy to generate test scripts. This copying mechanism allows comprehensive capture of user behavior without interfering with the original application execution, thus avoiding performance degradation and ensuring complete test data collection.

Inventive Principle:

Principle #26Copying

3Measurement precision

If inspection tools are used to identify objects, then accurate object detection can be achieved, but the process requires skilled technical capabilities and deep application understanding

Engineering Contradiction:

Improveobject detection accuracyVSAvoidtechnical skill requirement

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces manual inspection tools with automated computer vision and machine learning algorithms. The system automatically detects objects, identifies their properties, and extracts relevant information from video frames without requiring human expertise in inspection techniques or deep application knowledge, thereby reducing device complexity while maintaining detection accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system performs self-service by automatically analyzing video content to identify objects and generate test scripts. The machine learning model autonomously detects UI elements, determines their properties, and creates appropriate test cases without human intervention, eliminating the need for skilled operators while maintaining high accuracy.

Inventive Principle:

Principle #25Self-service

4Productivity

If test data is created quickly for efficient processing, then productivity improves, but the data may not accurately represent real user data

Engineering Contradiction:

Improvedata processing efficiencyVSAvoiddata representation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent uses video recording to capture actual user interactions, creating an authentic copy of real user behavior. This approach generates test data that naturally represents how users actually interact with the application, maintaining high data accuracy while achieving productivity through automated analysis of the recorded video content.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20260037411A1Generation of user interface tests from a video

Publication Date: 2026.02.05 MICRO FOCUS LLC

AI summary

Software applications require testing to ensure they operate as intended. Testing of an application under test (AUT) can be automated by executing a test script, but developing test scripts requires a skilled engineer to modify the AUT and/or instrumentation to be applied to a test platform to record user actions used to develop a test script. By obtaining a video of a user interacting with an AUT on an uninstructed test platform, an artificial intelligence (AI) may determine the actions the user takes and the objects to which those actions are directed. Additionally, the user's intentions may be determined so that a test script may be developed or fine-tuned to be successful even if the AUT has visually changed since the video was obtained.