Computer Vision Android Test Reproduction Across Device Fragmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Record and Replay (R&R) tools for android testing face challenges in overcoming device fragmentation due to changes in application versions and UI, requiring metadata that often changes, and lack portability and efficient device configuration.
Innovation Solution
A method using Computer Vision techniques to interpret test video steps by identifying touch coordinates and classifying widgets, generating human-readable test scripts that are portable across devices, using Screen2Text and V2S techniques, and heuristic rules to translate touch coordinates into android actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If metadata extraction methods are used to interpret test video steps, then test automation is achieved, but maintenance frequency increases due to metadata changes between application versions and devices
Solution Approach 1:
The patent uses computer vision to create visual copies of the UI interface elements and their positions from the test video. Instead of relying on metadata that changes with each application version, the system captures and processes visual frames to extract touch coordinates and widget descriptions, creating a visual representation that remains valid across different versions and devices.
Solution Approach 2:
The patent replaces the metadata-based interpretation mechanism with a computer vision-based mechanism. By using image processing techniques to analyze test video frames, extract touch coordinates, and identify widget descriptions, the system substitutes the fragile metadata extraction approach with a more robust visual analysis approach that does not depend on application-specific metadata structures.
2Measurement precision
If coordinate-sensitive touch recognition models are trained on each device model, then accurate test reproduction is achieved, but the process becomes time-consuming and impractical
Solution Approach 1:
The patent creates a universal test script representation that can be executed across multiple device models without requiring separate training for each device. By extracting touch coordinates and widget descriptions from video frames and representing them in a device-agnostic format, the system enables a single test script to be portable and reproducible on different Android devices with varying screen resolutions and densities.
Solution Approach 2:
The patent transforms the touch coordinate system from device-specific coordinates to normalized coordinates that can be adapted to different screen resolutions and densities. By processing video frames to extract relative positions and scaling factors, the system adjusts coordinates dynamically based on the target device's characteristics, eliminating the need for retraining touch recognition models on each device model.
3Reliability
If multiple device pre-configurations are required for recording test videos, then comprehensive test coverage is achieved, but ease of operation decreases for programmers and testers
Solution Approach 1:
The patent captures the complete visual state of the device screen during test execution by recording video frames. This visual copying approach eliminates the need for complex pre-configurations because the test script is derived directly from the visual evidence in the video, rather than requiring metadata annotations or device-specific setup. The video recording itself becomes the source truth for generating test steps.
Solution Approach 2:
The system performs automatic analysis of the recorded test video to extract touch coordinates, identify widgets, and generate test scripts without requiring manual configuration or intervention. The computer vision pipeline automatically processes the video frames, detects user interactions, and produces executable test cases, making the process self-service and reducing the operational burden on programmers and testers.
Data Source
AI summary
A method for producing android device test reproducible on any android device, comprising: receiving a previously recorded android test video file and processing the video file to extract video frames; searching for touch coordinates in the video frames; identifying the touch coordinates in the video frames and generating touch coordinate groups. The method includes translating touch coordinate groups into android actions using heuristic rules; recognizing and classifying widgets in the android actions video frames; generating a description for each of the recognized and classified widgets; associating a widget with each android action; generating a user-readable test step text file and a test step file with detailed information for each step, and iteratively founding the most similar widget on the device under test screen when compared with the human-readable described step at each timestamp at execution time.