GUI Testing Device with OCR Text Verification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Developing software applications for client computing devices with varying processing power and display resolutions poses challenges in creating user-friendly graphical user interfaces, especially in mobile applications with limited screen real estate, where testing frameworks are needed to handle multiple languages and user customizations.

Innovation Solution

A GUI testing device that generates multiple versions of a graphical user interface in different languages, performs optical character recognition (OCR) processing on text fields, and compares actual text strings with expected text strings to identify problematic fields, using a testing framework like XCUITest, UIAutomator, or Selenium, and optionally involves image pre-processing and distributed processing across multiple devices.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple language versions of GUI are generated and tested, then the adaptability of the software application is improved, but the time required for testing increases

Engineering Contradiction:

Improvemulti-language supportVSAvoidtesting time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-defining expected text strings for multiple language versions and pre-configuring OCR processing parameters before actual testing begins. This allows the testing framework to quickly compare actual OCR results against pre-prepared expectations without performing complex analysis during the testing phase, thus reducing testing time while maintaining multi-language adaptability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces manual testing mechanisms with automated OCR-based text extraction and comparison systems. Instead of manually verifying each language version, the system uses optical character recognition to automatically extract text from GUI screenshots and compares it against expected values, dramatically reducing the time required to test multiple language versions while maintaining comprehensive adaptability

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If OCR processing is performed on all text fields in all language versions, then the measurement precision of text field accuracy is improved, but the computational resources and time required increase

Engineering Contradiction:

Improvetext field accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system applies local quality by performing OCR processing selectively on specific text fields rather than uniformly on all fields. The testing framework identifies and extracts only those text fields that contain expected text strings, applying OCR processing only to these relevant locations. This localized approach maintains high measurement precision for critical fields while reducing overall processing time and computational resources

Inventive Principle:

Principle #3Local quality

3Reliability

If the testing framework tests all GUI fields across all language versions, then the reliability of GUI functionality is improved, but the device complexity increases

Engineering Contradiction:

ImproveGUI functionalityVSAvoidtesting framework complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The testing framework is segmented into distinct modular components: a screenshot capture module, an OCR processing module, an expected text definition module, and a comparison module. Each component handles a specific aspect of the testing process independently. This segmentation allows the system to maintain high reliability by thoroughly testing all GUI fields across language versions while managing complexity through modular, independently testable units that can be developed and maintained separately

Inventive Principle:

Principle #1Segmentation

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This solution enables efficient identification and flagging of GUI issues across various languages and screen resolutions, improving the usability of software applications by automating the detection of text field problems that may occur during translation or resizing, thus enhancing the user experience.

Implementation Method 1

perform optical character recognition (OCR) processing on the plurality of fields to generate a plurality of actual text strings

Methodology Applied

Scientific EffectOptical character recognition (OCR):

Data Source

PatentUS10990508B2Computing system with GUI testing device and related methods

Publication Date: 2021.04.27 CITRIX SYSTEMS INC

AI summary

A computing system may include a client computing device configured to execute a software application with an associated GUI. The GUI includes fields, and each field is to hold a text string. The computing system may include a GUI testing device in communication with the client computing device and configured to execute a testing framework for interacting with the software application to generate versions of the GUI, each of the versions being in a different language, and defining expected text strings in the fields. The GUI testing device may be configured to extract the fields from the versions of the GUI, perform OCR processing on the fields to generate actual text strings, and compare the actual text strings with the expected text strings.