Automated Software Test Quality Assessment via Multi-Source Data Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Software test automation systems face challenges in efficiently determining whether test failures are due to false positives or actual defects, requiring laborious manual steps to analyze test data and identify problematic areas in software programs.

Innovation Solution

A computer-implemented method that uses a test automation system to execute tests, collect and analyze data, including information from external sources like quality management tools and databases, to generate an automated quality assessment report, identifying areas of the software program that pass or fail and suggesting improvements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual analysis steps are performed to determine test failure causes, then measurement precision of defect identification is improved, but productivity deteriorates due to laborious manual steps

Engineering Contradiction:

Improvedefect identification accuracyVSAvoidtest analysis efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The test automation system automatically collects test data, analyzes failure causes, and generates quality assessments without requiring manual intervention. The system determines whether failures are false positives or actual defects through automated analysis of test data, environment information, and historical records, enabling self-service defect identification that maintains precision while improving productivity

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Manual mechanical analysis steps are replaced with automated computational processes. The system uses automated data collection, analysis algorithms, and information processing to substitute the manual mechanical steps that test engineers previously performed, thereby improving productivity while maintaining measurement precision through systematic automated analysis

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If comprehensive test data collection from multiple sources is performed, then quality assessment reliability is improved, but device complexity worsens due to integration of multiple data sources

Engineering Contradiction:

Improvequality assessment reliabilityVSAvoidsystem integration complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The test automation system is designed with multi-functionality to collect and process data from multiple sources including test execution systems, environment information systems, and historical defect databases. This universal capability allows the system to integrate diverse data sources while maintaining reliability through standardized data collection and processing mechanisms

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system employs intermediary components that facilitate data collection and integration from multiple sources. These intermediaries standardize data formats and communication protocols, reducing the complexity of integrating diverse data sources while ensuring comprehensive data collection for reliable quality assessments

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If automated data analysis is implemented, then productivity is improved, but measurement precision deteriorates due to automated determination of false positives versus actual defects

Engineering Contradiction:

Improvetest analysis speedVSAvoidfailure cause determination accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system implements feedback mechanisms where test results, environment information, and historical defect data are continuously analyzed and fed back into the automated analysis process. This feedback loop enables the system to learn from past failures and improve its accuracy in distinguishing false positives from actual defects, maintaining measurement precision while achieving high productivity through automation

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary actions by collecting and analyzing historical defect data, environment information, and test patterns before conducting new tests. This preliminary preparation enables the automated analysis system to make more accurate determinations about failure causes by comparing current test results against established patterns and historical data, thereby maintaining measurement precision while improving productivity

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10102113B2Software test automation systems and methods

Publication Date: 2018.10.16 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Described are a system and method for performing an automated quality assessment on a software program under test. A test automation system executes a test on a software program Data related to the test is automatically collected. The data includes first information determined by the test automation system in response to executing the test. The data further includes second information related to the test and received from a source other than the test automation system. The first information is analyzed. A quality assessment of the software program is generated from the analyzed first information and from the second information.