Test Scenario Selection via Multi-Organization Data Clustering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The high cost and labor intensity of regression testing in software systems, particularly for configuration changes, due to the complexity of identifying relevant tests and the lack of effective methods to sift through vast testing data from multiple organizations.

Innovation Solution

A system and method for generating and selecting test scenario templates based on data from multiple organizations, clustering similar test runs, and weighting connections between configuration elements and clusters to suggest relevant test scenarios for users, thereby optimizing the testing process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If full regression testing is performed to validate configuration changes, then reliability of software validation is improved, but time consumption and cost increase significantly

Engineering Contradiction:

Improvesoftware validation reliabilityVSAvoidtesting time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent extracts and identifies only the subset of test scenarios that are relevant to the specific configuration changes being made. By analyzing dependencies between configuration elements and test scenarios, the system selectively extracts the minimum necessary tests rather than executing the complete regression test suite, thus reducing time while maintaining validation reliability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs partial testing by executing only the necessary portion of the full regression test suite. Based on configuration change analysis and dependency tracking, it determines the exact scope of testing required, performing neither too little nor the complete excessive amount, but the optimal partial amount needed for validation.

Inventive Principle:

Principle #16Partial or excessive action

2Productivity

If testing knowledge is shared across multiple organizations, then productivity of testing process is improved, but complexity of managing and selecting relevant tests increases

Engineering Contradiction:

Improvetesting efficiencyVSAvoidtest selection complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary system that acts as a mediator between multiple organizations' testing knowledge and the specific testing need. This intermediary automatically analyzes configuration changes, queries the shared knowledge base from multiple organizations, and filters/selections the relevant test scenarios, thus enabling knowledge sharing without exposing the complexity of managing diverse test suites to the end user.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback mechanisms where testing results and configuration data are continuously analyzed to refine and improve the selection of relevant tests. The system learns from past testing patterns and configuration-change-test relationships, providing feedback that enhances the accuracy of test selection over time, making the process more efficient while managing complexity.

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If vast amounts of testing data are collected from multiple organizations, then adaptability of testing solutions is improved, but difficulty of sifting and selecting appropriate tests worsens

Engineering Contradiction:

Improvetesting solution adaptabilityVSAvoidtest selection difficulty

Core Design Contradiction:

Adaptability or versatilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent replaces the manual mechanical process of sifting through testing data with an automated computational system. The system uses algorithms to automatically analyze configuration changes, query collected testing data from multiple organizations, and identify relevant test scenarios, thus maintaining adaptability while eliminating the difficulty of manual data sifting.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system creates and maintains copies of testing knowledge and configuration data from multiple organizations in a centralized repository. By having structured copies of this data readily available, the system can quickly query and analyze relevant information without requiring direct access to or manual processing of the original vast data sources, thus improving adaptability while reducing selection difficulty.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS9317412B1Selecting a configuration test based on testing data from different organizations

Publication Date: 2016.04.19 PANAYA

AI summary

Utilizing data collected from users belonging to different organizations to select a test scenario template relevant for testing configuration elements involves the following: identifying runs of test scenarios run by users belonging to different organizations; clustering the runs of test scenarios to clusters that include similar runs; generating connections between the configuration elements and the clusters; weighting the connections based on the number of users that acted according to the connections; receiving certain configuration elements; selecting a certain cluster based on a subset of the certain configuration elements and the connections, where each element in the subset is connected to the certain cluster by a connection, and the sum of weights of connections between elements in the subset and the certain cluster, reaches a threshold; and selecting a test scenario template representing the certain cluster.