Software Testing Assurance via Inconsistent Treatment Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing automated testing methods often fail to ensure comprehensive coverage of software code updates, leading to potential bugs and security weaknesses due to incomplete test coverage, as they may not account for all execution scenarios and paths within complex software applications.

Innovation Solution

The implementation of software testing assurance technology that includes an assurance code which determines the extent of test coverage for code updates by comparing the current test-set with the previous test-set, flagging any lack of assurance when the test-set does not differ, thereby prompting developers to update tests to cover new or changed code scenarios.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If automated testing is used to test programs more thoroughly, then testing consistency is improved, but testing coverage remains incomplete due to computational resource limitations

Engineering Contradiction:

Improvetesting consistencyVSAvoidtesting coverage

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system performs preliminary analysis of code changes to identify which code paths are affected by the update. By pre-determining the impact scope before executing tests, the system can focus computational resources on relevant test cases, thereby improving coverage without requiring exhaustive testing of all possible paths.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of applying uniform testing to all code, the system applies differential testing quality based on code change impact. Code paths that are modified or affected by the update receive enhanced testing attention, while unchanged paths rely on existing test coverage. This localized approach optimizes resource allocation and improves effective coverage.

Inventive Principle:

Principle #3Local quality

2Difficulty of detecting and measuring

If code coverage metrics are used to measure testing completeness, then measurement simplicity is improved, but measurement accuracy deteriorates because code coverage does not equal testing coverage

Engineering Contradiction:

Improvemeasurement simplicityVSAvoidtesting coverage accuracy

Core Design Contradiction:

Difficulty of detecting and measuringVSMeasurement precision

Solution Approach 1:

The system introduces an intermediary analysis layer between code changes and test execution. This intermediary component analyzes the semantic impact of code changes, determines affected code paths, and maps them to relevant test cases. This intermediary step transforms the simple but inaccurate code coverage metric into a more accurate testing coverage measurement by considering the actual impact of changes.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback mechanisms that compare expected test coverage (based on code change impact analysis) with actual test coverage. This feedback loop identifies gaps where code paths should be tested but are not, allowing continuous improvement of testing accuracy while maintaining measurement simplicity through automated tracking.

Inventive Principle:

Principle #23Feedback

3Quantity of substance

If comprehensive testing of all code paths is attempted, then testing coverage is improved, but computational resource requirements and testing time increase exponentially

Engineering Contradiction:

Improvetesting coverageVSAvoidtesting time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The system segments the testing process into distinct phases: code change impact analysis, affected path identification, relevant test case selection, and targeted execution. This segmentation allows the system to focus computational resources on testing only the portions of code that are actually affected by changes, rather than re-testing entire codebases, thereby reducing testing time while maintaining coverage.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies partial action by testing only the necessary subset of code paths affected by changes rather than all possible paths. By using static analysis and impact propagation to identify minimal sufficient test sets, the system achieves adequate testing coverage without the exponential resource cost of comprehensive testing.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentEP3857382B1Software testing assurance through inconsistent treatment detection

Publication Date: 2023.07.05 MICROSOFT TECHNOLOGY LICENSING LLC

Figure 1~2
Figure 3~4
Figure 5~6(G)

AI summary

Tools and techniques are described to detect possible holes in automated testing of software under development. Full line coverage by tests does not necessarily indicate actual coverage of execution scenarios, e.g., condition coverage, decision coverage, and other kinds of execution scenario coverage may be lacking even when all source code statements are nominally covered. When source code changes are submitted, and corresponding test-sets remain unchanged, users are notified that adequate testing is not assured by the current test-set. Testing assurance code in a development tool chain may flag a pull request, test-set, or source code submission to indicate a lack of testing assurance. In some cases, an assurance-enhanced tool may require that new or different tests be provided with updated source code as a prerequisite for that source code to be accepted for inclusion in a repository or a build, for example.