Runtime NFR Analysis for Adaptive Test Case Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In DevOps environments, the impact of updates on application performance in production is unknown during development, leading to issues like failure events and sub-par runtime performance, which conventional methods struggle to predict and mitigate effectively.

Innovation Solution

A framework that analyzes application details across timestamp intervals, identifies Non-Functional Requirement (NFR) issues causing failures, and uses a weightage algorithm to create test cases that address these issues in subsequent operational cycles.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional development processes are used, then development isolation is maintained, but the impact of updates on production performance cannot be predicted

Engineering Contradiction:

Improveprediction accuracy of update impactVSAvoidinformation about downstream effects

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The patent creates virtual copies of the production environment and pre-populates them with historical runtime data before testing updates. This preliminary setup allows the impact of updates to be predicted in advance without affecting the actual production environment, resolving the contradiction between maintaining development isolation and predicting update impact.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent creates virtual copies (testbeds) of the production environment that replicate runtime conditions, application details, and operational data. These copies enable testing and prediction of update impacts without direct access to production systems, thus maintaining isolation while gaining predictive capability.

Inventive Principle:

Principle #26Copying

2Reliability

If runtime monitoring is implemented, then NFR issues can be identified, but test case coverage gaps may exist

Engineering Contradiction:

Improvedetection accuracy of NFR issuesVSAvoidtest case coverage completeness

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements a feedback mechanism where runtime monitoring data from production environments is continuously collected, analyzed, and used to automatically generate new test cases. This closed-loop system ensures that test case coverage evolves to match actual runtime conditions, improving NFR issue detection while adapting to coverage gaps systematically.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent creates a dynamic test case generation system that adapts to changing runtime conditions. Test cases are not static but are continuously updated based on monitored NFR issues, ensuring test coverage remains aligned with actual operational requirements and automatically fills coverage gaps as they emerge.

Inventive Principle:

Principle #15Dynamics

3Reliability

If test cases are created for every NFR issue, then mitigation coverage improves, but development cycle time increases

Engineering Contradiction:

Improvemitigation coverage of NFR issuesVSAvoiddevelopment cycle duration

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies partial action by prioritizing test case creation for the most critical NFR issues identified through weightage algorithms. Instead of creating test cases for all potential issues equally, the system focuses on high-impact areas first, achieving sufficient mitigation coverage while avoiding the time cost of exhaustive testing for every possible scenario.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent uses weightage algorithms that dynamically adjust parameters based on the severity and frequency of NFR issues. This allows the system to adaptively determine which test cases provide the most value, optimizing the balance between mitigation coverage and development time by changing the priority parameters of different test scenarios.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12430228B2Determining non-functional requirements (NFR) from a run time environment and incorporating into a development cycle

Publication Date: 2025.09.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A computer-implemented method, according to one embodiment, includes analyzing application details associated with a plurality of different timestamp intervals, in response to a determination, from results of analyzing runtime information associated with an application failure event and/or sub-par runtime performance that occurs during a first operational run cycle of an application, that the failure event and/or sub-par runtime performance is caused by a Non-Functional Requirement (NFR) issue. In response to a determination that a first of the timestamp intervals falls within a timestamp associated with the application failure event and/or sub-par runtime performance, first pattern information about the application details associated with the first timestamp interval is collected. The method further includes applying the first pattern information to a weightage algorithm. In response to a determination that test coverage gaps exist in a predetermined collection of test cases, test cases are created based on the first pattern information.