Predicting Test Case Status via Changeset Metadata Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing is a costly and resource-intensive process, as existing statistical or machine learning approaches typically assess test cases in aggregate, failing to provide a refined evaluation of individual changesets, which limits resource optimization.
Innovation Solution
The approach assesses the likely impact of individual changesets by condensing changeset and affected files into metadata, using supervised learning to correlate changesets with test case transitions, enabling a more refined evaluation and flexible classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If aggregate assessment of test cases is used, then resource consumption is reduced through statistical approaches, but measurement precision of individual changeset impact deteriorates
Solution Approach 1:
The patent segments the aggregate test case assessment into individual changeset-level assessments. By creating separate predictive models for each changeset rather than treating all test cases as a single aggregate group, the system achieves precise measurement of individual changeset impact while maintaining resource efficiency through targeted prediction.
Solution Approach 2:
The patent applies local quality by tailoring the assessment approach to individual changesets. Each changeset receives customized metadata extraction and model application based on its specific characteristics (files modified, lines changed, components affected), enabling precise local assessment rather than uniform aggregate treatment.
2Measurement precision
If individual changeset assessment is performed, then measurement precision of changeset impact is improved, but device complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-processing changesets into standardized metadata formats before assessment. By extracting relevant features (files modified, lines changed, components affected) and preparing training data in advance, the system reduces the complexity of individual changeset evaluation during actual execution, making the complex task manageable through upfront preparation.
Solution Approach 2:
The patent introduces metadata as an intermediary between the raw changeset data and the predictive model. This metadata layer (containing structured information about files, lines, and components) simplifies the interface between complex changeset variations and the assessment model, reducing system complexity by standardizing the interaction interface.
3Reliability
If comprehensive test execution is performed, then reliability of defect detection is improved, but productivity decreases due to resource ties
Solution Approach 1:
The patent applies partial action by executing only the subset of test cases predicted to be relevant for each changeset, rather than running the complete test suite. The predictive model identifies and executes only those tests likely to detect defects in the specific changeset, achieving sufficient defect detection reliability with reduced test execution volume, thereby improving productivity.
Solution Approach 2:
The patent changes the parameter of test execution from fixed (all tests always run) to dynamic (tests selected based on predictive analysis). By using model confidence scores and relevance predictions as selection parameters, the system adapts test execution scope to each changeset's characteristics, maintaining high defect detection reliability while optimizing resource utilization and productivity.
Data Source
Figure 1
AI summary
Problem Software testing is an important yet costly part of state-of-the-art software projects. Prior-art statistical or machine learning (ML) approaches allow for some reduction in test effort by indicating - before their execution - those test cases in a test suite which are likely to fail. Still, test execution ties up a significant part of development resources. The invention aims to free or optimize use of these resources. Solution Computer-implemented method of predicting status of a testcase of a software maintained by means of a code repository, characterized in, for each execution of the testcase, logging the status as a multi-valued attribute such as "pass" or "fail", for each two chronologically adjacent executions, retrieving from the repository any changeset committed interjacently and observing correlation between the changeset and transition in status between the two executions, such as from "pass" to "fail", "fail" to "pass", "fail" to "fail", or "pass" to "pass", based on the observed correlations, building a training dataset from input vectors representing the changesets, each input vector being labelled with the correlating transition, fitting a statistical classification model on the training dataset, and, by means of the model, predicting the transition resulting from committing a given further changeset to the repository.