Warning Message Classification via Sampling and Balancing

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Software developer tools generate a large number of unimportant warning messages, overwhelming developers, due to their inability to consider contextual information, leading to an imbalanced data set that affects the accuracy of classifiers in distinguishing between 'fix' and 'ignore' messages.

Innovation Solution

The approach involves generating an accurate classifier by sampling and balancing the imbalanced data set before feature selection, using machine learning methods to classify warning messages effectively, and employing a feedback mechanism to adjust sampling methods for improved performance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If fault finding tools generate warning messages for all potential flaws, then the completeness of flaw detection is improved, but the number of unimportant warning messages increases, overwhelming developers

Engineering Contradiction:
Improvecompleteness of flaw detectionVSAvoidnumber of warning messages
Core Design Contradiction:
ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the set of all warning messages into different priority levels or categories based on their importance. By dividing the monolithic set of warnings into segmented groups (e.g., critical, important, low priority), the system can present only the most relevant messages to developers while maintaining complete detection capability in the background.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent applies different quality standards or filtering criteria to different subsets of warning messages. Instead of treating all messages uniformly, it assigns local quality attributes (priority levels, severity ratings) to each message type, allowing developers to focus on high-quality (high-priority) messages while less important ones are automatically filtered or deprioritized.

Inventive Principle:
Principle #3Local quality

2Quantity of substance

If classifiers are trained on imbalanced data sets with many ignored messages, then the data set size is sufficient, but the accuracy of distinguishing 'fix' from 'ignore' messages deteriorates

Engineering Contradiction:
Improvedata set sizeVSAvoidclassification accuracy
Core Design Contradiction:
Quantity of substanceVSMeasurement precision

Solution Approach 1:

The patent introduces counterbalancing techniques to offset the dominance of the majority class (ignored messages) in the training data. This may involve weighting schemes where minority class samples (fix messages) are given higher importance, or synthetic generation of additional minority class samples to counterbalance the imbalance and achieve better classification accuracy.

Inventive Principle:
Principle #8Anti-weight (Counterweight)

Solution Approach 2:

The patent performs preliminary data preprocessing actions before training the classifier, including resampling, balancing, or augmenting the data set to address the imbalance issue in advance. This preliminary action ensures that the training data has appropriate class distribution, enabling the classifier to learn effective decision boundaries for both classes.

Inventive Principle:
Principle #10Preliminary action

3Measurement precision

If developers manually triage all warning messages to determine which to fix, then the accuracy of message prioritization is improved, but the time spent on triage increases significantly

Engineering Contradiction:
Improvemessage prioritization accuracyVSAvoidtriage time
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent implements self-service mechanisms where the system automatically prioritizes and filters warning messages using trained classifiers and learned patterns from historical data. Instead of requiring developer intervention for each message, the system serves itself by automatically assigning priorities, filtering low-value messages, and presenting only the most relevant warnings to developers, thereby reducing triage time while maintaining accuracy.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The patent incorporates feedback loops where the system learns from developer responses to previously prioritized messages. By analyzing which messages developers actually fix versus ignore, the system refines its classification model and prioritization algorithms over time, improving accuracy while maintaining automated operation, thus avoiding the need for manual triage.

Inventive Principle:
Principle #23Feedback

Data Source

PatentEP4170555A1Classifying warning messages generated by software developer tools
Publication Date: 2023.04.26 BLACKBERRY LTD
  • EP4170555A1 patent drawingFigure 1~2
  • EP4170555A1 patent drawingFigure 3
  • EP4170555A1 patent drawingFigure 4

AI summary

A method for classifying warning messages generated by software developer tools includes receiving a first data set. The first data set includes a first plurality of data entries, where each data entry is associated with a warning message generated based on a first set of software codes, includes indications for a plurality of features, and is associated with one of a plurality of class labels. A second data set is generated by sampling the first data set. Based on the second data set, at least one feature is selected from the plurality of features. A third data set is generated by filtering the second data set with the selected at least one feature. A machine learning classifier is determined based on the third data set. The machine learning classifier is used to classify a second warning message generated based on a second set of software codes to one of the plurality of class labels.