ML Training Object Evaluation via Overfitting Detection

Solution Overview

Problem

Existing machine learning algorithms face challenges in evaluating the predictive ability of new features or training samples due to overfitting and noise, leading to unreliable results, and require significant computational resources.

Innovation Solution

A method and system for training machine learning algorithms that iteratively generate prediction error indicators to detect overfitting points, retrain models at evaluation starting points before overfitting occurs, and select new features or samples using statistical hypothesis tests to improve predictive model performance while reducing computational resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If machine learning algorithms use more training data and features to improve predictive model accuracy, then the model performance improves, but computational resources and training time increase significantly

Engineering Contradiction:

Improvepredictive model accuracyVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary evaluation of training objects before full model training by generating prediction error indicators and detecting overfitting points early in the process. This preliminary assessment allows the system to identify promising training objects without committing full computational resources to training them, thereby reducing overall training time while maintaining model accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses partial training iterations to generate prediction error indicators rather than completing full training cycles for evaluation. By performing only the necessary portion of training to detect overfitting points and assess predictive ability, the system avoids excessive computational expenditure while still obtaining reliable evaluation metrics.

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If machine learning algorithms perform comprehensive model training to ensure accurate predictions, then predictive accuracy improves, but computational resources are consumed excessively

Engineering Contradiction:

Improvepredictive accuracyVSAvoidcomputational resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary evaluation of training objects before full model training by generating prediction error indicators and detecting overfitting points early in the process. This preliminary assessment allows the system to identify promising training objects without committing full computational resources to training them, thereby reducing overall training time while maintaining model accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses partial training iterations to generate prediction error indicators rather than completing full training cycles for evaluation. By performing only the necessary portion of training to detect overfitting points and assess predictive ability, the system avoids excessive computational expenditure while still obtaining reliable evaluation metrics.

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If machine learning algorithms evaluate new features or training samples using traditional methods, then evaluation completeness is achieved, but results are unreliable due to overfitting and noise

Engineering Contradiction:

Improveevaluation reliabilityVSAvoidevaluation process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system implements feedback mechanisms by monitoring prediction error indicators across multiple iterations and using statistical hypothesis tests to evaluate whether observed improvements are significant. This feedback loop allows the system to distinguish genuine predictive improvements from overfitting or noise, thereby improving evaluation reliability while maintaining manageable process complexity through systematic error tracking and statistical validation.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11416765B2Methods and systems for evaluating training objects by a machine learning algorithm

Publication Date: 2022.08.16 Y E HUB ARMENIA LLC

AI summary

Methods and systems for training a machine learning algorithm (MLA) comprising: acquiring a first set of training samples having a plurality of features, iteratively training a first predictive model based on the plurality of features and generating a respective first prediction error indicator. Analyzing the respective first prediction error indicator for each iteration to determine an overfitting point, and determining at least one evaluation starting point. Acquiring an indication of a new set of training objects, and iteratively retraining the first predictive model with at least one training object from the at least one evaluation starting point to obtain a plurality of retrained first predictive models and generating a respective retrained prediction error indicator. Based on a plurality of retrained prediction error indicators and a plurality of the associated first prediction error indicators, selecting one of the first set of training samples and the at least one training object.