Defective Data Extraction in Machine Learning Training Sets

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Machine learning models face inefficiencies in classifying input data due to the presence of unsuitable and defective data, such as outlier and overlap data, which can hinder proper learning and verification processes.

Innovation Solution

A method utilizing a vector neural network with multiple vector neuron layers to extract defective data by calculating feature spectra, determining degrees of similarity, and applying a defectiveness function to identify and remove outlier and overlap data based on threshold comparisons.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If training data includes all available data without filtering, then the quantity of training data is maximized, but the reliability of machine learning model deteriorates due to presence of defective data

Engineering Contradiction:

Improvequantity of training dataVSAvoidreliability of machine learning model

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent extracts defective data (outliers and overlap data) from the training dataset by calculating defectiveness indices based on feature spectrum similarities. This extraction process removes harmful elements while preserving the overall dataset quantity, thereby maintaining training data volume while improving model reliability through selective elimination of defective samples.

Inventive Principle:

Principle #2Taking out (Extraction)

2Reliability

If defective data is removed from training data, then the reliability of machine learning model is improved, but the quantity of training data decreases

Engineering Contradiction:

Improvereliability of machine learning modelVSAvoidquantity of training data

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent changes the parameter of data quality by introducing a defectiveness index calculation that evaluates each training sample based on feature spectrum similarity. By adjusting the threshold for defectiveness determination, the system can control the balance between removing defective data and preserving sufficient training data quantity, thereby improving reliability without excessive data loss.

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If manual inspection of training data is performed to identify defective data, then the measurement precision of defective data detection is improved, but the productivity of data preparation process deteriorates

Engineering Contradiction:

Improveprecision of defective data detectionVSAvoidproductivity of data preparation

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent replaces manual inspection with an automated computational system that calculates feature spectra and defectiveness indices using a trained machine learning model. This substitution of mechanical/manual processes with automated computational methods maintains high detection precision while dramatically improving productivity by enabling batch processing of large datasets without human intervention.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system performs self-service by automatically identifying and flagging defective data through the defectiveness index calculation mechanism. The machine learning model autonomously evaluates each training sample's feature spectrum similarity and determines defectiveness without requiring external manual review, thereby maintaining precision while enhancing processing efficiency and productivity.

Inventive Principle:

Principle #25Self-service

4Measurement precision

If feature spectrum calculation is performed for all training data, then the measurement precision of defective data identification is improved, but the use of energy and computational resources deteriorates

Engineering Contradiction:

Improveprecision of defective data identificationVSAvoidcomputational energy consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent applies partial action by calculating defectiveness indices selectively rather than uniformly processing all training data with equal depth. The system uses the trained machine learning model to identify and focus computational resources on samples with higher probabilities of being defective, thereby maintaining identification precision while reducing overall computational energy consumption through targeted analysis.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20230162489A1Method of extracting unsuiitable and defective data from plurality of pieces of training data used for learning of machine learning model, information processing device, and non-transitory computer-readable storage medium storing computer program

Publication Date: 2023.05.25 SEIKO EPSON CORP

AI summary

A method according to the present disclosure includes (b1) selecting a reference class from a plurality of classes, (b2) calculating a plurality of degrees of similarity between a feature spectrum corresponding to target training data and a plurality of the feature spectra belonging to the reference class, (b3) applying, to the plurality of degrees of similarity, a defectiveness function that is determined in advance, and calculating a defectiveness index with respect to the target training data, and (b4) determining whether the target training data is the defective data, based on a result of comparison between the defectiveness index and a threshold value.