Neural Network Sample Screening via Similarity Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge is to efficiently screen training samples for neural networks to prevent delays and noise interference, which can reduce the efficacy of the training process, especially when dealing with large datasets and erroneous samples.
Innovation Solution
An electronic device and method that calculates similarity vectors for multiple samples, identifying a representative sample with the maximum average similarity value and filtering out samples with minimal similarity, using modules like sample collection and screening modules within a processor and storage media to select or remove samples based on similarity metrics such as inner product, Euclidean distance, or Chebyshev distance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If more training data is used to improve neural network efficacy, then the training accuracy is improved, but the training time is extended
Solution Approach 1:
The patent extracts and removes noisy or redundant samples from the training dataset through similarity comparison. By calculating similarity between samples and identifying outliers (samples with low similarity to their cluster), the system extracts only the essential, high-quality training data needed for effective neural network training, thereby reducing training time while maintaining accuracy.
2Quantity of substance
If all training samples are used including noisy samples, then the data coverage is improved, but the training efficacy is reduced due to noise interference
Solution Approach 1:
The patent converts the harmful effect of noisy samples into a benefit by using them as comparison references. By calculating similarity between all samples including noisy ones, the system identifies which samples are actually noisy (those with low similarity to their cluster) and excludes them. The noisy samples thus serve the purpose of helping identify and remove themselves and similar noisy data, improving training efficacy while maintaining data coverage.
3Measurement precision
If similarity calculation is performed for all samples to identify representative samples, then the sample quality is improved, but the computational complexity is increased
Solution Approach 1:
The patent segments the training process into two stages: first, divide samples into clusters based on similarity; second, within each cluster, select representative samples by comparing similarity. This segmentation reduces computational complexity by avoiding full pairwise comparison of all samples, instead performing comparisons only within smaller clusters, thus maintaining sample quality while reducing overall computational burden.
Data Source
AI summary
An electronic device and a method for screening a sample are provided. The method includes the following steps. N samples corresponding to a first object are received, in which the N samples include a first sample. N similarity vectors respectively corresponding to the N samples are calculated, in which the N similarity vectors include a first similarity vector corresponding to the first sample. The first similarity vector includes multiple first similarities between the first sample and each of the N samples except the first sample. The first sample is determined to be a representative sample of the first object in response to an average value of the first similarities of the first similarity vector being the maximum value among average values of N similarities respectively corresponding to the N similarity vectors.