Training Data Selection Using Similarity-Based Diversity Screening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to select diverse data with varying properties for training machine learning models, leading to prolonged training times and increased annotation efforts without improving training effectiveness.
Innovation Solution
A data selection device and program that calculates target data features using feature calculation models, combines them with situation data features, and determines data selection based on similarity with already selected data to ensure diversity and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data with similar properties is used for training, then the training processing time is unnecessarily lengthened and annotation man-hours are increased, but the training effect is not improved
Solution Approach 1:
The system performs preliminary similarity assessment and clustering of training data before the actual model training begins. By pre-processing the dataset to identify and group similar data points, the system eliminates redundant training samples in advance, ensuring that only diverse and valuable data is used for training. This preliminary action prevents wasted training time and annotation efforts on duplicate or highly similar data points.
2Adaptability or versatility
If diverse data with different properties is used for training, then the model can address different situations, but existing methods cannot necessarily select various data having different properties
Solution Approach 1:
The system introduces similarity metrics and clustering algorithms as intermediary mechanisms between the raw training data and the model training process. These intermediaries automatically assess data diversity and select representative samples from different data clusters, ensuring the model receives diverse training data without requiring complex manual curation. The intermediary layer handles the complexity of diversity assessment, making the process systematic and scalable.
3Reliability
If various data having different properties are selected for training, then appropriate training can be performed, but existing methods fail to ensure data diversity
Solution Approach 1:
The system replaces manual or heuristic methods for assessing data diversity with automated computational approaches. Machine learning algorithms and similarity metrics automatically measure and evaluate data property diversity, replacing subjective or mechanical selection processes. This substitution enables reliable detection and measurement of data diversity across multiple dimensions, ensuring high-quality diverse training data selection through objective computational criteria.
Data Source
AI summary
A data selection device which selects data used for training a machine learning model from a dataset including a plurality of target data, includes a processor. The processor is configured to: calculate, from one target data in the dataset, a target data feature of the target data using a feature calculation model; and determine whether to select the target data as data used for training the machine learning model, based on similarity between an individual feature relating to the target data including the target data feature, and an individual feature of other target data in the dataset or an individual feature of data already selected for training the machine learning model.