Classification Model Selection for Reliable ML Data Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies face challenges in collecting high-quality teacher data and test data due to the mixing of low-quality data with inappropriate labels, which affects model performance and accuracy, and require significant manpower for data collection.
Innovation Solution
A data collection system and method that includes a classification model storing unit, model delivery unit, classification result storing unit, optimum model recommendation unit, and teacher data recording unit, which classifies and recommends appropriate models, corrects labels, and records high-quality data to reduce the inclusion of low-quality data, enhancing data reliability and user motivation to select accurate data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a large amount of data is collected from multiple users, then the quantity of teacher data is improved, but the quality deteriorates due to mixing of low-quality data with inappropriate labels
Solution Approach 1:
The system evaluates the performance of classification models using test data and feeds back this evaluation information to select appropriate models for processing teacher data. This feedback mechanism ensures that only high-quality data from reliable models is used, maintaining data quality while collecting from multiple users.
Solution Approach 2:
The system performs preliminary evaluation and selection of classification models before they are used to process teacher data. By pre-evaluating model performance on test data and selecting only high-performing models, the system prevents low-quality data from being collected in the first place, rather than filtering afterward.
2Reliability
If experienced personnel carefully assign labels to data, then the quality of teacher data is improved, but the productivity deteriorates due to significant manpower requirements
Solution Approach 1:
The system enables automatic evaluation of classification models and automatic selection of appropriate models for processing data. This self-service capability eliminates the need for manual label assignment by experienced personnel, significantly improving productivity while maintaining data quality through automated model performance assessment.
Solution Approach 2:
The system replaces the mechanical process of manual label assignment by experienced personnel with an automated classification model evaluation and selection system. The automated system uses test data to evaluate model performance and selects appropriate models, substituting human labor with computational processes while maintaining or improving data quality.
3Adaptability or versatility
If multiple classification models are provided, then the adaptability is improved, but the ease of operation deteriorates as users find it hard to determine which model to use
Solution Approach 1:
The system evaluates the performance of multiple classification models using test data and provides feedback information about each model's accuracy and suitability. This feedback enables users to easily identify the most appropriate model for their specific needs without having to manually assess multiple models, maintaining adaptability while improving ease of operation.
Solution Approach 2:
The system performs preliminary evaluation of multiple classification models using test data before presenting them to users. By pre-assessing model performance and identifying the most suitable model for each user's data characteristics, the system eliminates the complexity of model selection while preserving the benefits of having multiple models available.
Data Source
AI summary
A data collection system includes a classification model storing unit, a model delivery unit, a classification result storing unit, an optimum model recommendation unit, and a teacher data recording unit. The model delivery unit delivers a classification model to the user environment. The classification result storing unit classifies each of the classification models on the classification model storing unit using data with a label transmitted from the user environment as an input. The classification result storing unit stores a classification result including at least one of classification correctness or a percentage of correct answers for each input data. The optimum model recommendation unit presents an appropriate classification model for the input data based on the classification result for each of the classification models. The teacher data recording unit records the input data as teacher data or test data of the classification model.