Custom ML Evaluation Model Training via Segmentation and Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for evaluation are large and cumbersome, not customized to individual grader tendencies, and struggle with accuracy when faced with small datasets, making them unsuitable for grading unique questions or customized criteria, thus relying on human evaluators for small-scale assessments.
Innovation Solution
A system that customizes machine learning evaluation models by using pre-existing data for training, iteratively retraining and generating new data based on user inputs, allowing for accelerated training and improved performance with small datasets, and providing automated evaluation for custom prompts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pre-existing large evaluation models are used, then evaluation coverage is broad, but model size becomes large and cumbersome
Solution Approach 1:
The patent divides the evaluation model into multiple specialized sub-models, each trained on specific domains or question types. Instead of using one large monolithic model, the system segments the evaluation function across smaller, domain-specific models that can be selectively deployed based on the evaluation needs, thereby reducing individual model size while maintaining comprehensive coverage.
Solution Approach 2:
The patent creates a universal evaluation framework that can handle multiple evaluation types through a common architecture. The system uses a shared base model that can be adapted to different evaluation scenarios through configuration and data augmentation, allowing one model structure to serve multiple functions rather than requiring separate specialized models for each evaluation type.
2Measurement precision
If human evaluators are used for small-scale assessments, then evaluation accuracy is maintained, but productivity decreases
Solution Approach 1:
The patent implements self-service evaluation where the system automatically generates evaluation data and performs assessments without requiring human intervention. The model uses self-generated training data and automated evaluation protocols to maintain high accuracy while dramatically increasing throughput, allowing the system to evaluate itself and scale independently of human resources.
Solution Approach 2:
The patent incorporates feedback mechanisms where evaluation results are continuously fed back into the training process. The system uses automated feedback loops to refine model predictions, correct errors, and improve accuracy over time, enabling the system to achieve human-level accuracy through iterative learning rather than requiring human evaluators for each assessment.
3Adaptability or versatility
If machine learning models are trained on small datasets, then adaptability to custom prompts improves, but training accuracy deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-training models on large, diverse datasets before fine-tuning on small custom datasets. This two-stage approach allows the model to learn general evaluation patterns and facts during pre-training, then adapt to specific custom prompts through minimal fine-tuning data, maintaining accuracy while achieving specialized adaptation.
Solution Approach 2:
The patent uses data augmentation techniques as an intermediary between the small custom dataset and the final model training. The system generates synthetic training data and augmented versions of existing data to expand the effective training set size, allowing the model to learn from limited real data while maintaining robust accuracy through virtual expansion of the training corpus.
Data Source
AI summary
Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.