Custom Scoring Model Training via Iterative Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for evaluation are large, cumbersome, and not customized to individual grader tendencies, leading to reduced accuracy with smaller data sets and inability to effectively evaluate unique or customized criteria, relying on human evaluators for small-scale assessments.
Innovation Solution
A system for customizing machine learning evaluation models using pre-existing data, iterative retraining, and user feedback to accelerate training and improve performance with small data sets, enabling automated evaluation of custom prompts and interfaces for non-programmers to customize models according to specific attributes and preferences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are trained with large data sets, then evaluation accuracy is improved, but model size becomes large and cumbersome
Solution Approach 1:
The patent segments the training process into multiple stages: pre-training on large datasets to establish foundational evaluation capabilities, then fine-tuning on smaller, domain-specific datasets to customize for particular evaluation tasks. This segmentation allows the model to achieve high accuracy for specific purposes without requiring the entire large dataset to be processed simultaneously, thereby reducing the effective model size for each specific task.
Solution Approach 2:
The patent applies partial action by selectively training models on representative subsets of data rather than requiring complete datasets. The system identifies and trains on the most critical and informative data points, achieving sufficient evaluation accuracy without processing all available data, thus avoiding the complexity of handling massive datasets while maintaining high performance on key evaluation metrics.
2Manufacturing precision
If human evaluators are used for small-scale assessments, then customization to individual grader tendencies is achieved, but productivity is reduced
Solution Approach 1:
The patent implements feedback mechanisms where the system continuously monitors and adjusts its evaluation outputs based on comparisons with human evaluator responses. The model learns from discrepancies between automated and human evaluations, progressively improving its ability to replicate individual grader tendencies. This feedback loop enables the system to achieve human-level customization accuracy while maintaining automated processing speed.
Solution Approach 2:
The patent changes parameters by adapting model training to match specific evaluator profiles and grading tendencies. The system adjusts evaluation parameters, weightings, and decision thresholds to mirror individual human evaluators' preferences and patterns. This parameter customization allows automated models to replicate the precision of human evaluators while eliminating the time consumption associated with manual grading processes.
3Device complexity
If models are trained on small data sets, then model size is reduced, but evaluation accuracy deteriorates
Solution Approach 1:
The patent applies preliminary action through pre-training models on extensive datasets before fine-tuning on smaller, task-specific data. This preliminary training establishes a robust foundation of evaluation capabilities and general patterns, enabling the model to achieve high accuracy even when subsequent fine-tuning uses limited data. The preliminary action ensures that the model starts with sufficient knowledge base, reducing the impact of small dataset limitations.
Solution Approach 2:
The patent uses copying by creating simplified representations and summaries of complex evaluation patterns from large datasets. The system generates distilled knowledge, rule sets, and pattern templates that capture essential evaluation criteria in a condensed form. These copied representations can be applied to small datasets, allowing the model to maintain high evaluation accuracy without requiring the full complexity of large original datasets.
Data Source
AI summary
Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.