Automated Scoring of Scientific Visual Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human scoring of visual models created by students to represent scientific understanding is inconsistent and costly, making it difficult to scale up evaluations fairly and accurately.
Innovation Solution
A system for automated scoring that extracts construct-relevant features from visual models using a multidimensional scoring rubric, generating a statistical model to estimate learning progression levels, allowing for fair and valid assessment of scientific concepts like Matter, using computer processors and data structures to output scores on a graphical user interface or for printing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human scoring of visual models is used, then assessment accuracy and fairness are improved, but labor costs and time consumption increase significantly
Solution Approach 1:
The patent creates a digital copy of the human scoring process by training an automated scoring system on human-annotated visual models. The system learns from multiple human scorers' annotations and replicates their scoring patterns, achieving 88% agreement with human scorers while eliminating the need for actual human time investment in scoring each model.
Solution Approach 2:
The patent replaces the mechanical human scoring process with an automated computational system. Instead of human experts manually evaluating visual models, the system uses trained algorithms to automatically score models, achieving both speed and consistency while reducing labor costs and time consumption.
2Loss of information
If human scoring of visual models is used, then detailed feedback and nuanced evaluation are improved, but scalability and consistency deteriorate
Solution Approach 1:
The patent creates a universal scoring system that can evaluate any visual model within its domain consistently. The automated system applies the same scoring criteria uniformly across all models, eliminating scorer variability while maintaining the ability to provide detailed feedback on multiple dimensions of model quality.
Solution Approach 2:
The system copies the comprehensive feedback capabilities of human scorers by analyzing multiple human annotations and replicating their evaluative patterns. This allows the automated system to provide nuanced feedback across multiple dimensions while maintaining consistency and scalability.
3Productivity
If automated scoring is implemented, then scalability and consistency are improved, but assessment accuracy and fairness may deteriorate
Solution Approach 1:
The patent performs preliminary action by training the automated scoring system on extensive human-annotated data before deployment. Multiple human scorers annotate training models, and the system learns from these annotations, ensuring it captures human scoring patterns and nuances before being used for actual assessment, thereby maintaining accuracy while achieving scalability.
Solution Approach 2:
The system incorporates feedback mechanisms by training on human scorer annotations and continuously improving its scoring accuracy. The system learns from the feedback implicit in human scoring patterns and adjusts its scoring to align with human judgment, ensuring fairness and accuracy while maintaining automated scalability.
Data Source
AI summary
Systems and methods are provided for processing a drawing in a modeling prototype. A data structure associated with a visual model is accessed. The visual model is analyzed to extract construct-relevant features, where the construct-relevant features are extracted using a drawing object by identifying visual attributes of the visual model and populating a data structure for each object drawn. The visual model is analyzed to generate a statistical model, where the statistical model is generated using a multidimensional scoring rubric by targeting different constructs which compositely estimate learning progression levels, wherein the statistical model is based on features that are principally aligned with one or more of the constructs. An automated scoring is determined based on the construct-relevant features and the statistical model, where the automated scoring is stored in a computer readable medium. and is outputted for display, transmitted across a computer network, or printed.