Automated Constructed Response Scoring via Rubric Segmentation

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

The current process for scoring constructed response items is expensive, time-consuming, and limits the integration of these items into online adaptive testing due to the reliance on human scorers and the lack of efficient rubric refinement methods.

Innovation Solution

A system and method that utilize a computer-program product with a scoring manager, engine, and primitive library to automate the scoring and refinement of constructed response items, allowing for immediate scoring, reduced costs, and broader validation of scoring rubrics, enabling their use in computerized adaptive tests.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If human scorers manually evaluate constructed response items, then scoring reliability can be maintained, but the process becomes expensive and time-consuming

Engineering Contradiction:
Improvescoring reliabilityVSAvoidscoring efficiency
Core Design Contradiction:
ReliabilityVSProductivity

Solution Approach 1:

The patent replaces the manual mechanical scoring process with an automated computer-based system. The scoring engine automatically evaluates constructed response items against rubrics, eliminating the need for human scorers to manually read and score each response. This substitution of mechanical human evaluation with automated computational processing directly addresses the contradiction by maintaining scoring reliability through consistent algorithmic application of rubrics while dramatically improving productivity and reducing costs.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The scoring system performs self-service by automatically evaluating student responses without requiring external human intervention. The computerized scoring engine independently applies rubric criteria to construct responses, generates scores, and provides feedback, making the scoring process self-contained and autonomous. This self-service capability resolves the contradiction by maintaining reliable scoring through systematic automated evaluation while eliminating the time and cost constraints of manual scoring.

Inventive Principle:
Principle #25Self-service

2Measurement precision

If constructed response items are used to measure deeper knowledge levels, then assessment quality improves, but the scoring process becomes more complex and time-consuming

Engineering Contradiction:
Improveassessment qualityVSAvoidscoring process complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the scoring process into distinct modular components: rubric definition module, response evaluation module, scoring calculation module, and quality control module. Each component handles a specific aspect of scoring constructed response items, making the overall complex process manageable and automatable. This segmentation allows the system to maintain high measurement precision for assessing deeper knowledge while reducing the complexity of implementation through modular design.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent introduces a computerized scoring engine as an intermediary between the constructed response items and the scoring rubrics. This intermediary system automatically bridges the gap between complex student responses and evaluation criteria, processing the complexity internally while presenting simple, reliable scores. The intermediary handles the computational burden of evaluating nuanced constructed responses against detailed rubrics, maintaining assessment quality while simplifying the overall scoring process architecture.

Inventive Principle:
Principle #24Intermediary (Mediator)

3Measurement precision

If human scoring is used for constructed response items, then scoring accuracy can be maintained, but the process limits integration into online adaptive testing

Engineering Contradiction:
Improvescoring accuracyVSAvoidintegration into adaptive testing
Core Design Contradiction:
Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent replaces manual human scoring with an automated computerized scoring system that can be integrated into online adaptive testing platforms. The scoring engine processes constructed response items automatically, providing immediate scores that can be used for adaptive testing algorithms. This substitution enables seamless integration into computerized adaptive tests while maintaining scoring accuracy through consistent algorithmic evaluation, resolving the contradiction between accuracy and adaptability.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent enables continuous automated scoring action that operates simultaneously with test administration. Unlike manual scoring which requires discrete human intervention, the automated system provides continuous, real-time scoring capability that supports adaptive testing algorithms. This continuous useful action allows immediate use of scores for item selection and test adaptation, enhancing versatility while maintaining accuracy through systematic automated evaluation.

Inventive Principle:
Principle #20Continuity of useful action

4Manufacturing precision

If scoring rubrics are refined through manual rangefinding, then scoring criteria can be optimized, but the process is time-consuming and lacks large-scale validation

Engineering Contradiction:
Improverubric optimizationVSAvoidrubric refinement time
Core Design Contradiction:
Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent replaces manual rangefinding with automated computer-based rubric refinement and validation. The system automatically evaluates sample responses against proposed rubric criteria, identifies inconsistencies, and provides feedback for optimization. This automated mechanism accelerates the rubric refinement process while enabling large-scale validation that was not feasible with manual methods, resolving the contradiction between precision optimization and time consumption.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent implements automated feedback loops in the rubric refinement process. The scoring system automatically evaluates sample constructed responses against proposed rubric criteria and provides quantitative feedback on rubric performance, identifying areas for improvement and validating rubric effectiveness at scale. This feedback mechanism enables rapid iterative refinement of scoring criteria while providing comprehensive validation data, resolving the time loss associated with manual rangefinding.

Inventive Principle:
Principle #23Feedback

Data Source

PatentUS10643489B2Constructed response scoring mechanism
Publication Date: 2020.05.05 CAMBIUM ASSESSMENT INC
  • US10643489B2 patent drawing
  • US10643489B2 patent drawing
  • US10643489B2 patent drawing

AI summary

A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.