Multi-Stage Testing Calibration via Module Shift Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-Stage Testing (MST) faces challenges in calibrating item banks due to the need for large-scale pre-testing and the computational instability of existing concurrent calibration methods, which can lead to increased standard errors of estimated item parameters.
Innovation Solution
A processor-implemented method and system for MST calibration, which involves estimating the average difficulty level of each module, calculating the difference in difficulty levels as a shift factor, and aligning the modules to a common scale using this shift factor, thereby enabling assessment of users on a unified scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If concurrent calibration is used to estimate all item parameters in one go, then all item parameters can be aligned to the same scale, but the method becomes computationally unstable and demanding for large assessments
Solution Approach 1:
The patent segments the calibration process into two distinct stages: (1) within-module calibration where item parameters are estimated separately for each module, and (2) between-module linking where module difficulty parameters are estimated to align modules on a common scale. This segmentation reduces computational complexity and improves stability compared to concurrent calibration of all items simultaneously.
Solution Approach 2:
The patent performs preliminary calibration of item parameters within each module before performing the linking calibration between modules. This preliminary action allows the within-module item parameters to be estimated first, which then serves as the foundation for the subsequent between-module difficulty alignment, reducing the overall computational burden.
2Device complexity
If mean-mean linking or fixing item parameters is used to overcome computational instability, then computational stability is improved, but the standard errors of estimated item parameters significantly increase
Solution Approach 1:
The patent segments the calibration into within-module item parameter estimation and between-module difficulty parameter estimation. This allows both item parameters and module difficulty parameters to be estimated with appropriate precision, avoiding the significant standard error increase associated with mean-mean linking or fixing item parameters approaches.
3Measurement precision
If large-scale pre-testing is conducted to calibrate the item bank, then accurate item difficulty levels can be obtained, but the time and resource requirements increase significantly
Solution Approach 1:
The patent segments the item bank into multiple modules and calibrates each module separately using the extended Rasch model. This segmentation allows for more efficient calibration that requires less extensive pre-testing compared to calibrating all items simultaneously in a single large-scale pre-test.
Solution Approach 2:
The patent performs preliminary within-module calibration before between-module linking, which allows for more efficient use of test data and reduces the need for large-scale pre-testing to achieve accurate item difficulty estimates.
Data Source
AI summary
MST calibration is required to ensure that all examinees are assessed on a common scale. State of the art approaches have the disadvantages that they become computationally unstable and demanding or may have standard errors of estimated item parameters. Method and system in the embodiments disclosed herein provide a module-wise calibration approach in which a shift parameter value is estimated as difference in the average difficulty level between each two modules from among the plurality of modules. Further, based on the estimated value of the shift parameter, the estimated average difficulty level of each of the plurality of modules is aligned to a common scale. By aligning each of the plurality of modules to the common scale users who took different combinations of the plurality of modules are assessed on the common scale.