Harmonic Averaging Scoring for Digital Assessment Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems lack an efficient method to evaluate the quality of assessment items and their parts, as well as the latent abilities of responders, within digital assessment frameworks, leading to suboptimal content validation and user performance analysis.
Innovation Solution
A system that utilizes a modified two-parameter item response theory model, combined with gradient descent optimization and loss functions, to calculate difficulty, discrimination, and hint change metrics for assessment item parts, and generates health scores for content elements by processing assessment data from a sample population, leveraging general-purpose graphics processing units for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional assessment quality evaluation methods are used, then the assessment framework is simple to implement, but the evaluation accuracy and reliability of assessment item quality and responder abilities are insufficient
Solution Approach 1:
The patent segments the assessment quality evaluation into multiple hierarchical levels (assessment item level, assessment section level, course level) and divides the evaluation into multiple independent metrics (difficulty, discrimination, discrimination index, hint change). This segmentation allows complex evaluation to be broken down into manageable components, each evaluated separately using appropriate statistical methods, thereby improving overall evaluation accuracy while maintaining system manageability.
Solution Approach 2:
The patent introduces and modifies multiple parameters including difficulty parameter, discrimination parameter, discrimination index, and hint change parameter to comprehensively evaluate assessment quality. By changing and expanding the parameter set beyond traditional single-metric evaluation, the system achieves more precise measurement of assessment item quality and responder abilities while providing actionable insights for improvement.
2Measurement precision
If comprehensive quality metrics are evaluated for assessment items, then the content validation accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing difficulty, discrimination, and other metrics during assessment item development and validation phases. These pre-computed values are then readily available for quality evaluation without requiring reprocessing of raw assessment data, significantly reducing processing time during subsequent content validation while maintaining comprehensive accuracy.
Solution Approach 2:
The system continuously updates and maintains quality metrics as assessment data becomes available, rather than performing evaluation only at discrete time points. This continuous accumulation and updating of metrics allows for efficient real-time quality assessment without repeating comprehensive calculations, thereby reducing processing time while maintaining validation accuracy.
3Measurement precision
If detailed health scores and metrics are generated for all content elements, then the user performance analysis quality improves, but the information processing complexity increases
Solution Approach 1:
The patent segments the comprehensive performance analysis into distinct health score components (difficulty health score, discrimination health score, discrimination index health score, hint change health score) and associates each with specific content elements at different hierarchical levels. This segmentation allows detailed analysis to be performed on manageable portions of data independently, reducing overall processing complexity while maintaining comprehensive analysis quality.
Solution Approach 2:
The patent introduces health scores as intermediary metrics that mediate between raw assessment data and final performance analysis. These health scores serve as condensed, interpretable representations of complex data patterns, simplifying the information processing by transforming detailed raw data into summarized metrics that capture essential performance characteristics without requiring processing of all underlying data details.
4Reliability
If multiple metrics and weight values are assigned to assess item quality, then the evaluation reliability improves, but the calculation complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-determining and publishing weight values for each metric based on established psychometric theory and empirical research during the assessment development phase. These pre-established weights are then applied directly during quality evaluation without requiring complex real-time calculations or subjective weight assignment, thereby maintaining high evaluation reliability while simplifying the calculation process during actual assessment.
Solution Approach 2:
The patent standardizes the calculation approach by changing the weight assignment from ad-hoc to predetermined values based on psychometric theory. This parameter change transforms the calculation from a complex, subjective process to a systematic, objective application of pre-established weights, maintaining evaluation reliability while reducing calculation complexity through standardization.
Data Source
AI summary
Systems and methods of the present invention may be used to determine metrics and health scores for content that may correspond to an educational course or textbook, which may be in a digital format. The metrics and health scores may be determined at various hierarchical content levels, and may be used to quantitatively assess how well the corresponding content is performing based on responses submitted to assessment item parts of the content by responders. The metrics may include difficulty and discrimination metrics, which may be determined using maximum likelihood estimation methods based on a modified two parameter item response model. A content analytics interface corresponding to a given content element may be generated and displayed via a user device, and may include content health scores of subcontent within that content element. The subcontent may be ordered according to content health score.