Scripted Test Assembly System for Item Selection and Randomization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing test assembly methods, such as the weighted deviations model, fail to consistently meet test content, statistical, and security requirements, leading to inconsistent test scores and over-exposure of highly discriminating items, compromising measurement precision.
Innovation Solution
A deterministic item selection process using a test administration script and item collections, which separates content and statistical requirements, ensures all test takers receive correct content representation and equalizes item usage without compromising score precision by selecting items based on optimization statistics and randomization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the weighted deviations model is used for item selection, then test statistical and content requirements are met in probability, but individual test takers may receive tests that do not conform to specifications and highly discriminating items become over-exposed
Solution Approach 1:
The patent segments the item selection process into two distinct phases: (1) deterministic content representation assignment that guarantees specification compliance, and (2) stochastic item selection from pre-defined sets that maintains reliability. This segmentation resolves the contradiction by ensuring content requirements are met deterministically while allowing statistical requirements to be satisfied through controlled randomization within each content set.
Solution Approach 2:
The patent applies preliminary action by pre-defining multiple item sets for each content representation, where each set contains items that satisfy both content and statistical specifications. This preliminary preparation ensures that when items are selected during test assembly, the specifications are already guaranteed to be met, eliminating the need for complex real-time optimization and preventing specification violations.
2Adaptability or versatility
If the weighted deviations model is used, then content and statistical requirements are addressed simultaneously, but information is not distributed evenly and highly discriminating items become over-exposed
Solution Approach 1:
The patent segments the item pool into multiple distinct sets corresponding to different content representations. Each set is pre-configured to contain items with balanced discrimination characteristics. This segmentation prevents over-exposure of highly discriminating items by distributing them across different content sets, while still allowing flexible selection within each set to meet test specifications.
Solution Approach 2:
The patent applies local quality by ensuring that each content representation set contains items with locally optimized properties (balanced discrimination, appropriate difficulty ranges). This local optimization within each set ensures even distribution of information across all content areas while maintaining overall test flexibility and adaptability to different test taker abilities.
3Measurement precision
If deterministic content representation is implemented, then all test takers receive correct content representation and accurate item development forecasts are possible, but the system complexity increases
Solution Approach 1:
The patent reduces system complexity through preliminary action by pre-computing and storing multiple item sets for each content representation during test construction. This preliminary preparation transforms a potentially complex real-time optimization problem into a simpler selection process during test administration, where the system only needs to retrieve pre-defined sets rather than perform complex calculations for each test assembly.
Solution Approach 2:
The patent applies copying by creating multiple duplicate item sets for each content representation. These copies are pre-validated to ensure they meet content and statistical specifications. During test assembly, the system simply selects from these pre-copied validated sets, avoiding the need for complex real-time validation and optimization, thus reducing system complexity while maintaining measurement precision.
4Productivity
If the weighted deviations model is used, then item selection is performed, but the proportion of available items actually administered is limited and usage frequency is uneven
Solution Approach 1:
The patent segments the item pool into multiple content representation sets, each containing a diverse collection of items. This segmentation allows the system to administer items from different sets to different test takers or even to the same test taker across different test administrations, thereby increasing the proportion of unique items administered while ensuring even usage distribution across all items through controlled selection from multiple sources.
Data Source
AI summary
A method and system for assembling a test having a plurality of slots is disclosed. The method includes executing a test administration script to associate each of the plurality of slots of the test with an item collection from which an item to fill the slot is to be chosen and to determine a lookup table of optimization statistics and a degree of randomization to use in selecting items from the item collection. An interim test score is obtained based on a portion of the test previously assembled. Optimization statistics are applied to control a property of the test based on the interim score while taking into account the degree of randomization.