Off-Topic Detection via Similarity Grids and CNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer-implemented assessment engines face difficulties in accurately identifying off-topic spoken responses in high-stakes speaking assessments, particularly when new test questions are introduced and reference samples are limited, which can compromise the validity of assessment scores.
Innovation Solution
The approach involves transcribing spoken responses using automated speech recognition, generating a similarity grid representation by comparing the responses with prompts, and employing machine learning models like deep convolutional neural networks to score the similarity, with optional channels for scaling similarities based on word importance values, to determine the likelihood of an off-topic response.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional text comparison techniques are used to identify off-topic responses, then the system is simple to implement, but the detection precision and recall are insufficient
Solution Approach 1:
The patent transforms the off-topic detection problem from traditional text comparison into an image processing domain. By generating similarity grids that visualize word embedding similarities between response and prompt texts, the system applies 2D spatial relationships to enhance detection capabilities. The similarity grid converts textual similarity into a visual matrix where patterns of off-topic responses become detectable through image processing techniques.
Solution Approach 2:
The patent replaces conventional text-based comparison mechanisms with deep learning models trained on image data. Instead of using traditional string matching or simple semantic similarity measures, the system employs convolutional neural networks (CNNs) that were originally designed for image recognition. This substitution enables the system to detect complex patterns in similarity grids that indicate off-topic responses, achieving superior precision and recall.
2Reliability
If machine learning models are employed to improve detection accuracy, then off-topic responses can be identified more accurately, but the computational complexity and processing time increase
Solution Approach 1:
The patent performs preliminary processing by generating similarity grids before applying deep learning models. The similarity grid pre-computes word embedding similarities between response and prompt texts, organizing the data into a structured 2D format. This preliminary action transforms raw text data into a standardized image-like representation, making it easier for the CNN to process and reducing the computational burden during the actual detection phase.
Solution Approach 2:
The patent changes the parameter representation from traditional text features to image-based similarity grid parameters. By transforming the detection problem into the image domain, the system can leverage parameters such as pixel intensity, spatial patterns, and gradient information that are native to image processing. This parameter transformation enables the use of highly optimized CNN architectures that efficiently handle the detection task.
3Productivity
If reference samples are limited for new test questions, then new questions can be launched quickly, but the ability to detect off-topic responses deteriorates
Solution Approach 1:
The patent creates a universal detection system based on similarity grids that can handle both new and existing test questions effectively. The deep learning model is trained on similarity grid patterns from various questions and learns generalizable features of off-topic responses. This universal approach allows the system to detect off-topic responses in new questions even when reference samples are limited, as the model has learned transferable patterns rather than question-specific features.
Solution Approach 2:
The patent uses word embedding technology to capture semantic relationships between words, creating a representation that preserves meaning rather than exact wording. This copying of semantic information into the similarity grid allows the system to detect off-topic responses based on meaning similarity rather than surface-level text matching. As a result, the system can generalize to new questions with different wording but similar semantic patterns.
Data Source
AI summary
Data is received that encapsulates a spoken response to a prompt text comprising a string of words. Thereafter, the received data is transcribed into a string of words. The string of words is then compared with a prompt so that a similarity grid representation of the comparison can be generated that characterizes a level of similarity between the string of words in the spoken response and the string of words in the prompt text. The grid representation is then scored using at least one machine learning model. The score indicates a likelihood of the spoken response having been off-topic. Data providing the encapsulated score can then be provided. Related apparatus, systems, techniques and articles are also described.