Automated Online Testing Evaluation Using OCR and Constrained AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating participant online testing and providing feedback are manual, tedious, and time-consuming, and conventional AI engines lack the guidance and constraints needed to reliably generate accurate feedback on participant subject matter comprehension.
Innovation Solution
A system utilizing programmatic AI engine management with engineered prompts and constraints to automate the evaluation of online testing, incorporating AI tools for video and audio analysis, and generating coaching reports on participant comprehension and knowledge gaps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual coaching sessions are used to evaluate participants and provide feedback, then personalized feedback can be given, but the process becomes tedious and time-consuming
Solution Approach 1:
The system enables automated self-evaluation where the AI coach independently analyzes test responses, video frames, audio transcripts, and screenshots without requiring manual human intervention. The system serves itself by automatically generating coaching reports with mastery determinations, knowledge gap identifications, and antipattern detections based on multimodal data processing
Solution Approach 2:
The patent replaces the mechanical manual coaching process with an automated AI-based system that processes multimodal data (video, audio, text, screenshots) through machine learning models and AI tools to generate coaching feedback, eliminating the need for human coaches to manually review each participant's test-taking process
2Productivity
If conventional AI engines are used to automate evaluation, then processing speed increases, but they lack guidance and constraints needed for reliable feedback
Solution Approach 1:
The system implements feedback loops where AI-generated coaching reports are continuously refined based on participant responses and test outcomes. The AI coach receives feedback from multiple data sources including participant explanations, test results, and behavioral patterns to improve the reliability of its evaluations over time
Solution Approach 2:
The patent transforms the AI engine's operation by changing key parameters through engineered prompts and constraints that guide the AI's analysis process. These parameter changes include structured evaluation criteria, confidence thresholds, and decision-making rules that enable the AI to produce reliable coaching feedback
3Measurement precision
If comprehensive multimodal data is collected during testing, then analysis accuracy improves, but data processing complexity increases
Solution Approach 1:
The system segments the comprehensive multimodal data into distinct components (video frames, audio transcripts, screenshots, test responses) and processes each segment through specialized AI tools and machine learning models. This segmentation approach manages complexity by handling different data types independently while integrating results for comprehensive coaching reports
Solution Approach 2:
The patent introduces intermediary processing layers that translate complex multimodal data into structured formats suitable for analysis. These intermediaries include data preprocessing pipelines, feature extraction modules, and coordination mechanisms that manage the flow between different data sources and AI analysis tools
Data Source
AI summary
A system and method automatically evaluate participant online testing and automatically provide feedback regarding test results of a participant. Automatic evaluation includes processing video frames, audio, timestamped screenshots, and initial test results of the participant answering multiple test questions and transcribing audio. The system and method utilize optical character recognition on the screenshots to generate a set of text for each screenshot and compare the set of text from each timestamped screenshot to determine which screenshots are associated with each test question. The system and method further determine time periods for each question, segment the transcribed audio, the screenshots, and the video frames by question and select a screenshot and a video frame for each test question from the segmented screenshots and video frames. The system and method utilize a first AI tool and a second AI tool to determine whether the participant demonstrated mastery for each test question.