Behavioral Elicitation Event Database for Objective Training Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for evaluating soft skills in competency-based training, such as those used in pilot training, rely on instructor observation, which is subjective and lacks objectivity, traceability, and consistency.
Innovation Solution
The use of a behavioral elicitation event (BEE) database within a computing system that simulates a physical training environment, allowing for the automation of training performance evaluation by matching user actions with elicited actions and observable behaviors defined in the BEEs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If instructor observation is used to evaluate soft skills, then the evaluation can be performed manually, but the assessment lacks objectivity and consistency
Solution Approach 1:
The patent replaces the manual observation and subjective judgment mechanism with an automated computer-based evaluation system. The system automatically detects user actions, matches them against predefined competency frameworks, and generates objective assessments without human intervention in the evaluation process itself.
Solution Approach 2:
The evaluation system performs self-assessment by automatically monitoring user actions during training simulations and independently determining competency levels. The system serves itself by collecting data, analyzing performance, and generating evaluations without requiring external instructor judgment.
2Loss of information
If instructor observation is used to evaluate soft skills, then the evaluation process is simple, but traceability between actions and competencies is lacking
Solution Approach 1:
The system continuously monitors user actions during training simulations and provides immediate feedback by matching observed behaviors against the competency framework. This feedback loop ensures that traceability is maintained between specific actions and competency assessments, allowing the system to track exactly which actions demonstrate which competencies.
Solution Approach 2:
The evaluation system segments the training performance into discrete, measurable actions and matches them against corresponding competency elements. By breaking down the evaluation into separate, traceable components, the system maintains detailed information about which actions relate to which competencies without creating an unmanageable monolithic assessment.
3Reliability
If automated BEE database evaluation is implemented, then objectivity and traceability are improved, but the system complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-defining competency frameworks, behavioral elicitation events, and evaluation criteria before the actual training simulation. This preparation allows the automated evaluation system to operate consistently and reliably during execution, as all judgment criteria are established in advance, reducing the need for complex real-time decision-making logic.
Solution Approach 2:
The system creates a digital copy of the competency framework and evaluation criteria within the BEE database, allowing automated retrieval and matching during evaluation. This copying approach enables consistent, repeatable assessments without requiring the original complex judgment processes to be re-executed, thereby managing system complexity while maintaining reliability.
Data Source
AI summary
One example provides a method of evaluating a training performance of a training subject and performed in a training environment. The method receives a simulation state update from a computing system of the training environment, and queries a behavioral elicitation event (BEE) database to determine a set of active BEEs based upon the simulation state update. The method further receives a plurality of simulation action notifications from the computing system, and for each simulation action notification, determines whether an action represented by the simulation action notification matches an elicited action for an active BEE. The method further stores information regarding the elicited action as a response to the active BEE. The method determines a set of completed BEEs comprising any active BEEs that have all corresponding elicited actions met, and for each completed BEE, stores information regarding the completed BEE.