Clinical Note Evaluation With Pseudo-Reference Quality Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generated clinical notes often require significant editing to achieve acceptable quality, disrupting workflow due to varying quality based on input and model training, necessitating systems for evaluating note quality and generation models.
Innovation Solution
A system and method using machine learning models to evaluate clinical notes by comparing generated notes with pseudo-reference notes, extracting units, calculating sub-scores, and predicting quality scores, with a threshold for outputting notes or requesting manual drafting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If note generation models are used to produce clinical notes, then productivity is improved, but the quality of generated notes varies significantly requiring extensive editing
Solution Approach 1:
The system implements a feedback mechanism by generating pseudo-reference clinical notes and comparing them with actually-produced clinical notes using multiple evaluation metrics. This feedback loop enables automatic quality assessment and identification of gaps between generated and reference notes, allowing for continuous improvement of the note generation model while maintaining high productivity.
Solution Approach 2:
The system performs preliminary actions by generating pseudo-reference clinical notes before final evaluation. These pseudo-reference notes serve as benchmarks for comparing against actually-produced notes, enabling quality assessment to be conducted in advance and allowing editors to focus only on notes that fail to meet quality thresholds.
2Measurement precision
If multiple evaluation metrics are used to assess clinical notes, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The evaluation system is segmented into multiple independent evaluation metrics, each assessing specific aspects of clinical note quality (e.g., content accuracy, structure, completeness). This segmentation allows for comprehensive quality measurement while maintaining modularity, making the complex evaluation process more manageable and interpretable.
Solution Approach 2:
The system employs a universal evaluation framework that can assess multiple aspects of clinical note quality simultaneously using different metrics. This multi-functional approach enables a single evaluation system to handle diverse quality dimensions (content, structure, language) without requiring separate evaluation processes for each aspect.
3Loss of time
If generated clinical notes are automatically evaluated and filtered, then loss of time in editing is reduced, but reliability of note quality control may be compromised
Solution Approach 1:
The system introduces an intermediary automated evaluation layer between note generation and final manual review. This intermediary uses machine learning models and multiple metrics to assess note quality, providing an objective intermediate assessment that guides subsequent manual editing decisions while preserving the reliability of quality control through systematic evaluation.
Solution Approach 2:
The system replaces manual quality assessment mechanics with automated machine learning-based evaluation. This substitution significantly reduces the time required for quality checking while maintaining reliability through the use of trained models and multiple evaluation metrics that objectively assess note quality without human bias or fatigue.
Data Source
AI summary
A first system includes a processor configured to: compare, via a machine learning model, a generated clinical note and a plurality of pseudo-reference clinical notes based on at least one evaluation metric to predict a quality score of the generated clinical note; and output, via an output device, the generated clinical note if the quality score is greater than or equal to a predetermined acceptable threshold, and a request to manually draft a clinical note otherwise. A second system includes a processor configured to: compare, via a machine learning model, a plurality of generated clinical notes and at least one final clinical note based on a plurality of evaluation metrics to determine model scores associated with the note generation models that generated the plurality of generated clinical notes; and select one of the plurality of note generation models at least partially based on the model scores.