Multimodal Medical Report Generation With ML Refinement

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Manual generation of medical reports is time-consuming, prone to errors, and inconsistent, leading to potential inaccuracies in recording medical details and complications.

Innovation Solution

An apparatus utilizing machine learning models, including vision-language and speech recognition models, generates medical reports from multi-modal data such as video and audio recordings, patient vital signs, and medical records, refining the reports with a large language model to ensure accuracy and consistency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual report generation is used, then healthcare professionals can write detailed reports with clinical judgment, but the process is time-consuming and reduces productivity

Engineering Contradiction:

Improveaccuracy of medical detailsVSAvoidreport generation speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent introduces an automatic report generation system that acts as an intermediary between multi-modal medical data (video, audio, sensor data) and the final medical report. The system processes raw data through ML models to generate drafts, which are then refined by healthcare professionals, reducing their workload while maintaining accuracy through multiple refinement passes.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the manual mechanical process of report writing with an automated system using vision-language models, speech recognition models, and large language models. These AI models automatically extract information from video recordings, audio recordings, and sensor data to generate report drafts, substituting the manual writing process while allowing professional review.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If manual report generation is used, then reports can be customized to individual cases, but inconsistencies and errors increase

Engineering Contradiction:

Improvereport customizationVSAvoidconsistency of medical records

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent implements a feedback mechanism where the automatic report generation system produces drafts that are reviewed and refined by healthcare professionals. The system allows for iterative refinement passes where inconsistencies are identified and corrected, and the final report is validated against the original multi-modal data to ensure accuracy and consistency.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent creates a universal report generation framework that can handle multiple types of medical procedures and data formats (video, audio, sensor data) through a single integrated system. The system maintains consistency by using standardized processing pipelines while allowing customization for different procedure types through configurable parameters and refinement rules.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If handwritten reports are used, then reports can be written quickly by professionals, but legibility and understanding become difficult

Engineering Contradiction:

Improvereport writing speedVSAvoidreport readability

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent uses automatic speech recognition to create text copies of spoken medical procedures, and vision-language models to generate text descriptions from video recordings. These automated text generations replace handwritten reports entirely, providing machine-generated, perfectly legible text that maintains the speed advantage while eliminating readability issues.

Inventive Principle:

Principle #26Copying

4Productivity

If automated report generation is implemented, then time and errors are reduced, but system complexity increases

Engineering Contradiction:

Improvereport generation efficiencyVSAvoidsystem architecture

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the report generation system into distinct functional modules: a vision-language model for processing video data, a speech recognition model for audio data, a refinement module for improving draft quality, and a validation module for ensuring accuracy. This modular segmentation manages complexity by allowing each component to be developed, tested, and maintained independently while working together to achieve the overall goal of automated report generation.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250285718A1Systems and methods for automatic medical report generation

Publication Date: 2025.09.11 SHANGHAI UNITED IMAGING INTELLIGENCE CO LTD

AI summary

The decision process of a first machine learning (ML) model may be explained based on a second ML model implemented on an apparatus. The apparatus may obtain a prediction about an image made based on the first ML model. The apparatus may further determine visual concepts associated with the image that may have been used by the first ML model to make the prediction, and determine respective contributions of the visual concepts to the prediction made by the first ML model. The apparatus may then generate, based on the second ML model, a textual description that explains the respective contributions of the visual concepts to the prediction made by the first ML model. The second ML model may determine respective image features associated with the visual concepts, map the determined image features to corresponding text features, and generate the textual description based at least on the text features.