Lesion Annotation From Medical Reports Using Probabilistic 3D Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image annotation methods struggle to accurately correlate textual medical descriptions with spatial image annotations, particularly in scenarios where models like SAM, CLIP, or YOLO-World are not directly applicable, due to complex spatial references and variable terminology in clinical reports.
Innovation Solution
A computer-implemented method that processes medical descriptions and images to identify and annotate lesions by matching entities and attributes using a modular and 3D-aware framework, incorporating encoder-only transformer models, character convolutional neural networks, and conditional random fields to integrate textual and visual cues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If direct segmentation or object detection models (SAM, CLIP, YOLO-World) are used for medical image annotation, then the annotation process can be automated, but the accuracy of correlating textual descriptions with spatial annotations deteriorates due to complex spatial references and variable terminology
Solution Approach 1:
The patent segments the annotation task into multiple specialized modules: a text processing module that extracts entities and attributes from clinical reports, an image processing module that detects candidate lesions, and a matching module that correlates them using a probabilistic framework. This segmentation allows each module to specialize in handling specific aspects of the complexity, thereby maintaining automation while improving accuracy.
Solution Approach 2:
The patent introduces an intermediary probabilistic matching framework that acts as a mediator between textual descriptions and image annotations. This framework uses probability calculations to handle the uncertainty and variability in matching entities from clinical reports with detected lesions, resolving the contradiction by providing a flexible intermediate layer that accommodates complex spatial references and variable terminology.
2Ease of manufacture
If predefined models are used for annotation, then the process is simpler to implement, but the system becomes less adaptable to varying terminology and complex spatial references in clinical reports
Solution Approach 1:
The patent creates a universal annotation framework that can handle multiple types of medical images and clinical report formats through standardized processing modules. The text processing module extracts various entity types (lesions, anatomical structures, findings) using a unified approach, while the matching module adapts to different spatial references and terminology through probabilistic reasoning, achieving both ease of implementation and high adaptability.
Solution Approach 2:
The patent employs parameter changes by using a probabilistic framework where matching parameters (probability thresholds, attribute weights) can be adjusted based on the specific clinical context. This allows the system to adapt to varying terminology and spatial references by dynamically modifying matching parameters rather than requiring complete reconfiguration of the annotation system.
3Measurement precision
If manual annotation is performed to ensure accuracy, then the precision of lesion identification improves, but the time consumption and labor requirements increase significantly
Solution Approach 1:
The patent implements feedback mechanisms where the matching module uses probability calculations to evaluate and refine the correlation between textual entities and image lesions. This feedback loop allows the system to automatically adjust and improve matching accuracy based on the consistency between extracted attributes and detected lesion characteristics, achieving high precision without manual intervention.
Solution Approach 2:
The patent enables the annotation system to perform self-service by automatically extracting entities from clinical reports, detecting lesions in images, and matching them through probabilistic reasoning. The system serves itself by internally resolving ambiguities and validating matches without requiring manual annotation, thereby maintaining high precision while eliminating time-consuming manual labor.
Data Source
Figure 1
Figure 2A~2B
Figure 3~4
AI summary
A computer-implemented method and data processing device for detecting lesions from an image obtained by medical imaging and from a medical description, comprising: obtaining identified entities and corresponding attributes from the medical description; obtaining candidate lesions and corresponding attributes from the image obtained by medical imaging; matching the candidate lesions and corresponding attributes with the identified entities and corresponding attributes; and selecting the matched filtered candidate lesions and identified entities using a highest probability criterion corresponding to an entity identified in the medical description matching a candidate lesion obtained from the medical image.