Vision Transformer ECG Diagnosis Patch Tokenization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine learning methods, particularly convolutional neural networks (CNNs), are unreliable in accurately diagnosing cardiac conditions from electrocardiograms (ECGs) due to their high variance and requirement for large datasets, as well as their inability to identify subtle or chaotic patterns, especially when these patterns are not contiguous.

Innovation Solution

A vision transformer system that partitions ECG images into patches, generates numerical tokens, and uses a transformer module with multi-head attention and multi-layer perceptron components to produce a diagnostic message indicating the patient's heart health, leveraging pre-training on large unlabeled datasets and fine-tuning with ECG-specific data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If convolutional neural networks (CNNs) are used to process ECG data, then the system can handle high-dimensional data, but the model requires large amounts of data to prevent overfitting and has high variance

Engineering Contradiction:

Improveability to handle high-dimensional ECG dataVSAvoiddiagnostic accuracy with limited data

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The ECG image is divided into multiple patches (e.g., 16x16 pixel regions), and each patch is processed independently to generate tokens. This segmentation allows the model to capture local patterns while reducing the overall complexity and data requirements compared to processing the entire high-dimensional ECG image at once through a CNN.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary tokenization step that converts visual ECG patches into numerical tokens, which then serve as input to the transformer model. This intermediary representation bridges the gap between image-based ECG data and the transformer's sequence processing capability, enabling effective diagnosis with limited data while maintaining adaptability to high-dimensional inputs.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of manufacture

If CNNs are used to interpret ECGs as 2D images, then pre-trained models can be leveraged through transfer learning, but suboptimal results occur when there are substantial differences between pre-training and fine-tuning datasets

Engineering Contradiction:

Improveavailability of pre-trained modelsVSAvoiddiagnostic accuracy with domain mismatch

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The patent replaces the CNN-based transfer learning approach with a transformer architecture that processes ECG data through tokenization. This substitution eliminates the domain mismatch problem inherent in CNN transfer learning from natural images to medical ECGs, as the transformer does not rely on pre-trained visual features from unrelated domains, thereby improving reliability while maintaining ease of model deployment.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Device complexity

If machine learning models consider only contiguous regions of the ECG, then processing is simplified, but pathological patterns that occur in different parts of the ECG recording are missed

Engineering Contradiction:

Improvesimplicity of processing contiguous regionsVSAvoiddetection of non-contiguous pathological patterns

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent merges the processing of multiple discrete ECG patches into a unified diagnostic decision through the transformer's self-attention mechanism. By combining information from all patched regions of the ECG, the model can detect pathological patterns regardless of their spatial distribution, thereby improving measurement precision while maintaining manageable processing complexity through the tokenization framework.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent transforms the spatial dimension of ECG data into a sequence dimension through patching and tokenization. This dimensional transformation allows the transformer to process non-contiguous regions effectively by treating each patch as a sequence element, enabling the model to capture global dependencies across the entire ECG recording rather than being constrained to local contiguous regions.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20240389920A1Vision transformer system and method configured to generate a patient diagnosis from an electrocardiogram

Publication Date: 2024.11.28 MT SINAI SCHOOL OF MEDICINE

AI summary

A vision transformer system and method generate a diagnosis from an electrocardiogram (ECG) of the patient. A patch generating module generates image patches of the ECG. A tokenization module generates numerical patch-based tokens corresponding to image patches. A transformer module generates a numerical classification token from the numerical patch-based tokens. A classification module generates and outputs a diagnosis message from the numerical classification token, wherein the diagnosis message is the patient diagnosis corresponding to the patient ECG and indicating a state of health of the heart of the patient. A masking module mask a preset portion of the plurality of patches, and the numerical classification token is generated from the plurality of numerical patch-based tokens, the unmasked patches, and the masked patches. The tokenization module receives ECG training data to be trained to generate the numerical classification token. The method implements the vision transformer system.