During the diagnostic process, clinicians leverage multimodal information, such as the chief complaint, medical images, and laboratory-test results. Deep-
learning models for aiding diagnosis have yet to meet this requirement. In some aspects, a
transformer-based representation-learning model can be used as a clinical-
diagnostic aid that processes multimodal input in a unified manner. Rather than learning modality-specific features, the model uses embedding
layers to convert images and unstructured and
structured text into visual tokens and text tokens, and bidirectional blocks with intramodal and intermodal attention to learn holistic representations of radiographs, the unstructured chief complaint and clinical history, structured
clinical information such as laboratory-test results and patient demographic information. The
unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of
pulmonary disease (by 12%and 9%, respectively) and in the prediction of adverse clinical outcomes in patients with COVID-19 (by 29%and 7%, respectively). Leveraging unified multimodal
transformer-based models may help streamline the triaging of patients and facilitate the
clinical decision process.