Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using an adapted multi-
modal sequence generation neural network, e.g., a multi-
modal language model neural network, to perform medical tasks. For example, the image
encoder neural network of the multi-
modal sequence generation neural network can be fine-tuned to
encode two-dimensional medical images to allow the neural network to perform medical tasks that require reasoning about two-dimensional medical images. As another example, the video
encoder neural network of the multi-modal sequence generation neural network can be fine-tuned to
encode volumetric medical images to allow the neural network to perform medical tasks that require reasoning about volumetric medical images. As another example, the image
encoder neural network of the multi-modal sequence generation neural network can be fine-tuned to
encode non-image medical data, e.g.,
genomics data, to allow the neural network to perform medical tasks that require reasoning about non-image medical data.