Identifying highly relevant MRI scans from historical databases

Abstract

Methods have been developed for selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current MRI scan. As such, the methods can help identify previous MR images from historical patients that are likely to display the same medical condition as the current patient. In turn, this can help a medical doctor to appropriately treat the current patient, such as by diagnosing the patient's condition or by choosing an appropriate medical intervention. Additionally, the system can record how users interact with the search results, and this user interaction data can be used to update and improve the selection algorithm for future searches.

Description

Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2IDENTIFYING HIGHLY RELEVANT MRI SCANS FROM HISTORICAL DATABASESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 716,586, filed November 5, 2024, which application is incorporated herein by reference in its entirety.INTRODUCTION

[0002] Magnetic resonance imaging (MRI) is a non-invasive medical imaging technique that is widely used in clinical diagnosis and pathological analysis. In principle, MRI techniques use the interaction between radio frequency (RF) pulses, a strong magnetic field, and body tissues to generate images of the organs from inside a body. In most medical applications, hydrogen nuclei (which consist solely of a proton) that are in tissues create a signal that is processed to form images of the body. Only a small part of the hydrogen nuclei contributes to the measured signal due to their different alignment in the magnetic field. The protons are capable of absorbing energy when exposed to a short RF pulse (electromagnetic energy) at their resonance frequency. After absorbing energy from the RF pulse, the nuclei are promoted to the excited state where they only stay for a short time period, and then return to their initial state of equilibrium (so-called decay) by releasing energy. This transmission of energy by the nuclei as they return to their initial state is what is observed as the MRI signal. The subtle differences of that signal from different tissues combined with advanced signal processing algorithms implemented on modem computing devices is what enables various organs to be distinguished. Any imaging plane, or slice, can be projected, and then stored or printed for further diagnosis and treatment.

[0003] During the diagnosis and treatment of a patient, it can be helpful for medical doctors to compare the current patient’s MRI scan to similar scans from historical patients. However, this typically requires the medical doctor to examine their personal memory for potentially similar patients that they personally treated in the past. However, this approach has several disadvantages. First, the human memory is imperfect, and therefore the medical doctor might forget about historical patients that had similar MRI scans. Second, a single medical doctor has only diagnosed and treated a limited number of patients, especially at the beginning of their career. Therefore, the medical doctor cannot personally remember the MRI scans of patients that were treated by other medical doctors.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2SUMMARY

[0004] Provided are methods and systems for selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current MRI scan. As such, the methods find use in identifying previous MR images from historical patients that are likely to display the same medical condition as the current patient. In turn, this enables a medical doctor to appropriately treat the current patient, such as by diagnosing the patient’s condition or by choosing an appropriate medical intervention. Additionally, systems of the present disclosure can record how users interact with the search results, and this user interaction data can be used to update and improve the selection algorithm for future searches.BRIEF DESCRIPTION OF THE FIGURES

[0005] FIG. 1 : Schematic illustration of steps in a computer-implemented method according to embodiments of the present disclosure (top). Example query data for the current test MRI scan (middle). Example selected entries (bottom).

[0006] FIG. 2: An example user interface where a user enters a scan ID and example image outputs are displayed to the user, along with a similarity rating. The user can then view a selected MRI using the "View" button or choose to remove the result using the "Remove" button.DETAILED DESCRIPTION

[0007] Before the present invention is described in greater detail, it is to be understood that this invention is not limited to particular embodiments described, as such may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0008] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limits of that range is also specifically disclosed. Each smaller range between any stated value or intervening value in a stated range and any other stated or intervening value in that stated range is encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included or excluded in the range, and each range where either, neither or both limits are included in the smaller ranges is also encompassed within the invention, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0009] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, some potential and exemplary methods and materials may now be described. Any and all publications mentioned herein are incorporated herein by reference to disclose and describe the methods and / or materials in connection with which the publications are cited. It is understood that the present disclosure supersedes any disclosure of an incorporated publication to the extent there is a contradiction.

[0010] It must be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any element, e.g., any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only” and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0011] The publications discussed herein are provided solely for their disclosure prior to the filing date of the present application. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed. To the extent the definition or usage of any term herein conflicts with a definition or usage of a term in an application or reference incorporated by reference herein, the instant application shall control.

[0012] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. Any recited method can be carried out in the order of events recited or in any other order which is logically possible.DEFINITIONS

[0013] The term “current test MRI scan” is used interchangeably herein with “test MRI scan”. The terms “MR image”, “image”, “MRI scan” and “scan” are used interchangeably herein. The terms “historical patient medical record” and “patient medical record” are used interchangeably herein.

[0014] The term “computer-implemented method” means that the method includes at least one step implemented using one or more processors and one or more non-transitory computer-readable media.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0015] A variety of processor-based devices may be employed to implement the embodiments of the present disclosure. Such devices may include device architecture wherein the components of the device are in electrical communication with each other using a bus. Device architecture can include a processing unit (CPU or processor), as well as a cache, that arc variously coupled to the device bus. The bus couples various device components including device memory, (e.g., read only memory (ROM) and random access memory (RAM), to the processor.

[0016] Device architecture can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of the processor. Device architecture can copy data from the memory and / or the storage device to the cache for quick access by the processor. In this way, the cache can provide a performance boost that avoids processor delays while waiting for data. These and other modules can control or be configured to control the processor to perform various actions. Other device memory may be available for use as well. Memory can include multiple different types of memory with different performance characteristics. Processor can include any general purpose processor and a hardware module or software module, such as first, second and third modules stored in the storage device, configured to control the processor as well as a special-purpose processor where software instructions are incorporated into the actual processor design. The processor may essentially be a completely self-contained computing device, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0017] To enable user interaction with the computing device architecture, an input device can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device can also be one or more of a number of output mechanisms. In some instances, multimodal devices can enable a user to provide multiple types of input to communicate with the computing device architecture. A communications interface can generally govern and manage the user input and device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0018] The storage device is typically a non-volatile memory and can be a hard disk or other types of computer-readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), and hybrids thereof.

[0019] The storage device can include software modules for controlling the processor. Other hardware or software modules are contemplated. The storage device can be connected to the device bus. In one aspect, a hardware module that performs a particular function can include theAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor, bus, output device, and so forth, to carry out various functions of the disclosed technology.

[0020] Embodiments within the scope of the present disclosure may also include tangible and / or non-transitory computer-readable storage media or devices for carrying or having computerexecutable instructions or data structures stored thereon. Such tangible computer-readable storage devices can be any available device that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as described above. By way of example, and not limitation, such tangible computer-readable devices can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device which can be used to carry or store desired program code in the form of computer-executable instructions, data structures, or processor chip design. Examples of memory storage devices include tape drives, removable hard disc drives, flash drives, solid state drives, and floppy disks. When information or instructions are provided via a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable storage devices.

[0021] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform tasks or implement abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0022] Other embodiments of the disclosure may be practiced in network computing environments with many types of computer device configurations, including personal computers, hand-held devices, multi-processor devices, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by aAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.COMPUTER-IMPLEMENTED METHODS AND SOFTWARE

[0023] Provided are methods for selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current test MRI scan.MRI scan

[0024] As used herein, an “MRI scan” includes: a first set comprising a plurality of two-dimensional images that were each recorded using the same first MRI sequence as each other, but recorded along different planes of a patient’s body; and a second set comprising a plurality of two-dimensional images that were each recorded using the same second MRI sequence as each other, but recorded along different planes of the patient’s body, wherein the second MRI sequence is different than the first MRI sequence,

[0025] As known by skilled artisans, an “MRI sequence” refers to the parameters used by an MRI machine to record an MR image. For instance, some of the parameters used by the MRI machine include the number of radiofrequency pulses and gradients. Examples of categories of MRI sequences include T1 weighted (T1W), T2 weighted (T2W), proton density (PD), and diffusion weighted. Each of these categories of MRI sequences have different advantages and disadvantages for observing different types of tissues under different conditions. Background about MRI sequences and the technology of MRI are provided in various publications, including Plewes (“Physics of MRI: A Primer”, Review: MR Physics for Clinicians, doi: 10.1002 / jmri.23642) and Sunder S. Rajan (“MRI: A Conceptual Overview”, Springer, 1997, ISBN-13: 978-0387949116).

[0026] Thus, a certain MRI sequence is used to record a first two-dimensional image of a region of the patient (e.g. an image in a first x-y plane). Then, the same MRI sequence is used to record a second two-dimension image in a new plane (e.g. the second image is in a second x-y plane that is parallel to the first x-y plane but has a different z- value). This process can be repeated to record a third image, a fourth image, a fifth image, or any suitable number of images. If all of the images are parallel to one another, then they can be referred to as different “slices” of the same patient with the same MRI sequence. Since all of the images are recorded with the same first MRI sequence, then all the images belong to the first set.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0027] Afterwards, a second MRI sequence can be used that is different than the first MRI sequence. In an analogous manner, the second MRI sequence can be used to record multiple two- dimensional images, thereby generating the second set of images, such as a second set of “slices”.

[0028] As such, the MRI scan includes at least a first set and a second set, and optionally a third set, a fourth set, a fifth set, or any suitable number of sets. Each set includes multiple two- dimensional images that were recorded along different planes, but with the same MRI sequence as each other.General computer-implemented method

[0029] Provided arc computer-implemented methods for selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current test MRI scan.

[0030] The computer-implemented method includes executing on a processor the steps of:(a) receiving the test MRI scan;(b) generating test extracted features from at least two images from the test MRI scan; wherein a database comprises a plurality of entries that each comprise: an MRI scan; extracted features associated with images from the MRI scan; a patient medical record associated with the MRI scan;(c) selecting, using a trained model, two or more entries from the database that have extracted features that are similar to the generated test extracted features,(d) instructing a display to visually show at least one image from each of the two or more selected entries;(e) receiving notice that a user chose one of the shown images; and(f) instructing the display to visually show patient information, wherein the patient information comprises a first section of the patient medical record associated with the chosen image.

[0031] The first step is receiving a “test MRI scan”. The word “test” does not structurally limit the scope of the MRI scan, and merely indicates that the MRI scan is from the patient currently being diagnosed and treated (i.e. in contrast to MRI scans from historical patients).

[0032] As such, the first step of “receiving the test MRI scan” means that the processor receives the test MRI scan, such as from computerized storage on the same computer, or from another networked computer.

[0033] The second step involves generating test extracted features from at least two images of the test MRI scan. “Feature extraction” is a well-known process in machine learning, computerAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 science, and data analysis where input data is analyzed and transformed to thereby create “features” that help represent the original input data. For example, de-la-Bandera et al. states that “feature extraction is a family of dimensionality reduction techniques where a new set of features is built from the original feature set. In order to reduce dimensionality, the number of the new features is lower than the number of the original ones. Feature extraction is a tool that projects the original features onto a more convenient and reduced basis. These new features are computed in such a way that they retain as much information as possible from the original feature set” (Sensors (Basel), 2020, 20(23): 6944, 10.3390 / s20236944). Feature extraction has been used in fields such as image processing, natural language processing, and signal processing. Since the input data can have multiple characteristics that might be redundant or irrelevant to the desired analysis, feature extraction can simplify the data by capturing and highlighting the most important characteristics from the original data. Feature extraction from images is described in numerous publications including WO 2020 / 207377 (“image recognition”), WO 2019 / 105106 (“image categorizing” and “personalized recommendation”), WO 2020 / 253773 (“medical image classification”), and WO 2021 / 031815 (“medical image segmentation”).

[0034] Additionally, the method involves a database that has a plurality of entries. Each entry includes an MRI scan, extracted features associated with images of the MRI scan, and a patient medical record associated with the MRI scan. Thus, the database pertains to historical MRI scans from previous patients (i.e. in contrast to the current patient). Whereas the second step involved generating test extracted features from the test MRI scan, an analogous process was performed on the historical MR images to generate historical extracted features. This generation of historical extracted features is not recited in the claimed method since it was already performed and the database already contains the historical extracted features.

[0035] The third step involves selecting, using a trained model, two or more entries from the database. Selections are made of entries that have extracted features that are similar to the generated test extracted features. As such, the selecting step is choosing historical MRI scans that appear to be similar to the current test MRI scan based on a comparison of extracted features. This comparison and selection is performed using the trained model.

[0036] Various types of models can be used to perform the selecting step. For example, in some cases the trained model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof. The terms “artificial neural network” and “neural network” are used interchangeably herein. In some cases, the trained model comprises a similarity metric. Techniques for training and utilizing such trained models are well known in the art. For example, a review of neural networks is provided by Abiodun et al (Heliyon, 2018, doi: 10.1016 / j.heliyon.2018.e00938) and Liu et al provides aAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 survey of deep neural network architectures and their applications (Neurocomputing, 2017, 234, 1 1, doi: 10.1016 / j.neucom.2016.12.038). The use of similarity metrics for the analysis of images is discussed in Prieto et al (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25, 10, 1265, doi: 10.1109 / TPAMI.2003.1233900), Sinha ct al (Perception, 2011, 40, 11, doi: 10.1068 / p7063), and Melbourne et al (SPIE Medical Imaging in San Diego in the United States, 2010, 762335, doi: 10.1117 / 12.840389).

[0037] Notably, the selection is not based on only a single MR image from the current patient. Instead, extracted features from multiple MR sequences from the current MRI scan are used during the selection step. By simultaneously leveraging multiple MR sequences instead of only a single MR image, the present selection step can be more effective at finding historical MR images that display the same condition as the current patient. This enables the medical doctor to more appropriately treat the current patient, such as by diagnosing the patient’ s condition or by choosing an appropriate medical intervention.

[0038] Fourth, the method includes instructing a display to visually show at least one image from each of the two or more selected entries. For example, the display can be a computer monitor, an integrated display of a laptop computer, an integrated display of a tablet computer, a television, or a display integrated into a medical device. For example, the computer monitor can show the window of a computer program, and the window can simultaneously show an image from the first selected entry along with an image from the second selected entry. As another example, the computer program window can show the first image and then show the second image once a user (e.g. a medical doctor) scrolls down within the window or indicates that the first image is not relevant or desired. The processor can send the instructions to the display in any suitable manner, such as sending the instructions to a graphics card (e.g. GPU), which then sends the instructions to the display. The processor and display can be connected through a wired connection (e.g. Universal Serial Bus (USB) Type-C®) or wireless connection (e.g. Bluetooth®).

[0039] Based on this display of the image, the user (e.g. medical doctor) can interact with the display through any suitable manner. For instance, the user can use a mouse or keyboard. The user can also use a touch interface. If the user deems one of the images to be relevant, the user can select that image. Accordingly, the processor will receive notice that the user chose one of the displayed images.

[0040] In response, the processor will instruct the display to visually show patient information. As used herein, “patient information” includes a first section of the patient medical record associated with the chosen image. The patient medical record can include one or more data from the group consisting of: age of the patient, sex of the patient, reason for recording the MRI scan, the identity and optionally the results of additional medical tests performed on the patient after theAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2MRI scan was recorded, a diagnosis of a condition based on the MRI scan (e.g. an ICD-10 diagnosis), a written analysis of the MRI scan, and a summary of the patient medical record.

[0041] Thus, since the patient information includes a first section of the patient medical record, the patient information can include one or more of the data recited above. In some cases, the displayed patient information includes all of the infomiation included in the patient medical record, and therefore the whole patient medical record is displayed.

[0042] In some cases, the summary of the patient medical record was created with generative artificial intelligence (Al) based on the patient medical record. As such, in some cases the method further includes the step of executing on the processor the further step of creating the summary of the patient medical record with generative Al. In some cases, the generative Al is natural language processing (NLP) generative Al. In some cases, the Al summary is generated in response to, and therefore performed after, the user chose the corresponding entry.

[0043] Thus, as described above, the computer-implemented method includes executing on the processor the steps of receiving the test MRI scan, generating test extracted features from two or more images, selecting two or more database entries based on extracted features and the trained model, instructing the display to visually show images from the selected entries, receiving notice that the user chose a displayed image, and instructing the display to visually show patient information.User feedback data

[0044] The methods can also include the collection of user feedback data, which can be advantageously used to improve the quality of the results displayed to users during future executions of the method on future patients. For example, in some cases the processor execution also includes: collecting user feedback data comprising: (i) how the user responds to the displayed two or more MR images, (ii) how the user responds to the displayed patient information, or (iii) a combination thereof.

[0045] Examples of user feedback data include: an MR image being chosen by the user for displaying the patient information, an MR image being flagged as relevant or potentially relevant by the user, an MR image being flagged as irrelevant by the user, the amount of time a user was shown an MR image, the amount of time the user was shown the patient information, and a request by the user to display a second section of the patient medical record.

[0046] For example, the review of the displayed images by the user can cause the user to evaluate whether the displayed image is relevant to the current patient. The user can then manually tell the computer system of whether the image is relevant, potentially relevant, or not. This canAttorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-2 be done in an analogous manner to other recommendation algorithms known in the art. Even if the user does not directly tell the user of the assessed relevance, a user might view and consider images and patient information for longer or shorter periods of time depending on whether it appears relevant. Thus, the amount of time a user is shown particular information can used as an estimation for how long the user was consciously considering the information, and therefore an estimation of whether the user considered it relevant. Similarly, requests for more information (e.g. a second section of the patient medical record) indicates that the user considers the particular MRI scan to be at least potentially relevant to the current patient.

[0047] In some cases, the method further includes causing the processor to update the trained model based on the collected user feedback data.Additional aspects of the computer- implemented method

[0048] As described above, the selection involves test extracted features from two or more images. This can include multiple images from the same set, a single image from multiple sets, or multiple images each from multiple sets. These possibilities are represented in the following table at three embodiments.

[0049] Type A: In some embodiments, the at least two images from the test MRI scan comprises at least two images from the first test set.

[0050] Type B : In some embodiments, the at least two images from the test MRI scan comprises at least one image from the first test set and at least one image from the second test set.

[0051] Type C: In some embodiments, the at least two images from the test MRI scan comprises at least two images from the first test set and at least two images from the second test set.

[0052] Additionally, in some cases the at least two images from the test MRI scan can includes images from 3 or more sets, such as 4 or more, 5 or more, 6 or more, or 8 or more. In some cases, there can be at least 2 images used from each set, such as 3 or more images, 4 or more images, or 5 or more images. In some embodiments, the at least two images from the MRI test scan comprises all the images from the test MRI scan. In other embodiments, the at least two images from the MRI scan does not include at least one image from the test MRI scan.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0053] In some embodiments, the method includes further steps. For example, in some cases additional steps performed by the processor include: receiving a request that a user desires to view a second section of the patient medical record; and displaying the second section of the patient medical record.

[0054] For example, a user can select an image and the display can show patient information that only includes the age and sex of the patient. As used herein, the “image” shown to user can be either the full resolution MR image or a reduced resolution version of the MR image, e.g. a thumbnail image. If desired, the user can then request a second section of the patient medical record, which could also include data such as the identity of additional medical tests performed, and the results of those additional tests, and an ICD-10 diagnosis. Hence, the user could perform an initial assessment to determine if the entry is relevant (e.g. if the sexes and approximate ages are about the same), and then request the additional information.

[0055] After reviewing the first chosen image and its associated patient information, the medical doctor can then choose to continue to review additional images and entries. If the medical doctor chooses a second image, then the computer processor can: receiving notice that the user chose a second one of the displayed MR images; and displaying second patient information, wherein the second patient information comprises a first section of the second patient medical record associated with the second chosen MR image.

[0056] Additionally, the review by the user can include obtaining information about the MRI sequence used to collect the images. As such, the steps executed by the processor can include: optionally receiving a request that a user desires to view data regarding the MRI sequence used to obtain the MR image; and displaying the data regarding the MRI sequence used to obtain the MR image.

[0057] The step of “receiving the request” regarding the MRI sequence is optional because in some embodiments the MRI sequence data is displayed to the user by default, i.e. without being specifically requested by the user.

[0058] In some cases, the MRI scan is a brain MRI scan (e.g. some or all of the body parts shown on the MRI scan are located in the head). In some cases, the MRI scan includes at least a portion of the posterior half of the brain.

[0059] Additionally, as discussed above, the processor executing the computer-implemented method can interact with only other parts of the same computer (e.g. long term computer memory storage) or also interact with computers that it is networked with. In some cases, the test MRI scan and the database are located on computer storage devices that are located within 20 km of one another, such as within 10 km or within 5 km or within 1 km. In some cases, the test MRIAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 scan are located on the same private network. As used herein, “private network” refers to a computer network that uses a private address space of internet protocol (IP) addresses. In some cases, the test MRI scan and the database are located on the same local area network (LAN). A “local area network” is a computer network that connects computers within a limited geographic area, such as a residence, school, university campus, hospital campus, or office building. In some cases, the test MRI scan and the database are located on the same wide area network (WAN). A “wide area network” is a computer network that connects computers over a larger geographic area that a LAN, and sometimes involves leased telecommunication circuits. In some cases, the test MRI scan and the database are located on different computers that communicate with one another using private internet protocol addresses (e.g. in contrast to public internet addresses). “Private internet protocol addresses” are not allocated to any particular organization, i.e. since they are only used internally, in contrast to a public IP address which is only allocated to a single person or organization, i.e. since they are used publicly. lienee, if the communication between database computer and test MRI scan computer is via private IP addresses, then the communication must be within an internal network.

[0060] Locating the test MRI scan and database within the same computer network, or within the same geographic area, can provide certain advantages. For instance, many MRI scans are personal medical records which are protected against unrestricted release to the public for privacy reasons. As such, performing the method exclusively within a single non-public computer network allows the necessary comparison steps to be performed without releasing the information to the general public, or without risking such a data break by transmitting the information over the public internet. This also allows a large internal database to be utilized without needing to obtain MRI scans from an external third party.

[0061] Also provided by the present disclosure is a non-transitory computer readable storage medium with computer executable instructions stored thereon for performing the computer- implemented methods. The non-transitory computer readable storage medium and instructions can perform any of the aspects discussed above regarding the processor-executed method.METHODS PERFORMED BY A PERSON

[0062] Provided by the present disclosure are methods of treating a patient by comparing a test magnetic resonance imaging (MRI) scan of the patient with similar historical MRI scans and associated historical patient information. The methods can be performed by any suitable person, such as a medical doctor, a surgeon, or a radiologist.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0063] Provided is a method of treating a patient by comparing a test magnetic resonance imaging (MRI) scan of the patient with similar historical MRI scans and associated historical patient information, wherein a computer system comprises a processor and a non-transitory computer readable storage medium as described herein, wherein the method comprises:(i) instructing the processor of the computer system to execute the steps of:(a) receive the test MRI scan;(b) generating test extracted features from at least two images from the test MRI scan; wherein a database comprises a plurality of entries that each comprise: an MRI scan; extracted features associated with images from the MRI scan; a patient medical record associated with the MRI scan;(c) selecting, using a trained model, two or more entries from the database that have extracted features that are similar to the generated test extracted features;(d) instruct a display to visually show at least one image from each of the two or more selected entries;(ii) viewing the displayed MR images;(iii) choosing one or more of the displayed MR images with the computer system, thereby causing the processor to:(e) receive notice that a user chose one of the shown images; and(f) instruct the display to visually show patient information, wherein the patient information comprises a first section of the patient medical record associated with the chosen image;(iv) viewing the displayed patient information associated with the chosen MR image; and(v) treating the patient based on the test MRI scan, one or more viewed MR images, and one or more viewed displayed patient information.

[0064] Thus, in step (i), the user instructs the processor of the computer system to perform steps (a) through (d). During step (d), the processor instructs the display to visually show at least one image from each of two or more selected entries. Accordingly, in step (ii) the user then views the displayed MR images and in step (iii) chooses one or more of the displayed MR images. Based on this choice, the processor then performs steps (e) and (f), which results in the display of the patient information. Accordingly, in step (iv) the user views the displayed patient information and in step (v) the user treats the patients.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2

[0065] Examples of treating the patient include requesting that another person (e.g. a nurse) perform an additional medical test and actually performing the additional medical test. In some cases, the treating includes diagnosing the patient with a condition. In some cases, treating includes requesting or performing the administration of a pharmaceutically active ingredient (API) or conducting a surgery.

[0066] In some cases, the method further includes: requesting a second section of the patient medical record from the computer system; and viewing the displayed second section of the patient medical record.

[0067] In such cases, the processor of the computer system performs the corresponding tasks.

[0068] In some cases, the method further includes: choosing a second one or more of the displayed MR images with the computer system; and viewing the second displayed patient information associated with the chosen second MR image.

[0069] In such cases, the processor of the computer system also performs the corresponding tasks.METHOD OF TRAINING A MODEL AND SOFTWARE

[0070] Also provided by the disclosure are a computer-implemented method of training a model for selecting historical magnetic resonance imaging (MRI) scans that appear to show a medical condition that is radiographically related to a medical condition shown on a current test MRI scan. The method includes:(a) obtaining a first training set comprising MRI scans;(b) training, with a processor and the first training set, a model to select an MRI scan that is likely to show the same medical condition by extracting features from each of two or more images of the MRI scan.

[0071] Once trained, these models can be used to perform the methods described above.

[0072] In some cases, the model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof. In some cases, the model comprises an artificial neural network. In some cases, the model comprises a similarity metric.

[0073] In some embodiments, the training is performed with a machine learning algorithm, a deep learning algorithm, a deep neural network algorithm, a convolutional neural network algorithm, a metric learning algorithm, an attention-based algorithm, or a combination thereof. In some cases, the training is performed with a metric learning algorithm and optionally one or moreAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 additional algorithms. In some cases, the metric learning algorithm is a supervised metric learning algorithm from similar and dissimilar pairs. For example, Shu et al describes classification of similar and dissimilar pairs (Machine Learning, 2023, 113, 3463, doi: 10.1007 / s 10994-023- 06434-6).

[0074] In some cases, the method can further include updating the model based on user feedback data, which is described above. In such cases the method further includes: creating a second training set comprising the MRI scans and the user feedback data; and retraining the trained model with the second training set.

[0075] The present disclosure further provides a non-transitory computer readable storage medium with computer executable instructions stored thereon executed by a processor to perform a method of training a model. The method includes:(a) obtaining a first training set comprising MR images; and(b) training, with a processor and the first training set, a model to select an MRI scan that is likely to show the same medical condition by extracting features from each of two or more images of the MRI scan.

[0076] In some cases, the method further includes: receiving user feedback data comprising: (i) how the user responds to the displayed two or more MRI scans, (ii) how the user responds to the displayed at least some of the patient medical record, or (iii) a combination thereof; creating a second training set comprising the MRI scans and the user feedback data; and retraining the trained model with the second training set.

[0077] Although the foregoing invention has been described in some detail by way of illustration and example for purposes of clarity of understanding, it is readily apparent to those of ordinary skill in the art in light of the teachings of this invention that certain changes and modifications may be made thereto without departing from the spirit or scope of the appended claims.

[0078] Accordingly, the preceding merely illustrates the principles of the invention. It will be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention as well as specific examplesAttorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.

[0079] The scope of the present invention, therefore, is not intended to be limited to the exemplary embodiments shown and described herein. Rather, the scope and spirit of present invention is embodied by the appended claims. In the claims, 35 U.S.C. §112(f) is expressly defined as being invoked for a limitation in the claim only when the exact phrase "means for" or the exact phrase "step for" is recited at the beginning of such limitation in the claim; if such exact phrase is not used in a limitation in the claim, then 35 U.S.C. § 112(f) is not invoked.

Claims

Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2CLAIMSWhat Is Claimed Is:

1. A computer-implemented method for selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current test MRI scan, wherein an MRI scan comprises: a first set comprising a plurality of two-dimensional images that were each recorded using the same first MRI sequence as each other, but recorded along different planes of a patient’s body: and a second set comprising a plurality of two-dimensional images that were each recorded using the same second MRI sequence as each other, but recorded along different planes of the patient’ s body, wherein the second MRI sequence is different than the first MRI sequence, wherein the method comprises executing on a processor the steps of:(a) receiving the test MRI scan;(b) generating test extracted features from at least two images from the test MRI scan, wherein a database comprises a plurality of entries that each comprise: an MRI scan, extracted features associated with images from the MRI scan, and a patient medical record associated with the MRI scan;(c) selecting, using a trained model, two or more entries from the database that have extracted features that are similar to the generated test extracted features,(d) instructing a display to visually show at least one image from each of the two or more selected entries;(e) receiving notice that a user chose one of the shown images; and(f) instructing the display to visually show patient information, wherein the patient information comprises a first section of the patient medical record associated with the chosen image.

2. The method of claim 1, wherein the at least two images from the test MRI scan comprises at least two images from the first test set.

3. The method of claim 1, wherein the at least two images from the test MRI scan comprises at least one image from the first test set and at least one image from the second test set.Attorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-24. The method of claim 3, wherein the at least two images from the test MRI scan comprises at least two images from the first test set and at least two images from the second test set.

5. The method of any one of claims 1-4, wherein the at least two images from the test MRI scan comprises all the images of the test MRI scan.

6. The method of any one of claims 1-5, wherein the method further comprises executing on the processor the further steps of: receiving a request that a user desires to view a second section of the patient medical record; and displaying the second section of the patient medical record.

7. The method of any one of claims 1-6, wherein the method further comprises executing on the processor the further steps of: receiving notice that the user chose a second one of the displayed MR images; and displaying second patient information, wherein the second patient information comprises a first section of the second patient medical record associated with the second chosen MR image.

8. The method of any one of claims 1-7, wherein the MRI scan further comprises data regarding the MRI sequence used to obtain the MR images, wherein the method further comprises executing on the processor the further steps of: optionally receiving a request that a user desires to view data regarding the MRI sequence used to obtain the MR image; and displaying the data regarding the MRI sequence used to obtain the MR image.

9. The method of any one of claims 1-8, wherein the method further comprises executing on the processor the further step of: collecting user feedback data comprising: (i) how the user responds to the displayed two or more MR images, (ii) how the user responds to the displayed patient information, or (iii) a combination thereof.

10. The method of claim 9, wherein the user feedback data comprises an MR image being chosen by the user for displaying the patient information.

11. The method of any one of claims 9-10, wherein the user feedback data comprises an MR image being flagged as relevant, potentially relevant, irrelevant, or a combination thereof by the user.Attorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-212. The method of any one of claims 9-11, wherein the user feedback data comprises the amount of time a user was shown an MR image, the patient information, or a combination thereof.

13. The method of any one of claims 9-12, wherein the user feedback data comprises a request from the user to display a second section of the patient medical record.

14. The method of any one of claims 9-13, wherein the method further comprises executing on the processor the further step of: retraining the trained model based on the collected user feedback data.

15. The method of any one of claims 1-14, wherein the patient information comprises the age and sex of the patient.

16. The method of any one of claims 1-1 , wherein the patient information comprises a reason for recording the MRI scan.

17. The method of any one of claims 1-16, wherein the patient information comprises the identity and optionally the results of additional medical tests performed on the patient after the MRI scan was recorded.

18. The method of any one of claims 1-17, wherein the patient information comprises a diagnosis of a condition based on the MRI scan.

19. The method of claim 18, wherein the diagnosis is an ICD-10 diagnosis.

20. The method of any one of claims 1-19, wherein the patient information comprises a written analysis of the MRI scan.

21. The method of any one of claims 1-20, wherein the patient information comprises a summary of the patient medical record.

22. The method of claim 21, wherein the summary was created with generative artificial intelligence (Al) based on the patient medical record.

23. The method of claim 22, wherein the method further comprises executing on the processor the further step of: creating the summary of the patient medical record with generative Al.

24. The method of claim 23, wherein the generative Al is a natural language processing (NLP) generative Al.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-225. The method of any one of claims 1-24, wherein the trained model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof.

26. The method of claim 25, wherein the trained model comprises a similarity metric.

27. The method of any one of claims 1-26, wherein the MRI scan is a brain MRI scan.

28. The method of any one of claims 1-27, wherein the test MRI scan and the database are located on computer storage devices that are within 20 km of one another.

29. The method of any one of claims 1-28, wherein the test MRI scan and the database are located on the same private network.

30. The method of any one of claims 1-29, wherein the test MRI scan and the database are located on the same local area network (LAN).

31. The method of any one of claims 1-30, wherein the test MRI scan and the database are located on different computers that communicate with one another using private internet protocol addresses.

32. A non-transitory computer readable storage medium with computer executable instructions stored thereon executed by a processor to perform a method of selecting and displaying historical magnetic resonance imaging (MRI) images and associated historical patient medical records that are related to a current test MRI scan, wherein an MRI scan comprises: a first set comprising a plurality of two-dimensional images that were each recorded using the same first MRI sequence as each other, but recorded along different planes of a patient's body; and a second set comprising a plurality of two-dimensional images that were each recorded using the same second MRI sequence as each other, but recorded along different planes of the patient’ s body, wherein the second MRI sequence is different than the first MRI sequence, wherein the method comprising:(a) receiving the test MRI scan;(b) generating test extracted features from at least two images from the test MRI scan; wherein a database comprises a plurality of entries that each comprise:Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-2 an MRI scan; extracted features associated with images from the MRI scan; a patient medical record associated with the MRI scan;(c) selecting, using a trained model, two or more entries from the database that have extracted features that are similar to the generated test extracted features;(d) instructing a display to visually show at least one image from each of the two or more selected entries;(e) receiving notice that a user chose one of the shown images; and(f) instructing the display to visually show patient information, wherein the patient information comprises a first section of the patient medical record associated with the chosen image.

33. The non-transitory computer readable storage medium of claim 32, wherein the trained model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof.

34. The non-transitory computer readable storage medium of claim 33, wherein the trained model comprises a similarity metric.

35. A method of treating a patient by comparing a test magnetic resonance imaging (MRI) scan of the patient with similar historical MRI scans and associated historical patient information, wherein a computer system comprises a processor and a non-transitory computer readable storage medium of any one of claims 32-34, wherein the method comprises:(i) instructing the processor of the computer system to execute the steps of:(a) receive the test MRI scan;(b) generate test extracted features from at least two images from the test MRI scan; wherein a database comprises a plurality of entries that each comprise: an MRI scan; extracted features associated with images from the MRI scan; a patient medical record associated with the MRI scan;(c) select, using a trained model, two or more entries from the database that have extracted features that are similar to the generated test extracted features;(d) instruct a display to visually show at least one image from each of the two or more selected entries;Attorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-2(ii) viewing the displayed MR images;(iii) choosing one or more of the displayed MR images with the computer system, thereby causing the processor to:(c) receive notice that a user chose one of the shown images; and(f) instruct the display to visually show patient information, wherein the patient information comprises a first section of the patient medical record associated with the chosen image.(iv) viewing the displayed patient information associated with the chosen MR image; and(v) treating the patient based on the test MRI scan, one or more viewed MR images, and one or more viewed displayed patient information.

36. The method of claim 35, further comprising: requesting a second section of the patient medical record from the computer system; and viewing the displayed second section of the patient medical record.

37. The method of any one of claims 35-36, further comprising: choosing a second one or more of the displayed MR images with the computer system; and viewing the second displayed patient information associated with the chosen second MR image.

38. The method of any one of claims 35-27, wherein the treating comprises requesting an additional medical test, performing an additional medical test, or a combination thereof.

39. The method of any one of claims 35-38, wherein the treating comprises diagnosing the patient with a condition.

40. The method of any one of claims 35-39, wherein the treating comprises requesting or performing the administration of a pharmaceutically active ingredient (API) or conducting a surgery.

41. The method of any one of claims 35-40, wherein the trained model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof.

42. The method of claim 41, wherein the trained model comprises a similarity metric.Attorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-243. A computer-implemented method of training a trained model for selecting historical magnetic resonance imaging (MRI) scans that appear to show a medical condition that is radiographically related to a medical condition shown on a current test MRI scan, wherein an MRI scan comprises: a first set comprising a plurality of two-dimensional images that were each recorded using the same first MRI sequence as each other, but recorded along different planes of a patient’ s body; and a second set comprising a plurality of two-dimensional images that were each recorded using the same second MRI sequence as each other, but recorded along different planes of the patient’s body, wherein the second MRI sequence is different than the first MRI sequence, wherein the method comprises:(a) obtaining a first training set comprising MRI scans;(b) training, with a processor and the first training set, a model to select an MRI scan that is likely to show the same medical condition by extracting features from each of two or more images of the MRI scan.

44. The method of claim 43, wherein the method further comprises: creating a second training set comprising the MRI scans and the user feedback data; retraining the trained model with the second training set.

45. The method of any one of claims 43-44, wherein the model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof.

46. The method of claim 45, wherein the model comprises an artificial neural network.

47. The method of claim 45, wherein the model comprises a similarity metric.

48. The method of any one of claims 43-47, wherein the training is performed with a machine learning algorithm, a deep learning algorithm, a deep neural network algorithm, a convolutional neural network algorithm, a metric learning algorithm, an attention-based algorithm, or a combination thereof.

49. The method of claim 48, wherein the training is performed with a metric learning algorithm and optionally one or more additional algorithms.Attorney Reference: UCSF-802WO Client Reference: SF2024-158-PCT-250. The method of claim 49, wherein the metric learning algorithm is a supervised metric learning algorithm from similar and dissimilar pairs.

51. The method of any one of claims 43-50, wherein the MRI scans comprise brain MR images.

52. A non-transitory computer readable storage medium with computer executable instructions stored thereon executed by a processor to perform a method of training a trained model for selecting historical magnetic resonance imaging (MRI) scans that appear to show a medical condition that is radiographically related to a medical condition shown on a current test MRI scan, wherein an MRI scan comprises: a first set comprising a plurality of two-dimensional images that were each recorded using the same first MRI sequence as each other, but recorded along different planes of a patient’s body: and a second set comprising a plurality of two-dimensional images that were each recorded using the same second MRI sequence as each other, but recorded along different planes of the patient’s body, wherein the second MRI sequence is different than the first MRI sequence, wherein the method comprising:(a) obtaining a first training set comprising MR images; and(b) training, with a processor and the first training set, a model to select an MRI scan that is likely to show the same medical condition by extracting features from each of two or more images of the MRI scan.

53. The non-transitory computer readable storage medium of claim 52, wherein the method further comprises: receiving user feedback data comprising: (i) how the user responds to the displayed two or more MRI scans, (ii) how the user responds to the displayed at least some of the patient medical record, or (iii) a combination thereof; creating a second training set comprising the MRI scans and the user feedback data; and retraining the trained model with the second training set.

54. The non-transitory computer readable storage medium of any one of claims 52-53, wherein the model comprises a similarity metric, an artificial neural network, a deep neural network, a deep learning network, a convolutional neural network, or a combination thereof.Attorney Reference: UCSF-802WOClient Reference: SF2024-158-PCT-255. The non-transitory computer readable storage medium of claim 54, wherein the model comprises an artificial neural network.

56. The non-transitory computer readable storage medium of claim 54, wherein the model comprises a similarity metric.

57. The non-transitory computer readable storage medium of any one of claims 52-56, wherein the training is performed with a machine learning algorithm, a deep learning algorithm, a deep neural network algorithm, a convolutional neural network algorithm, a metric learning algorithm, an attention-based algorithm, or a combination thereof.

58. The non-transitory computer readable storage medium of claim 57, wherein the training is performed with a metric learning algorithm and optionally one or more additional algorithms.

59. The non-transitory computer readable storage medium of claim 58, wherein the metric learning algorithm is a supervised metric learning algorithm from similar and dissimilar pairs.

60. The non-transitory computer readable storage medium of any one of claims 52-59, wherein the MRI scans comprise brain MR images.

Images