Prompt-based guidance for medical professionals

Abstract

A method of providing guidance to medical professionals with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base. The method comprises receiving, at a first electronic device, a prompt from a first medical professional, and converting the prompt into a machine-searchable string. A trained machine learning model is directed to retrieve one or more results from the authoritative knowledge base. A query may be transmitted to a second medical professional if the trained machine learning model determines that the one or more results are not sufficiently relevant. A reply to the query from the second medical professional may be received, and the reply may be outputted on the first electronic device.

Description

PROMPT-BASED GUIDANCE FOR MEDICAL PROFESSIONALSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and all the benefits of United States Provisional Patent Application No. 63 / 741,572, filed on January 3, 2025, the entire contents of which are hereby incorporated by reference.BACKGROUND

[0002] Large language models (LLM) offer a variety of benefits in healthcare by processing and analyzing vast amounts of data and providing insights that enhance diagnostic accuracy and personalized treatment plans. LLMs may be used in many different applications for the healthcare industry. Some applications may include providing intelligent answers to prompts from medical professionals, summarizing and updating medical records, responding to medical emergencies, or streamlining communication between medical professionals.

[0003] While LLMs offer numerous advantages, they also come with certain disadvantages. One concern is the potential for inaccurate data that the LLM is trained on, which can lead to erroneous or flawed generated content. This may be particularly problematic in the healthcare industry, where new research and treatment protocols are frequently emerging to better harness new technologies and treat novel patient conditions.

[0004] Retrieval-augmented generation (RAG) has been used with LLMs generally to integrate real-time data retrieval into the text generation process. As such, there exists a need in the healthcare industry to leverage RAG to enhance the capabilities of LLMs to better provide patient care and to help streamline the workflows of medical professionals.SUMMARY

[0005] The present disclosure is generally directed to enhancing the accuracy of the LLM by integrating RAG to search external knowledge sources such as specific hospital guidelines, latest best practices, clinical databases, and / or subject matter expert provided knowledge.

[0006] According to a first aspect, a method of providing guidance to medical professionals with a system is disclosed. The system includes electronic devices connected on anetwork, one or more processors, and a database storing an authoritative knowledge base. The method includes receiving, at a first electronic device, a prompt from a first medical professional. The prompt is converted into a machine- searchable string. The method further comprises directing, via retrieval-augmented generation, a trained machine learning model to retrieve one or more results from the authoritative knowledge base. A query is transmitted to a second electronic device via the network and to a second medical professional if the trained machine learning model determines that the one or more results are not sufficiently relevant. A reply to the query is received from the second medical professional via the second electronic device. The method includes transmitting, to the first electronic device via the network, the reply or output based on the reply. The reply or the output is displayed on the first electronic device.

[0007] According to a second aspect, a method of providing guidance to medical professionals is disclosed. The system includes electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base. The method includes receiving, at a first electronic device, a prompt from a first medical professional. The one or more processors convert the prompt into a machine-searchable string. Via retrieval-augmented generation, a trained machine learning model is directed to retrieve one or more results from the authoritative knowledge base. A relevancy score is assigned to the one or more results. The method further includes comparing the relevancy scores of the one or more results against a relevancy threshold. If a predefined minimum number of the one or more results are less than the relevancy threshold, the method further includes transmitting, to a second electronic device via the network, a query to a second medical professional. A reply to the query from the second medical professional is received via the second electronic device. The reply or output is transmitted, based on the reply, to the first electronic device via the network. The reply or the output may be displayed on the first electronic device.

[0008] According to a third aspect, a method of facilitating compliance during a medical procedure is provided. The system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base. The method includes monitoring, with one or more of the electronic devices, actions taken by one or more medical professionals during the medical procedure. The actions are converted by the one or more processors into machine- searchable strings. A procedure type is determined based on the machine-searchable strings. The method further includes directing, via retrieval- augmented generation, atrained machine learning model to retrieve a compliance protocol associated with the procedure type from the authoritative knowledge base. Instances of noncompliance are identified by comparing the actions taken by the one or more medical professionals with associated aspects of the compliance protocol. The instances of noncompliance may be logged in the database. The log of the instances of noncompliance may be displayed on the one or more of the electronic devices or another display.

[0009] According to a fourth aspect, a method of facilitating compliance during a medical procedure is provided. The system includes electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base. The method includes receiving, at the one or more processors, an input or an indication of a determined abnormality of one or more steps of the medical procedure. The determined abnormality is converted, with the one or more processors, into a machine-searchable string. A trained machine learning model is directed to retrieve, via retrieval-augmented generation, a remedial protocol associated with the determined abnormality from the authoritative knowledge base. The method further includes displaying, on the one or more of the electronic devices or another display, steps of the remedial protocol. Subsequent actions taken by one or more medical professionals may be monitored with one or more of the electronic devices. Instances of noncompliance may be monitored by comparing the subsequent actions taken by the one or more medical professionals with the steps of the remedial protocol. The method may include logging the instances of noncompliance in the database, and displaying, on one or more of the electronic devices or another display, the log of the instances of noncompliance.

[0010] According to a fifth aspect, a method of providing guidance to medical professionals is disclosed. The system includes electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base. The method includes receiving, at a first electronic device connected, a prompt from a first medical professional. A query is transmitted, to a second electronic device via the network, to a second medical professional based on the prompt. A reply from the second medical professional may be received via the second electronic device. The method further includes directing, via retrieval-augmented generation, a trained machine learning model to compare the reply with the authoritative knowledge base. The reply from the second medical professional may be transmitted to a third medical professional via a third electronic device based on the comparison. The method furtherincludes receiving, via the third electronic device, confirmation from the third medical professional that the reply from the second medical professional is compliant with acceptable medical practices. Thereafter, the authoritative knowledge base is updated with the reply from the second medical professional. The reply to the first medical professional may be displayed on the first electronic device.BRIEF DESCRIPTION OF THE FIGURES

[0011] FIG. 1 is a functional block diagram of a communications system in a hospital setting.

[0012] FIG. 2 is a functional block diagram of a RAG-based system.

[0013] FIG. 3 is a flowchart of a method of providing guidance to medical professionals.

[0014] FIG. 4 is a flowchart of another method of providing guidance to medical professionals including a relevancy score and a relevancy threshold.

[0015] FIG. 5 is a flowchart of a method of facilitating compliance to a compliance protocol during a medical procedure.

[0016] FIG. 6 is a flowchart of a method of facilitating compliance to a remedial protocol during a medical procedure.

[0017] FIG. 7 is a flowchart of a method of providing guidance to medical professionals during the medical procedure, and asynchronously updating an authoritative knowledge base.

[0018] FIG. 8 is a flowchart of a method of providing guidance to medical professionals, and displaying results based on quantity of protocol contexts from an authoritative knowledge base.

[0019] FIG. 9 is a flowchart of another method of providing guidance to medical professionals, and displaying results based on quantity of protocol contexts from an authoritative knowledge base.DETAILED DESCRIPTION

[0020] In a hospital setting, effective communication helps ensure that optimal care is delivered to each patient. Clear and timely exchanges of information among healthcareprofessionals, support staff, and software interfaces of medical equipment help make sure that medical staff are well-informed and coordinated.

[0021] Communication can originate from various sources and be directed to multiple receivers, some of which are illustrated in FIG. 1. FIG. 1 shows a functional block diagram depicting a communication system 10 showing an overview of communication paths between various aspects within a hospital setting. The communication system 10 includes a patient 12 that is connected to a monitoring system 14. The monitoring system 14 may include a mobile tablet 16, which may display pertinent alerts based on a condition of the patient 12. The monitoring system 14 may also send alerts to a communications hub 18, which helps route alerts pertaining to the condition of the patient 12 to the appropriate receivers. These receivers include medical professionals 20, a patient information record 30 showing the medical history for the patient 12, mobile devices 28, a vision dashboard 22, phones / SMS 24, and / or other devices 26. The communication system 10 shown in FIG. 1 is illustrative of a hospital setting and could apply to different settings in the medical industry that rely on communication between medical professionals 20 over a network and may include other sources and receivers of communication that are not depicted in FIG. 1.

[0022] The patient 12 can be any individual receiving medical care in a hospital setting. This may include surgery in an operating room, preoperative observations, and / or postoperative observations, to name a few. The monitoring system 14 may be any system that is designed to monitor conditions or possible complications of the patient 12, which may include any number of sensors (e.g., EKG, load cells, temperature sensors, motion sensors, etc.) attached to or working with the patient 12 to detect alerts. The monitoring system 14 may track vital signs such as heart rate, blood pressure, oxygen saturation, respiratory rates, and the like. The monitoring system 14 may include thresholds for displaying the detected alerts and may be modified based on the specifics of the patient 12 and / or the preferences of the medical professionals 20. The monitoring system 14 may further include a camera with machine vision to detect the patient 12, one or more of the medical professionals 20, and / or objects within the operating room to provide such alerts. The monitoring system 14 may also include a display such as the mobile tablet 16, which helps illustrate and visualize the status of the patient 12. The monitoring system 14 helps with providing real-time data to the medical professionals 20. enabling them to deliver precise and effective care. As previously mentioned, the communications hub 18 helps route alerts sent by the monitoringsystem 14 to the appropriate receivers. The communications hub 18 will be discussed in greater detail below. One of the receivers that the communications hub 18 may route alerts to are the medical professionals 20 themselves.

[0023] The medical professionals 20 are professionals that work within the hospital setting. The medical professionals 20 may have mobile devices 28 enabling them to receive alerts from the communications hub 18. The mobile devices 28 may also permit the medical professionals 20 to acknowledge that they received any alerts and to manually escalate to another medical professional to sufficiently respond to the alert. The medical professionals 20 may be nurses, doctors, respiratory therapists, radiologists, emergency response teams, and the like. The mobile devices 28 may include smartphones, tablets, pagers, wearable devices, or other devices that help the medical professionals 20 use to help them send and receive information that enhance their ability to provide timely and effective patient care.

[0024] Continuing to refer to FIG. 1, the communications hub 18 may also send information to update the patient information record 30. The update information may include comprehensive documentation such as the nature of the alert, risk factors associated with the alert, lab results, or changes to the condition of the patient 12. Details about the time and circumstances of the alert, the immediate actions taken by the medical professionals 20, and any subsequent changes in the treatment plan for the patient 12 may also be recorded. An example of the patient information record 30 is the electronic health record (EHR) which is a digital version of a paper chart of the patient 12, which may contain a wide range of data, including medical history, diagnoses, medications, treatment plans, immunization dates, allergies, radiology images, and laboratory test results.

[0025] Additionally or alternatively, the communications hub 18 may route alerts to the vision dashboard 22, the phones / SMS 24, and / or other devices 26. The vision dashboard 22 may be a data visualization tool designed to provide the medical professionals 20 with a visual representation of various aspects of hospital operations, patient care, and safety protocols. As an example, the vision dashboard 22 may display information about bed occupancy rates, patient How, compliance with safety protocols, and alarm activity for hospital beds and stretchers, to name a few. The phones / SMS 24 are another receiver that may be used to relay alerts and / or information to the medical professionals 20. The phones / SMS 24 may include traditional landline phones that help facilitate reliable and immediate communication among the medical professionals 20. Forexample, this may include direct communication between different departments within the hospital. The phones / SMS 24 may further include text messages sent to the medical professionals 20 that help provide a quick and efficient way for doctors, nurses, and other healthcare staff to share information, coordinate care, and respond to patient needs. The other devices 26 may include any other suitable device that may receive alerts sent from the communications hub 18, which may include pagers and two-way radios.

[0026] In addition to the above capabilities, the communications hub 18 can help in answering prompts from the medical professionals 20, and escalate those prompts to other medical staff by leveraging large language model (LLM) functionality paired with retrieval augmented generation (RAG), which is described in detail below.

[0027] Referring to FIG. 2, a functional block diagram of a RAG-based system 32 is shown. The RAG-based system 32 may be integrated into the communications hub 18 (shown in FIG. 1), or remote computing may be used to perform the methods disclosed herein. As shown in FIG. 2, the RAG-based system 32 receives a prompt 34, which can come in various forms. The prompt 34 can be a question posed by one of the medical professionals 20, for example, through verbal input to a microphone of the mobile device 28. Alternatively, the prompt 34 may be an alert that is sent by the monitoring system 14, such as FIG. 2 showing the prompt 34 as a hemorrhage alert on a visual display 36. The RAG-based system 32, via the LLM functionality, may process the prompt 34 by breaking it down into smaller units called tokens, which are then analyzed through multiple layers of neural networks. These neural networks are trained on vast amounts of data, which is used to understand the context of the prompt 34 by leveraging patterns and knowledge learned during the training.

[0028] However, as mentioned above, the data being used to train the LLM may cause inaccurate responses because the data may not be in lockstep with the latest medical information and / or hospital-specific protocols. As such, the RAG-based system 32 is configured to search relevant information by leveraging an authoritative knowledge base 44 to provide the prompt 34 with additional context. This process is called retrieval where the prompt 34 is converted into an augmented prompt 40. The augmented prompt 40 is then sent to an LLM endpoint 42, which generates a response using both its pre-trained knowledge and the newly retrieved data taken from the authoritative knowledge base 44. As shown in FIG. 2, the authoritative knowledge base 44 may include best practice guidelines, hospital policies, electronic medical records (EMRs), and / orsubject matter expert knowledge, to name a few. In other words, the authoritative knowledge base 44 may be closed or cordoned off from unvetted information, for example, on the world wide web. Other examples (not shown in FIG. 2) of sources of information within the authoritative knowledge base 44 may include local hospital guidelines or policies, guidelines from a governing body such as the Association of periOperative Registered Nurses (AORN), clinical protocols and checklists, and evidence-based guidelines for managing specific conditions. Local hospital guidelines may include emergency response plans to help maintain a safe environment for both patients and staff and administrative procedures, which may govern patient admissions, discharge processes, and electronic health record management to help provide a smooth and efficient hospital operations. It should be appreciated that the authoritative knowledge base 44 may be different than what is illustrated in FIG. 2 and may include other sources of data depending on the specific needs of the users or the hospital. The RAG-based system 32 may be customized with sources of information are stored within the authoritative knowledge base 44 to better provide guidance to the medical professionals 20.

[0029] Moving to FIG. 3, which shows a flowchart of a method 100 of providing guidance to medical professionals 20 with a system including electronic devices connected on a network, one or more processors, and a database storing the authoritative knowledge base 44. The system herein will refer to the RAG-based system 32. The method 100 includes a step 102 of receiving, at a first electronic device, the prompt 34 from a first medical professional. The first electronic device may be the mobile devices 28 (shown in FIG. 1) that are connected to the network, which enables the medical professionals 20 to send and receive information to help provide patient care. As mentioned above, the mobile devices 28 may be wirelessly connected to the network including smartphones, tablets, pagers, specialized healthcare communication platforms such as badges, and the like.

[0030] The prompt 34 sent by the first medical professional may take on various forms depending on the needs of the first medical professional and the complexity of the task. The prompt 34 may be an audio input wherein the first electronic device includes a microphone and the method 100 further includes a step (not shown in FIG. 3) of recording, via the microphone, the audio input from the first medical professional. For example, the audio input may be a question that the first medical professional speaks aloud, which is recorded by the microphone and interpreted as the prompt 34 by the RAG-based system 32.

[0031] The prompt 34 may be a manual input wherein the first electronic device includes a user interface and the method 100 further includes a step (not shown in FIG. 3) of receiving, via the user interface, the manual input from the first medical professional. For example, the manual input may be a question that the first medical professional types into the user interface and is then interpreted as the prompt 34 by the RAG-based system 32.

[0032] Additionally or alternatively, the prompt 34 may be a gesture wherein the first electronic device includes a camera and the method 100 further includes a step (not shown in FIG.3) of capturing, via the camera, the gesture from the first medical professional. The gesture may include hand signals or body movements by the first medical professional. The camera may capture pictures and / or video to record the gesture. For example, the first medical professional may make a hand gesture such as pointing their finger at an area of interest, which may be recorded by the camera and interpreted as the prompt 34 by the RAG-based system 32. It should be appreciated to those skilled in the art that the prompt 34 may take on a number of forms to enhance the ability of the first medical professional to convey the prompt 34 in a timely and efficient manner.

[0033] In addition to the prompt 34 being in different forms, the prompt 34 may also be conveyed in a natural and user-friendly manner. The prompt 34 may be a simple question or a series of questions with multiple parts. The prompt 34 may be one or multiple complete sentences or may be comprised of incomplete sentences. The flexibility in how the first medical professional creates the prompt 34 offers several advantages. It allows the first medical professional to tailor the prompt 34 to match their specific needs by providing more detail if necessary. The flexibility also offers the first medical professional to provide the prompt 34 in a more natural and intuitive way, which may make it easier for the medical professional to describe the question they need answered. This may be advantageous because the first medical professional may be in the process of providing care to the patient 12 and can create the prompt 34 in a natural and conversational manner, which may help streamline the workflow of the first medical professional. The prompt 34 may closely mimic everyday language and the typical flow of clinical conversations, which may make it easier for the first medical professional, or any user, to interact with the RAG-based system 32 without any specialized training or technical knowledge. This intuitive approach allows for the first medical professional to quickly and efficiently obtain information, make informed decisions, and streamline their workflow. For example, a doctor might ask, “Best practices for severe blood loss for this patient?” In this case, the doctor does not need to specifically tailor the question in aspecific way. Rather, the doctor can ask the question in a natural way, which helps the doctor quickly access the relevant information that can help support timely and informed decisionmaking.

[0034] Continuing to refer to FIG. 3, the method 100 further includes a step 104 of converting, with the one or more processors, the prompt 34 into a machine-searchable string. As mentioned above, the prompt 34 may be the audio input, the manual input, or the gesture by the first medical professional. Converting the prompt 34 into the machine- searchable string allows the RAG-based system 32 to efficiently process the prompt 34 and match it against vast databases of information. By transforming the prompt 34 into the machine- searchable string, the RAG-based system 32 can quickly identify relevant documents, data points, or knowledge snippets that are pertinent to the prompt 34. This process enhances the accuracy and relevance of the responses by ensuring that users receive precise and contextually appropriate information. Additionally, it optimizes the performance of the RAG-based system 32 by streamlining the search and retrieval operations, ultimately leading to faster and more reliable outputs.

[0035] The method 100 further includes a step 106 of directing, via retrieval-augmented generation, a trained machine learning model to retrieve one or more results from the authoritative knowledge base 44. The method 100 also includes a step 110 of transmitting, to a second electronic device via the network, a query to a second medical professional if the trained machine learning model determines that the one or more results are not sufficiently relevant. It should be appreciated that the transmitted query to the second medical professional could be based on a single retrieved result. The method 100 may also optionally include a step 108 of displaying, on the first electronic device connected to the network, the one or more results (referred herein as results) if the trained machine learning model determines that the results are sufficiently relevant. The machine learning model determines the relevancy of the results by ensuring that the generated content is factually accurate and grounded in the retrieved documents within the authoritative knowledge base 44. This process may include comparing the generated response with the context provided by the retrieved documents to verify that the information is directly supported by the sources within the authoritative knowledge base 44. Additionally, this process may include consistency checks that help identify any discrepancies between the generated response and the authoritative knowledge base 44, which help in ensuring that the generated response does not contain hallucinations or fabricated details.

[0036] As mentioned above, if the machine learning model determines that the results are sufficiently relevant, then the method 100 may optionally proceed to step 108. The RAG-based system 32 may display the results in a customizable and user-friendly interface that integrates both the sources of information within the authoritative knowledge base 44 and the results. The RAGbased system 32 may also include highlights or links back to the sources of retrieved information within the authoritative knowledge base 44, which may help ensure transparency and trustworthiness of the generated content.

[0037] However, if the machine learning model determines that the results are not sufficiently relevant, the method 100 includes the step 110 of transmitting the query to the second medical professional and may optionally include a step (not shown) of displaying that there were no results, or that the results were not sufficiently relevant. The query may represent the prompt 34 or may represent both the prompt 34 and the results, which may be sent together to the second medical professional. Providing both the prompt 34 and the results can be beneficial as it aids the second medical professional in comprehending the intent of the first medical professional. Additionally, it may offer valuable context on how the RAG-based system 32 interpreted the prompt 34. The query also helps the second medical professional in fact-checking the results, which helps in highlighting any discrepancies or misunderstandings between the prompt 34 and the results.

[0038] Continuing to refer to FIG. 3, the method 100 further includes a step 112 of receiving, via the second electronic device, a reply to the query from the second medical professional. The reply from the second medical professional may include an acknowledgement that the results are sufficiently relevant. The reply may also include supplemental responses to bolster the relevancy of the results to help interpret and answer the prompt 34. Alternatively, the reply may include an answer to the prompt 34 that overrides the results. For example, a junior nurse may send the prompt 34 and the RAG-based system 32 generates the results. The machine learning model deems the results as insufficiently relevant and forwards the query to a senior nurse for review. The senior nurse acknowledges that the results are not sufficiently relevant and the senior nurse sends the reply, which supplants the results.

[0039] The method 100 further includes a step 114 of transmitting, to the first electronic device via the network, the reply or output based on the reply. The method 100 also further includes a step 116 of displaying, on the first electronic device, the reply or the output. Thegenerated response can either be the reply from the second medical professional or the output based on the reply from the second medical professional. In other words, the generated response can either be the reply directly from the second medical professional, which overrode the results because they were not sufficiently relevant, or the generated response can be a combination of the reply from the second medical professional and the results. The reply or the output based on the reply are displayed on the first electronic device to answer the prompt 34 that was initially sent by the first medical professional. The method 100 may optionally further include a step (not shown in FIG. 3) of updating the authoritative knowledge base 44 with the reply or the output based on the reply.

[0040] Moving to FIG. 4, which shows a flowchart of a method 200 of providing guidance to medical professionals 20 during the medical procedure with the system including electronic devices connected on the network, one or more processors, and the database storing the authoritative knowledge base 44. The method 200 is analogous to the method 100 shown in FIG.3 but includes additional steps in how the machine learning model determines the relevancy of the results. The method 200 includes a step 202 of receiving, at the first electronic device, the prompt 34 from the first medical professional, which is analogous to the step 102 of the method 100 shown in FIG. 3. As previously mentioned, the prompt 34 may include the audio input recorded by the microphone of the first electronic device, the manual input into the user interface of the first electronic device, and / or the gesture captured by the camera of the first electronic device. The method 200 further includes a step 204 of converting, with the one or more processors, the prompt 34 into the machine-searchable string, which is analogous to the step 104 of the method 100 shown in FIG. 3. The method 200 further includes a step 206 of directing, via retrieval-augmented generation, the trained machine learning model to retrieve one or more results from the authoritative knowledge base 44, which is analogous to the step 106 of the method 100 shown in FIG. 3. It should be appreciated that the trained machine learning model may retrieve a single result from the authoritative knowledge base 44.

[0041] The method 200 further includes a step 208 of assigning a relevancy score to the one or more results (referred herein as results). As previously mentioned, the machine learning model determines the relevancy of the results by ensuring that the generated content is factually accurate and grounded in the retrieved documents within the authoritative knowledge base 44 to verify that the generated content does not contain hallucinations or fabricated details. This processmay further include assigning the relevancy score to the results by evaluating how accurate and grounded the results are within the authoritative knowledge base 44. This evaluation can involve various metrics, such as semantic similarity, coverage of key points, and coherence, each of which are described in detail below.

[0042] Semantic similarity measures how semantically close the generated response is to the retrieved documents, which may include techniques such as Bidirectional Encoder Representations from Transformers (BERTScore), which is an advanced evaluation metric for text generation tasks to measure the similarity between two pieces of text. Another technique to help with semantic similarity may include Recall-Oriented Understudy for Gisting Evaluation (ROUGE), which measures the overlap of n-grams between the generated text and reference text. N-grams are continuous sequences of n items (typically words) from a given text. ROUGE uses n-grams to measure the overlap between the generated text and reference text, which helps evaluate the quality of the generated responses. Another technique to help with semantic similarity may include Metric for Evaluation of Translation with Explicit Ordering (METEOR), which considers synonyms, stemming, and paraphrasing to help provide an accurate relevancy score. The RAGbased system 32 may use the techniques described above or may use other techniques not discussed herein to determine the relevancy score of the results.

[0043] Coverage of key points measures whether the key points from the retrieved documents within the authoritative knowledge base 44 are covered in the generated response. This helps ensure that the response is comprehensive and informative. The RAG-based system 32 may assess coverage of key points by how well the results addresses the various facets of the prompt 34. This may involve decomposing the prompt 34 into sub-questions that cover different aspects of the main question being asked in the prompt 34, which may include categorizing sub-questions into core, background, and follow-up types.

[0044] Coherence is measured by evaluating the logical flow and consistency of the results. This involves ensuring that the results are factually accurate and make sense as a whole, with each part of the response logically following from the previous parts. The system may use various techniques to achieve this, such as checking for grammatical correctness, maintaining a consistent narrative, and ensuring that the response stays on topic.

[0045] It should be noted that the RAG-based system 32 may employ a variety of techniques to evaluate the results and determine the relevancy score. The choice of techniques canvary depending on the specific application and requirements. While the methods described, such as semantic similarity, coverage, and coherence, may be used, they are not exhaustive. Those skilled in the art will appreciate that other techniques may also be valuable. The flexibility to select and combine different evaluation methods allows for a more tailored and effective assessment of the results, ensuring that the RAG-based system 32 meets diverse needs and contexts.

[0046] The method 200 further includes a step 210 of comparing the relevancy scores of the results against a relevancy threshold. The relevancy threshold helps establish a baseline for what is considered relevant. The relevancy threshold may be adjusted based on the requirements of the RAG-based system 32 and user preferences. For instance, in some cases, users might prioritize higher precision to ensure that only the most relevant information is retrieved, which may set the relevancy threshold to a higher value. However, in other scenarios, the relevancy threshold may be set to a lower value to help capture a broader range of relevant content for the results. It should be appreciated to those skilled in the art that the relevancy threshold is a flexible parameter that can be fine-tuned to balance the trade-offs between precision and recall, which helps the RAG-based system 32 meet the specific needs and expectations of its users.

[0047] Continuing to refer to FIG. 4, the method 200 may optionally further include a step 212 of displaying, on the first electronic device connected to the network, the results if the assigned relevancy score of the results is higher than the relevancy threshold. As such, steps 214-220 may be optional if the relevancy score is higher than the relevancy threshold. If the RAGbased system 32 determines, via the aforementioned techniques, that the results exceeds the relevancy threshold, the results are displayed for the first medical professional who queried the RAG-based system 32 with the prompt 34.

[0048] The method 200 further includes a step 214 of transmitting, to a second electronic device via the network, a query to a second medical professional if a predefined minimum number of the results are less than the relevancy threshold. The query may include the prompt 34, the results, and the relevancy score of the results. The predefined minimum number is a metric that may be customized by the users of the RAG-based system 32. The predefined minimum number may be a low value to help capture only the most relevant generated content, or the predefined minimum number may be a higher value to help capture a larger number of results that exceed the relevancy threshold. As mentioned, the threshold of whether there is a response that is sufficiently relevant may be modified based on the specifics of the patient or the medicalstaff preferences as detected by any number of sensors attached to or working with the patient to detect alerts.

[0049] The method 200 further includes a step 216 of receiving, via the second electronic device, a reply to the query from the second medical professional. The method 200 further includes a step 218 of transmitting, to the first electronic device via the network, the reply or output based on the reply. The method 200 further includes a step 220 of displaying, on the first electronic device, the reply or the output. The steps 216, 218, 220 of method 200 are analogous to the steps 112, 114, 116 of the method 100 (shown in FIG. 3). The method 200 may optionally further include a step (not shown in FIG. 4) of updating the authoritative knowledge base 44 with the reply or the output based on the reply. Additionally, the update may occur asynchronously, which means the updates to the authoritative knowledge base 44 may occur independently of other tasks within the RAG-based system 32. This helps ensure that other tasks by the RAG-based system 32 are not delayed while the update occurs.

[0050] Healthcare compliance helps ensure that patients ultimately receive proper treatment. Healthcare compliance may encompass adherence to a wide range of local, state, and federal regulations designed to ensure patient safety, prevent fraud, waste, and abuse, and maintain high standards of care. Healthcare compliance involves following established guidelines and protocols to minimize risks, improve patient outcomes, and ensure that healthcare services are delivered safely and effectively. Adhering to established guidelines helps standardize procedures, which reduces variability in patient care and helps minimize the likelihood of errors. This standardization helps create a more predictable and safer environment for both patients and healthcare providers. Noncompliance, on the other hand, can compromise patient safety, leading to adverse health outcomes, increased infection rates, and overall poor quality of care. Harnessing LLMs with RAG-enabled techniques can help bolster adherence and compliance to various protocols and regulations.

[0051] FIG. 5 shows a flowchart of a method 300 of facilitating compliance during the medical procedure with the system including electronic devices connected on the network, one or more processors, and the database storing the authoritative knowledge base 44. The method 300 includes a step 302 of monitoring, with one or more of the electronic devices, actions taken by one or more medical professionals during the medical procedure.

[0052] The actions taken by the medical professionals 20 during the medical procedure may include preparation, wherein the medical professionals 20 set up the necessary equipment and ensure the patient 12 is ready for the medical procedure. For example, the medical professionals 20 may include patient assessment, where the medical professionals 20 review the medical history of the patient 12, current medications, and any allergies to tailor the medical procedure accordingly. Another example of preparation is equipment sterilization, which may involve the thorough cleaning and sterilization of all instruments to help prevent infection. Yet another example is the preparation of the surgical environment by ensuring it is clean, organized, and equipped with all necessary supplies. Other examples may include preoperative instructions that may be provided to the patient 12, such as fasting guidelines and medication adjustments.

[0053] The actions taken by the medical professionals 20 may include assessment of the condition and vital signs of the patient 12 to tailor the medical procedure to the specific needs of the patient 12. For example, the medical professionals 20 may perform a physical examination, where vital signs such as blood pressure, heart rate, and respiratory rate are measured. It should be appreciated that devices used to perform physical examinations could be part of the monitoring system 14. Additionally, assessment may include laboratory tests like blood work and imaging studies to help provide a comprehensive view of the health of the patient 12. The actions taken by the medical professionals 20 may further include documenting the assessment of the patient 12, which may include dictating notes to the RAG-based system 32. For example, the medical professionals 20 may dictate their clinical observation of the condition of the patient 12, which may include recommendations for medication or other treatment.

[0054] The actions taken by the medical professionals 20 may further include intervention, where the medical professionals 20 perform the necessary surgical actions to complete the medical procedure. For example, intervention may include surgical steps during an appendectomy where the medical professionals 20 make a small incision in the lower right abdomen to access the appendix, separating the appendix from the surrounding tissue and blood vessels, and finally removing the detached appendix from the body.

[0055] The actions taken by the medical professionals 20 may further include administering post-procedure care to the patient 12. This may include helping ensure the recovery of the patient 12 is successful regarding wound care, pain management, and instructions on followup care. It should be noted that the previously mentioned actions by the medical professionalsduring the medical procedure are not exhaustive and may include other actions that help provide a successful outcome for the medical procedure.

[0056] The electronic devices connected to the network may include the microphone, wherein the step 302 of monitoring the subsequent actions taken by the one or more medical professionals 20 further comprises listening, via the microphone, to audible discussions between the one or more medical professionals. Audible discussions between medical professionals 20 during the medical procedure may help ensure that all team members are aware of the status of the patient 12, which helps in making timely and accurate decisions. Clear communication helps coordinate the actions of the surgical team, which helps reduce the risk of errors and enhancing the efficiency of the medical procedure. Audible discussions also allow for the immediate sharing of information, such as unexpected findings or complications, enabling the medical professionals to respond swiftly and appropriately. The audible discussions may further include dictating the initial assessment of the patient 12 or dictating notes or observations as treatment of the patient 12 progresses.

[0057] Additionally or alternatively, the electronic devices may include the user interface wherein the step 302 of monitoring the subsequent actions taken by the one or more medical professionals 20 further comprises receiving, via the user interface, the manual input by the one or more medical professionals. The manual input may be the prompt 34 that is typed into the user interface of the electronic device that the medical professionals 20 use to ask questions or to communicate with other medical professionals 20 that may not be in the immediate vicinity during the medical procedure. The manual input may also be an acknowledgement of an alert that is displayed on the electronic device. In other words, the manual input is an action that is taken by the medical professional via the user input that is related to the patient 12 and / or the medical procedure.

[0058] Continuing with FIG. 5, the method 300 further includes a step 304 of converting, with the one or more processors, the actions into machine- searchable strings. As previously mentioned, by transforming the actions into machine-searchable strings, the RAGbased system 32 can quickly identify relevant documents, data points, or knowledge snippets that are pertinent to the prompt 34 from the first medical professional. This process enhances the accuracy and relevance of the responses by ensuring that users receive precise and contextually appropriate information. Additionally, it optimizes the performance of the RAG-based system 32by streamlining the search and retrieval operations, ultimately leading to faster and more reliable outputs.

[0059] The method 300 further includes a step 306 of determining a procedure type based on the machine- searchable strings. The RAG-based system 32 determines the procedure type by employing natural language techniques to parse the machine-searchable strings to extract key medical terms and phrases. These terms and phrases may be matched against comprehensive medical knowledge, which is stored in the vast amounts of data that trained the LLM, to determine the procedure type. Additionally or alternatively, the RAG-based system 32 may leverage the authoritative knowledge base 44 to determine the procedure type. By identifying the procedure type, the RAG-based system 32 can retrieve and generate content that is tailored to the nuances and requirements of the medical procedure.

[0060] With continuing reference to FIG. 5, the method 300 further includes a step 308 of directing, via retrieval-augmented generation, a trained machine learning model to retrieve a compliance protocol associated with the procedure type from the authoritative knowledge base 44. The compliance protocol may be specific to the procedure type or the compliance protocol may be a formalized, written guideline that the medical professionals follow to maintain consistent and high standards of care. For example, the compliance protocol may be the Fast-Track Protocol for cardiac surgery, which includes preoperative patient education, optimized anesthesia techniques, early extubation, and postoperative pain management. Another example of the compliance protocol is the Universal Protocol for Preventing Wrong Site, Wrong Procedure, and Wrong Person Surgery, which is designed to help prevent surgical errors by requiring three individual steps: conducting a pre-procedure verification process, marking the surgical site, and performing a time-out immediately before the procedure begins. Another example of the compliance protocol is the World Health Organization (WHO) Surgical Safety Checklist, which aims to improve the safety of surgical procedures by ensuring that critical safety steps are follow. The examples of compliance protocols listed above are illustrative and it should be appreciated to those skilled in the art that other compliance protocols may be references by the RAG-based system 32.

[0061] The method 300 further includes a step 310 of identifying instances of noncompliance by comparing the actions taken by the one or more medical professionals 20 with associated aspects of the compliance protocol. The method 300 further includes a step 312 of logging the instances of noncompliance in the database. The method 300 also may include a step314 of displaying, on the one or more of the electronic devices or another display, the log of the instances of noncompliance.

[0062] Instances of noncompliance may come in various forms and levels and may involve actions or audible discussions by the medical professionals 20 that do not adhere to specific steps outlined in the compliance protocol. For example, noncompliance may be failing to properly follow sterilization techniques outlined in the compliance protocol. Another example could be not adhering to medication administration protocols, such as incorrect dosing or timing of antibiotics, which may lead to ineffective infection prevention. Another example could be audible discussions between the medical professionals 20 that involve subsequent steps during the medical procedure that do not adhere to the compliance protocol. Identified noncompliance may also include dictated notes (e.g.. initial clinical observations or ongoing treatment updates) that deviate from the requirements of the compliance protocol, whether by omitting required information, prescribing / administering an incorrect type or dosage of medication, including extraneous or premature assessments, or documenting actions in a manner inconsistent with established procedural steps.

[0063] The RAG-based system 32 logs the instances of noncompliance into the database, which may include logging at least one of the actions taken by the one or more medical professionals, the procedure type, a section of the compliance protocol pertaining to the noncompliance, and a timestamp. Logging the instances of noncompliance may help in maintaining high standards of patient care and safety. By systematically documenting instances of noncompliance, hospitals can conduct thorough root cause analyses to understand why the instances of noncompliance occurred and implement corrective actions to help prevent recurrence. Additionally, logging the instances of noncompliance may help in building a culture of transparency and accountability within the organization.

[0064] The method 300 may optionally include a step (not shown in FIG. 5) of transmitting, via the one or more electronic devices, the instances of noncompliance and the section of the compliance protocol pertaining to the instances of noncompliance to a senior medical professional. The method 300 may further include receiving, via the one or more electronic devices, a confirmation from the senior medical professional and transmitting, via the one or more electronic devices, the confirmation from the senior medical professional to the one or more medical professionals 20.

[0065] There may be instances of noncompliance that may be justified or correct given the circumstances regarding the medical procedure. During a medical emergency, strict adherence to the compliance protocol might be bypassed to save the life of the patient 12. For example, if the patient 12 arrives at the hospital needing emergency life-saving surgery, some steps in the compliance protocol may be necessary to skip such as a comprehensive preoperative checklist, as an example. In such cases, the goal is to stabilize the condition of the patient 12 and to prevent further complications. As another example, the medical professionals 20 may be faced with a novel medical situation such as an outbreak of a virus where strict adherence to the compliance protocol may be impractical or impossible. Medical professionals may need to modify standard protocols to prioritize urgent medical procedures while ensuring patient safety. In other words, there may be situations where the instances of noncompliance may be appropriate and correct. The senior medical professional may be a hospital employee with expertise and authority to review the instances of noncompliance and provide oversight and accountability. The senior medical professional may send the confirmation back to the medical professionals 20, which may be displayed on the electronic devices.

[0066] Moving to FIG. 6, a flowchart of a method 400 of providing guidance to the medical professionals 20 during the medical procedure with the system including electronic devices connected on the network, one or more processors, and the database storing the authoritative knowledge base 44 is shown. The method 400 includes a step 402 of receiving, at the one or more processors, an input or an indication of a determined abnormality of one or more steps of the medical procedure. The determined abnormality may be an unexpected or unusual finding that deviates from the normal or anticipated course of action during the medical procedure. For example, during an abdominal surgery intended to remove the appendix from the patient 12, the medical professionals 20 may discover an unexpected tumor in a different part of the abdomen. The determined abnormality may alter the surgical plan for the medical procedure that may require immediate decisions about whether to address the determined abnormality or to schedule additional medical procedures. Other examples of determined abnormalities include unforeseen complications during the medical procedure such as excessive bleeding or adverse reactions to anesthesia. It should be appreciated to those skilled in the art that the determined abnormalities may vary widely depending on many factors such as the specific procedure type and the overall health of the patient 12.

[0067] The method 400 includes a step 404 of converting, with the one or more processors, the determined abnormality into the machine- searchable string. As previously mentioned, converting into the machine-searchable string helps the RAG-based system 32 to quickly identify relevant documents, data points, and / or knowledge snippets that are pertinent to the determined abnormality during the medical procedure.

[0068] Continuing to refer to FIG. 6. the method 400 further includes a step 406 of directing, via retrieval-augmented generation, the trained machine learning model to retrieve a remedial protocol associated with the determined abnormality from the authoritative knowledge base 44. The remedial protocol may include structured approaches for the medical professionals 20 to follow for addressing the determined abnormality in standardized way. Approaching the determined abnormality in a standardized way may help in optimizing outcomes by making sure that the medical professionals 20 are on the same page. Additionally, standardized responses may help in allowing for better communication between the medical professionals 20 in their analysis of the determined abnormality. The remedial protocol is stored in the authoritative knowledge base 44 to help ensure that the remedial protocol is up-to-date and comprehensive, which may help in reducing errors by leveraging the most recent and accurate information in addressing the determined abnormality.

[0069] The method 400 further includes a step 408 of displaying, on the one or more of the electronic devices or another display, steps of the remedial protocol. Depending on the specific determined abnormality during the medical procedure, the remedial protocol may include multiple steps to effectively address the abnormality. For example, the remedial protocol may include steps pertaining to stabilizing the patient 12. Following stabilization, the remedial protocol may include steps pertaining to determining the cause and extent of the determined abnormality, which may involve diagnostic tests or consultations with other medical professionals. The remedial protocol may further include steps focusing on corrective measures, which may include surgical interventions or other medical treatments. Additionally, the remedial protocol may further include post-procedure monitoring and follow-up care to help ensure the recovery of the patient 12 and help prevent reoccurrence. The steps of the remedial action may be displayed on the electronic devices, which may include the mobile devices 28 (shown in FIG. 1) that may include smartphones, tablets, pagers, specialized healthcare communication platforms such as badges, and the like. The steps may be displayed on the mobile tablet 16 (also shown in FIG. 1) as well.

[0070] The method 400 further includes a step 410 of monitoring, with one or more of the electronic devices, subsequent actions taken by one or more medical professionals. The subsequent actions taken by the medical professionals 20 may be various actions that help address the determine abnormality. As previously mentioned, subsequent actions may include stabilizing the patient 12, performing additional diagnostic tests, consulting with other medical professionals, adjusting the surgical plan, and / or scheduling additional surgical procedures, to name a few.

[0071] The method 400 further includes a step 412 of identifying instances of noncompliance by comparing the subsequent actions taken by the one or more medical professionals 20 with the steps of the remedial protocol. As previously mentioned, the remedial protocol may include steps that properly and accurately address the determined abnormality. The RAG-based system 32 compares the subsequent actions of the medical professionals 20 to the steps within the remedial action. This comparison may occur in a step-by-step fashion wherein the RAG-based system 32 assesses whether there are any deviations or instances of noncompliance of the subsequent actions taken.

[0072] The method 400 further includes a step 414 of logging the instances of noncompliance in the database. The method 400 further includes a step 416 of displaying, on one or more of the electronic devices or another display, the log of the instances of noncompliance. Steps 414, 416 of the method 400 are analogous to the steps 312, 314 of the method 300, which is shown in FIG. 5.

[0073] The method 400 may optionally include a step 418 of transmitting, via the one or more electronic devices, the instances of noncompliance and a section of the remedial protocol pertaining to the instances of noncompliance to the senior medical professional. The method 400 may optionally include a step (not shown in FIG. 6) of receiving, via the one or more electronic devices, a confirmation from the senior medical professional. The method 400 may also include a step 420 of transmitting, via the one or more electronic devices, the confirmation from the senior medical professional to the one or more medical professionals 20.

[0074] As previously mentioned, there may be instances of noncompliance that may be justified or correct given the circumstances regarding the medical procedure. During a medical emergency, strict adherence to the compliance protocol might be bypassed to save the life of the patient 12. The RAG-based system 32 may route the instances of noncompliance to the remedial protocol to the senior medical professional so that the subsequent actions may be assessed todetermine if the instances of noncompliance were correct given the circumstances surrounding the determined abnormality. As previously mentioned, the senior medical professional may be a hospital employee with expertise and authority to review the instances of noncompliance and provide oversight and accountability. The senior medical professional may send the confirmation back to the medical professionals 20, which may be displayed on the electronic devices.

[0075] The method 400 may optionally include a step 422 of updating the authoritative knowledge base based on the confirmation from the senior medical professional. If the instances of noncompliance to the remedial protocol are deemed correct by the senior medical professional, the authoritative knowledge base 44 is updated to help maintain the accuracy, relevance, and reliability of the responses and functionality of the RAG-based system 32. For example, the medical professionals may encounter a similar determined abnormality and the authoritative knowledge base 44 may include the previous instances of noncompliance as proper steps within the remedial protocol. Additionally, the update may occur asynchronously, which means the updates to the authoritative knowledge base 44 may occur independently of other tasks within the RAG-based system 32. This helps ensure that other tasks by the RAG-based system 32 are not delayed while the update occurs.

[0076] Communication is an important aspect of the day-to-day operation of a hospital. Communication between the medical professionals 20, which may include doctors and nurses, helps in accurately diagnosing and treating patients. One aspect of that communication collaboration between the medical professionals 20, which may include asking questions. By asking questions, medical professionals 20 can help clarify uncertainties, confirm diagnoses, and discuss treatment plans, which helps in making informed decisions. However, collaboration between the medical professionals 20 can lead to adverse outcomes if incorrect advice is given or miscommunication occurs. For example, there may be a scenario where a junior nurse asks a senior nurse a question and the senior nurse provides a response. However, despite the expertise and experience of the senior nurse, the response was inaccurate due to misinterpreting the question or surrounding circumstances. It is possible that the senior nurse may recognize the inadequacy of their response, which provokes the senior nurse to seek out an additional medical professional, such as a doctor, to help answer the junior nurse’s question. However, it is also possible that the senior nurse may mistakenly think that their response is correct. In either case, double-checking the response from the senior nurse may help in catching such inaccuracies prior to providingtreatment to the patient 12. The RAG-based system 32 may help in streamlining that process by leveraging the authoritative knowledge base 44, which is shown in FIG. 7 and discussed in detail below.

[0077] Referring to FIG. 7, which shows a flowchart of a method 500 of providing guidance to medical professionals during the medical procedure with the system including electronic devices connected on the network, one or more processors, and the database storing the authoritative knowledge base 44. The method 300 includes a step 502 of receiving, at the first electronic device connected, the prompt 34 from the first medical professional. The method 500 further includes a step 504 of transmitting, to the second electronic device via the network, the query to the second medical professional based on the prompt 34. The method 500 further includes a step 506 of receiving, via the second electronic device, the reply from the second medical professional. The method 500 further includes a step of 508 of directing, via retrieval-augmented generation, the trained machine learning model to compare the reply with the authoritative knowledge base 44. To continue the above example scenario, the first medical professional may be the junior nurse and the second medical professional may be the senior nurse.

[0078] As previously discussed, the RAG-based system 32 may leverage the authoritative knowledge base 44 to double-check and compare the reply from the second medical professional to ensure that the reply complies with the up-to-date authoritative sources. As previously mentioned, the authoritative knowledge base 44 may include hospital policies, guidelines of a governing body, clinical databases, entries from a subject-matter expert, and the like.

[0079] The method 500 further includes a step 510 of transmitting, to a third electronic device via the network, the reply from the second medical professional to a third medical professional based on the comparison. The RAG-based system 32 may use various information to determine the third medical professional, which may include context surrounding the prompt 34 from the first medical professional, the reply from the second medical professional, availability of staff, staff roles, experience of staff members, to name a few. To continue the above example scenario, the third medical professional may be a doctor within the hospital who gets sent the reply from the senior nurse based on the comparison of the reply against the authoritative knowledge base 44. The comparison may reveal that the reply from the second medical professional is a slight deviation from the authoritative knowledge base 44. For example, the slight deviation could entailchoosing to use a different type of suture material than what is typically recommended for a specific procedure. While this choice might not significantly impact the overall outcome, it could potentially affect the healing process or increase the risk of minor complications. Alternatively, the comparison may reveal that the reply from the second medical professional is a large deviation from the authoritative knowledge base 44. For example, the large deviation could entail performing a surgery without adhering to established sterile protocols, such as failing to properly sterilize instruments or neglecting to follow hand hygiene guidelines, which may lead to severe infections or prolonged hospital stays.

[0080] The method 500 may optionally include a step (not shown in FIG. 7) of displaying, on the first electronic device, the reply from the second medical professional to the first medical professional if the reply is compliant with the acceptable medical practices found in the authoritative knowledge base 44. This optional step helps streamline the communication process by not necessarily routing the reply from the second medical professional to the third medical professional. For example, if the reply from the senior nurse is compared against the authoritative knowledge base 44 and found to be compliant, there may be no need to forward the reply to the doctor for review.

[0081] With continuing reference to FIG. 7, the method 500 further includes a step 512 of receiving, via the third electronic device, confirmation from the third medical professional that the reply from the second medical professional is compliant with acceptable medical practices. The confirmation from the third medical professional may be to confirm that the reply, despite not being completely compliant with the authoritative knowledge base 44, is correct given the circumstances of the patient 12 and the procedure type.

[0082] The method 500 further includes a step 514 of thereafter, updating the authoritative knowledge base 44 with the reply from the second medical professional. The method 500 further includes a step 516 of displaying, on the first electronic device, the reply to the first medical professional. As previously mentioned, an important aspect of the authoritative knowledge base 44 is that it houses information that reflects latest developments, trends, and data, which may be particularly important within the medical field. After the third medical professional confirms that the reply from the second medical professional is correct taking into account the comparison between the reply and the authoritative knowledge base 44, the authoritative knowledge base 44 is updated with the reply from the second medical professional and displayed on the first electronicdevice for the first medical professional to view. As previously mentioned, the update may occur asynchronously, which means the updates to the authoritative knowledge base 44 may occur independently of other tasks within the RAG-based system 32. This helps ensure that other tasks by the RAG-based system 32 are not delayed while the update occurs. Continuing the example scenario, after the confirmation from the doctor and the update to the authoritative knowledge base 44, future replies from the senior nurse to answer a similar prompt from the junior nurse will not be flagged as being inconsistent with the authoritative knowledge base 44. This may help streamline the communication process by not having to forward the reply from the senior nurse to the doctor.

[0083] As previously discussed, the RAG-based system 32 accesses the authoritative knowledge base 44, which may include best practice guidelines, hospital policies, electronic medical records (EMRs), subject matter expert knowledge, and the like. A prompt may be conveyed in a natural and user-friendly manner and could include a simple question, a series of questions with multiple parts, and / or can include incomplete sentences. This flexibility allows the medical professionals 20 to enter a given prompt without having to structure their questions in a rigid format, which may help in saving time and mental effort during a medical procedure or other clinical work. However, this flexibility may also cause the RAG-based system 32 to generate or provide multiple relevant responses using differing protocols, which may leave a medical professional uncertain which protocol to follow. Such a concept may be considered an ambiguous query.

[0084] Referring to FIG. 8, which shows a flowchart of a method 600 of providing guidance to medical professionals 20. and displaying results based on the quantity of protocol contexts from the authoritative knowledge base 44. The method 600 includes a step 602 of receiving, at the first electronic device, the prompt 34 from a medical professional, which is analogous to the step 102 of the method 100 shown in FIG. 3. As previously mentioned, the prompt 34 may include the audio input recorded by the microphone of the first electronic device, the manual input into the user interface of the first electronic device, and / or the gesture captured by the camera of the first electronic device. The method 600 further includes a step 604 of converting, with the one or more processors, the prompt 34 into the machine-searchable string, which is analogous to the step 104 of the method 100 shown in FIG. 3. The method 600 further includes a step 606 of directing, via retrieval-augmented generation, the trained machine learning model toretrieve the one or more results from the authoritative knowledge base 44, which is analogous to the step 106 of the method 100 shown in FIG. 3. It should be appreciated that the machine learning model may retrieve a single result from the authoritative knowledge base 44.

[0085] The results retrieved by the RAG-based system 32 in step 606 may be based on multiple protocol contexts within the authoritative knowledge base 44. A protocol context refers to the specific clinical framework or category that defines how a medical protocol applies to a particular situation, patient group, or condition. For example, a protocol context may be based on such factors such as age group (adult vs. pediatric), patient weight, clinical setting (emergency vs. elective), specialty guidelines (cardiology vs. anesthesiology), and the like. A given prompt may correspond to multiple protocol contexts. As such, a given prompt may produce differing results based on different protocol contexts within the authoritative knowledge base 44.

[0086] For example, a medical professional may submit a prompt such as asking for a medication dosage without specifying whether it applies to adults or children. The system 32 retrieves all applicable documents from the authoritative knowledge base 44, encompassing protocol contexts for all age groups rather than restricting the search to a specific demographic. If both adult and pediatric protocols exist, the system 32 does not inherently detect ambiguity in the question itself. Instead, the system 32 infers ambiguity because multiple protocol contexts were returned during the retrieval process. As a result, the method 600 includes a step 608 of displaying the one or more results based on the multiple protocol contexts along with a step 610 of displaying a relevant disclaimer. To continue the above example, the system 32 would present results based on both the adult and pediatric medical protocols, as each provides a valid response to the prompt. Alongside the displayed results, the system 32 provides a disclaimer indicating that the guidance reflects available protocols and may not encompass all patient groups or clinical circumstances. This disclaimer helps promote transparency and helps reinforces the need for professional judgment in applying the information.

[0087] Alternatively, the results retrieved by the system 32 in step 606 may be based on a single protocol context. The method 600 includes a step 612 of displaying the results based on a single protocol context along with a step 614 of displaying a relevant disclaimer. A single protocol context would be when the authoritative knowledge base 44 contains guidance for only one patient group or scenario relevant to the prompt. For example, if the medical professional submitted a prompt requesting the medication dosage and the authoritative knowledge base 44only included a protocol context for adult patients, the system 32 will display the medication dosage exclusively for adult patients. Additionally, the system 32 will display the disclaimer indicating that the guidance reflects available protocols corresponding to adult dosages and may not encompass all patient groups or clinical circumstances. If the medical professional is treating a pediatric patient and receives adult dosage guidance along with the accompanying disclaimer, the medical professional would be informed that the information applies only to adults and may not be suitable for their case.

[0088] In situations where multiple protocol contexts exist within the authoritative knowledge base 44, the system 32 may prompt the medical professional for additional details to refine the query and deliver more precise guidance. As such, FIG. 9 shows another method 700 of providing guidance to medical professionals 20, and displaying results based on quantity of relevant protocol contexts from the authoritative knowledge base 44.

[0089] The method 700 includes a step 702 of receiving, at the first electronic device, the prompt 34 from a medical professional, which is analogous to the step 602 of the method 600 shown in FIG. 8. The method 700 further includes a step 704 of converting, with the one or more processors, the prompt 34 into the machine- searchable string, which is analogous to the step 604 of the method 600 shown in FIG. 8. The method 600 further includes a step 706 of directing, via retrieval- augmented generation, the trained machine learning model to retrieve the one or more results from the authoritative knowledge base 44, which is analogous to the step 606 of the method 600 shown in FIG. 8. It should be appreciated that the trained machine learning model may retrieve a single result from the authoritative knowledge base 44.

[0090] If the results retrieved by the system 32 in step 706 are based on a single protocol context, the method 700 proceeds with a step 716 of displaying the one or more results based on a single protocol context along with a step 718 of displaying a relevant disclaimer, which are analogous to the steps 612, 614 of the method 600 shown in FIG. 8.

[0091] However, if the results retrieved by the system 32 in step 706 are based on multiple protocol contexts, the method 700 proceeds to a step 708 of displaying a clarifying query based on the multiple protocol contexts. The clarifying query asks the medical professional for additional details (e.g., patient age, weight, or clinical setting) to help the system 32 provide results based on a single protocol context. For example, if the medical professional submits a prompt requesting medication dosage without specifying the patient’s age, the system 32 would retrieveresults based on multiple protocol contexts with some results tailored for adult patients and other results for pediatric patients. Before presenting these results, the system 32 displays a clarifying query to confirm the patient’s age, enabling it to narrow the search and display guidance based on a single protocol context. As such, the method 700 further includes a step 710 of receiving a response from the medical professional intended to narrow the results to a single protocol context.

[0092] There may be instances where the response by the medical professional may not immediately help the system 32 narrow down to a single protocol context. When ambiguity remains after the first clarifying query, the method 700 may include an additional step (not shown) of displaying a subsequent clarifying query for more details from the medical professional. For example, the first clarifying query might request the patient’s age to distinguish between adult and pediatric protocols. If multiple protocols still apply, such as those for different clinical environments, the system 32 may then present a second clarifying query asking for the specific clinical setting, such as emergency care versus outpatient treatment. This iterative clarification process helps ensure that the system 32 progressively refines its understanding of the prompt, reducing ambiguity and delivering guidance based on a single protocol context.

[0093] Continuing with FIG. 9, the method 700 further includes a step 712 of displaying the one or more results based on the response and the single protocol context along with a step 714 of displaying the relevant disclaimer. The relevant disclaimer may inform the medical professional that the results are based on the information currently available within the authoritative knowledge base 44 and based on the response to the clarifying query provided by the medical professional. It should be appreciated that between multiple clarifying queries, the system 32 may display interim results accompanied by the relevant disclaimer to help keep the medical professional informed throughout the refinement process. This helps ensure that the user has immediate access to potentially relevant guidance while the system 32 continues to gather additional details to improve accuracy of the results. By providing interim results and relevant disclaimers, the system helps promote transparency and helps reduce the risk of misinterpretation during the iterative clarification process.

[0094] Several methods have been discussed in the foregoing description. However, the methods discussed herein are not intended to be exhaustive or limit the disclosure to any particular form. The terminology which has been used is intended to be in the nature of words ofdescription rather than of limitation. Many modifications and variations are possible in light of the above teachings and the disclosure may be practiced otherwise than as specifically described.

Claims

CLAIMS1. A method of providing guidance to medical professionals with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base, the method comprising:receiving, at a first electronic device, a prompt from a first medical professional: converting, with the one or more processors, the prompt into a machine- searchable string; directing, via retrieval- augmented generation, a trained machine learning model to retrieve one or more results from the authoritative knowledge base;transmitting, to a second electronic device via the network, a query to a second medical professional if the trained machine learning model determines that the one or more results are not sufficiently relevant;receiving, via the second electronic device, a reply to the query from the second medical professional;transmitting, to the first electronic device via the network, the reply or output based on the reply; anddisplaying, on the first electronic device, the reply or the output.

2. The method of claim 1, further comprising displaying, on the first electronic device connected to the network, the one or more results if the trained machine learning model determines that the one or more results are sufficiently relevant.

3. The method of claim 2, wherein the query sent to the second electronic device includes the prompt from the first medical professional and the one or more results.

4. A method of providing guidance to medical professionals with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base, the method comprising:receiving, at a first electronic device, a prompt from a first medical professional; converting, with the one or more processors, the prompt into a machine- searchable string: directing, via retrieval- augmented generation, a trained machine learning model to retrieve one or more results from the authoritative knowledge base;assigning a relevancy score to the one or more results;comparing the relevancy scores of the one or more results against a relevancy threshold; transmitting, to a second electronic device via the network, a query to a second medical professional if a predefined minimum number of the one or more results are less than the relevancy threshold;receiving, via the second electronic device, a reply to the query from the second medical professional;transmitting, to the first electronic device via the network, the reply or output based on the reply; anddisplaying, on the first electronic device, the reply or the output.

5. The method of claim 4, further comprising displaying, on the first electronic device connected to the network, the one or more results if the assigned relevancy score of the one or more results is higher than the relevancy threshold.

6. The method of claim 5, wherein the query sent to the second electronic device includes the prompt from the first medical professional, the one or more results, and the relevancy score of the one or more results.

7. The method of claims 1-6, wherein the first electronic device includes a microphone, the method further comprising:recording, via the microphone, an audio input from the first medical professional; and converting, with the one or more processors, the audio input into the machine- searchable string.

8. The method of claims 1-6, wherein the first electronic device includes a user interface, the method further comprising:receiving, via the user interface, a manual input from the first medical professional; and converting, with the one or more processors, the manual input into the machine-searchable string.

9. The method of claims 1 -6, wherein the first electronic device includes a camera, the method further comprising:capturing, via the camera, a gesture from the first medical professional; and converting, with the one or more processors, the gesture into the machine- searchable string.

10. The method of claims 1-9, wherein the method further comprises updating the authoritative knowledge base with the reply or the output based on the reply.

11. A method of facilitating compliance during a medical procedure with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base, the method comprising:monitoring, with one or more of the electronic devices, actions taken by one or more medical professionals during the medical procedure;converting, with the one or more processors, the actions into machine- searchable strings; determining a procedure type based on the machine-searchable strings;directing, via retrieval- augmented generation, a trained machine learning model to retrieve a compliance protocol associated with the procedure type from the authoritative knowledge base;identifying instances of noncompliance by comparing the actions taken by the one or more medical professionals with associated aspects of the compliance protocol;logging the instances of noncompliance in the database; anddisplaying, on the one or more of the electronic devices or another display, the log of the instances of noncompliance.

12. The method of claim 11, wherein logging the instances of noncompliance in the database comprises logging at least one of the actions taken by the one or more medical professionals, the procedure type, a section of the compliance protocol pertaining to the noncompliance, and a timestamp.

13. A method of facilitating compliance during a medical procedure with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base, the method comprising:receiving, at the one or more processors, an input or an indication of a determined abnormality of one or more steps of the medical procedure;converting, with the one or more processors, the determined abnormality into a machine-searchable string;directing, via retrieval- augmented generation, a trained machine learning model to retrieve a remedial protocol associated with the determined abnormality from the authoritative knowledge base;displaying, on the one or more of the electronic devices or another display, steps of the remedial protocol;monitoring, with one or more of the electronic devices, subsequent actions taken by one or more medical professionals;identifying instances of noncompliance by comparing the subsequent actions taken by the one or more medical professionals with the steps of the remedial protocol;logging the instances of noncompliance in the database; anddisplaying, on one or more of the electronic devices or another display, the log of the instances of noncompliance.

14. The method of claim 13, wherein logging the instances of noncompliance in the database comprises logging at least one of the determined abnormalities, a determined type of the medical procedure, a section of the remedial protocol pertaining to the noncompliance, and a timestamp.

15. The method of claim 13 or 14, further comprising:transmitting, via the one or more electronic devices, the instances of noncompliance and a section of the remedial protocol pertaining to the instances of noncompliance to a senior medical professional;receiving, via the one or more electronic devices, a confirmation from the senior medical professional; andtransmitting, via the one or more electronic devices, the confirmation from the senior medical professional to the one or more medical professionals.

16. The method of claim 15, wherein the method further comprises updating the authoritative knowledge base based on the confirmation from the senior medical professional.

17. The method of any one of claims 11-16, wherein the one or more electronic devices includes a microphone, wherein the step of monitoring the subsequent actions taken by the one or more medical professionals further comprises:listening, via the microphone, to audible discussions between the one or more medical professionals; andconverting, with the one or more processors, the audible discussions into the machine-searchable string.

18. The method of any one of claims 11-16, wherein the one or more electronic devices includes a user interface, wherein the step of monitoring the subsequent actions taken by the one or more medical professionals further comprises:receiving, via the user interface, a manual input by the one or more medical professionals; andconverting, with the one or more processors, the manual input into the machine-searchable string.

19. A method of providing guidance to medical professionals with a system including electronic devices connected on a network, one or more processors, and a database storing an authoritative knowledge base, the method comprising:receiving, at a first electronic device connected, a prompt from a first medical professional;transmitting, to a second electronic device via the network, a query to a second medical professional based on the prompt;receiving, via the second electronic device, a reply from the second medical professional;directing, via retrieval-augmented generation, a trained machine learning model to compare the reply with the authoritative knowledge base;transmitting, to a third electronic device via the network, the reply from the second medical professional to a third medical professional based on the comparison;receiving, via the third electronic device, confirmation from the third medical professional that the reply from the second medical professional is compliant with acceptable medical practices;thereafter, updating the authoritative knowledge base with the reply from the second medical professional; anddisplaying, on the first electronic device, the reply to the first medical professional.

20. The method of claims 1-19, wherein the authoritative knowledge base include data from at least one of local hospital policies, guidelines of a governing body, clinical databases, and entries from a subject matter expert.

21. The method of claims 1-20, wherein the electronic devices are mobile devices wirelessly connected to the network.

Images