Temperature-based endoscopic video recording and curation

The temperature-based endoscopic video recording system addresses inefficiencies in manual recording by automating start/stop based on temperature transitions, enhancing reliability and creating a concise report for clinical use.

WO2026236426A1PCT designated stage Publication Date: 2026-11-12GYRUS ACMI INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/027005
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-07
Filing Date
2026-05-07
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Endoscopic video recording systems require manual operation, which is labor-intensive, prone to human error, and inefficient in capturing critical procedure data, leading to resource wastage and loss of valuable clinical information.

Method used

A temperature-based endoscopic video recording system that automatically starts and stops recording using a temperature sensor to detect transitions between ambient and body temperatures, adjusting recording modes based on temperature variations, and generates a curated procedure report.

Benefits of technology

Enhances recording reliability, reduces human error, optimizes procedure time, and creates a concise, efficient video report for clinical use, improving documentation and reducing storage costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026027005_12112026_PF_FP_ABST
    Figure US2026027005_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Systems and methods for automated endoscopic video recording and curation in an endoscopy procedure are disclosed. An exemplary endoscope system (300) includes an endoscope (310) and a controller circuit (320). The endoscope includes an imaging device (312) to produce images or video streams of a target anatomy, and a temperature sensor (316) to sense temperature during the procedure. The controller circuit triggers a recorder device to start video recording based on a transition from an ambient temperature to a body temperature and stop video recording based on a transition from the body temperature to the ambient temperature, and adjusts the video recording mode in accordance with intra-body temperature variation or treatment-related temperature variation. A procedure video report can be generated using a selected or adapted portion of the recorded images or video streams and procedure metadata or contextual information.
Need to check novelty before this filing date? Find Prior Art

Description

Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 TEMPERATURE-BASED ENDOSCOPIC VIDEO RECORDING AND CURATIONPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 801,476, filed May 7, 2025, the contents of which are incorporated herein by reference.FIELD OF THE DISCLOSURE

[0002] The present document relates generally to endoscopy systems, and more particularly to systems and methods for temperature-based endoscopic video recording and automated video curation.BACKGROUND

[0003] Endoscopes have been used in a variety of clinical procedures, including, for example, illuminating, imaging, detecting and diagnosing one or more disease states, providing fluid delivery (e.g., saline or other preparations via a fluid channel) toward an anatomical region, providing passage (e.g., via a working channel) of one or more therapeutic devices or biological matter collection devices for sampling or treating an anatomical region, and providing suction passageways for collecting fluids (e.g., saline or other preparations), among other procedures. Examples of such anatomical region can include gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary duct, intestines, colon, and the like), renal area (e.g., kidney(s), ureter, bladder, urethra) and other internal organs (e.g., reproductive systems, sinus cavities, submucosal regions, respiratory tract), and the like.

[0004] Some endoscopes include a working channel through which an operator can perform suction, placement of diagnostic or therapeutic devices (e.g., a brush, a biopsy needle or forceps, a stent, a basket, or a balloon), or minimally invasive surgeries such as tissue sampling or removal of unwanted tissue (e.g., benign or malignant strictures) or foreign objects (e.g., calculi). Some endoscopes can be used with a laser or plasma system to deliver energy to an anatomical target (e.g., soft or hard tissue or calculi) to achieve desired treatment. For example, laser has been used in applications of tissue ablation,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 coagulation, vaporization, fragmentation, and lithotripsy to break down calculi in kidney, gallbladder, ureter, among other stone-forming regions, or to ablate large calculi into smaller fragments. One example of endoscopy is colonoscopy, which has been shown a potential to reduce incidence and mortality rate of colorectal cancer. Colonoscopy is typically performed with rapid advance of a colonoscope to the cecum, and an endoscopist can perform thorough inspection to identify any anomalies (e.g., polyps) and to perform necessary treatment (e.g., polypectomy) during the withdrawal of the colonoscope.

[0005] Endoscope systems generally have image or video recording and still photography capabilities that allow an endoscopist or other human specialists to manually record images or video streams during the procedure. An imaging device, such as a camera, can be incorporated into an endoscope to take live images or video streams. The imaging device can operate under a preset imaging mode to obtain live images or video streams. The endoscope systems may also allow a user (e.g., and endoscopist or other clinical staff) to document the scenes or other findings during an endoscopy procedure, which is an important and valuable feature for diagnosis, treatment planning, patient education, and record-keeping.SUMMARY

[0006] Endoscope systems generally have standard video recording and still photography capabilities. A user can manually start and stop image or video recording (such as by pressing a button on an imaging device or other tools or devices, or activating from a user interface of the endoscope system), or manually change the imaging or video recording mode or operating state, during the procedure. Manual video recording may require more human effort and take more time. For example, while an endoscopist is focusing on the procedure, additional staff member(s) may be required to take charge of video recording during the procedure. A nurse may always start and stop recording for certain procedure types, or the endoscopist may provide a verbal cue to the nurse to start and stop recording. In addition to extra labor and time involved, manual video recording may be prone to human operation errors due to certain human factors, such as fatigue, distraction, or lack of training. Inadvertent miss of recording, or inappropriate start and / or stop recording times, may cause loss of criticalDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 procedure data, or recording and storing data of little clinical value (e.g., data recorded when the endoscope is outside the patient such as during the procedure preparation or post withdrawal of the endoscope from the patient) which can be a source of inefficient use of clinical and device resources such as storage space.

[0007] The present inventors have recognized an unmet need for more reliable and efficient techniques to record endoscopic images and video data during a procedure. The present document describes temperature-based endoscopic video recording and automated video curation in an endoscopy procedure. An exemplary endoscope system includes an endoscope having an imaging device to produce images or video streams of a target anatomy, and a temperature sensor to sense an ambient temperature when the endoscope is outside the patient and a body temperature when the endoscope is inside the patient. A controller circuit of the system to trigger a recorder device to start recording the images or video streams based on a first transition from the ambient temperature to the body temperature, and to stop recording the images or video streams based on a second transition from the body temperature to the ambient temperature. The temperature sensor may further sense an intra-body temperature variation between the target anatomy and a non-target anatomy inside the body, and / or a temperature variation in relation to treatment of the target anatomy. In accordance with such intra-body temperature variation or treatment-related temperature variation, the controller circuit can adjust an imaging or video recording mode (e.g., one or more image or video quality parameters). The controller circuit may select or adapt a portion, less than an entirety, of the recorded images or video streams, and generate a procedure video report using the selected or adapted portion of the recorded images or video streams along with procedure metadata and contextual information about one or more workflow events during the procedure. The procedure report can be presented to a user, assist in subsequent diagnosis and / or treatment planning, clinical trial recruitment, physician training, patient education, or serve as a guide to future clinical research.

[0008] Example 1 is an endoscope system, comprising: an endoscope, including: an imaging device to produce images or video streams of a target anatomy of a patient during an endoscopy procedure; and a temperature sensor to sense temperature including an ambient temperature when the endoscope isDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 outside the patient and a body temperature when the endoscope is inside the patient; and a controller circuit configured to monitor the sensed temperature, and to trigger a recorder device to start recording the images or video streams based on a first transition from the sensed ambient temperature to the sensed body temperature, and to stop recording the images or video streams based on a second transition from the sensed body temperature to the sensed ambient temperature.

[0009] In Example 2, the subject matter of Example 1 optionally includes the temperature sensor that can be disposed at a distal tip of the endoscope.

[0010] In Example 3, the subject matter of any one or more of Examples 1-2 optionally include the controller circuit that can be configured to trigger the recorder device to start recording the images or video streams when an increase or a rate of increase in the sensed temperature exceeds a first threshold for a first specific period, and to stop recording the images or video streams when a decrease or a rate of decrease in the sensed temperature exceeds a second threshold for a second specific period.

[0011] In Example 4, the subject matter of any one or more of Examples 1-3 optionally includes the controller circuit that can be further configured to: track procedure time based on respective times of the triggered start of recording and the triggered stop of recording; and predict an outcome of the endoscopy procedure based at least in part on the tracked procedure time.

[0012] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes the controller circuit that can be configured to, based at least in part on a variation of the sensed body temperature, adjust an imaging or video recording mode, including at least one image or video quality parameter, to record the images or video streams.

[0013] In Example 6, the subject matter of Example 5 optionally includes the controller circuit that can be configured to: establish, or receive, a correspondence between body temperatures and corresponding values of the at least one image or video quality parameter; and adjust the imaging or video recording mode using the sensed body temperature and the established or received correspondence.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0014] In Example 7, the subject matter of any one or more of Examples 5-6 optionally includes wherein the variation of the sensed body temperature includes an intra-body temperature variation between the target anatomy and a non-target anatomy inside the body, wherein the controller circuit is configured to adjust an imaging or video recording mode to increase the image or video quality when the sensed intra-body temperature variation indicates an increased temperature above a threshold.

[0015] In Example 8, the subject matter of any one or more of Examples 5-7 optionally include wherein the variation of the sensed body temperature includes a temperature variation in relation to laser treatment of the target anatomy, wherein the controller circuit is configured to adjust an imaging or video recording mode to increase the image or video quality when the temperature variation in relation to the laser treatment of the target anatomy indicates an increased temperature above a threshold.

[0016] In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes the controller circuit that can be further configured to: select or adapt a portion, less than an entirety, of the recorded images or video streams; and generate a procedure video report using the selected or adapted portion of the recorded images or video streams.

[0017] In Example 10, the subject matter of Example 9 optionally includes the controller circuit that can be further configured to analyze the recorded images or video streams to identify an area of interest (AO I) in the target anatomy during the procedure, and to select or adapt the portion spatially related to the identified AOI.

[0018] In Example 11, the subject matter of Example 10 optionally includes the AOI that can include a pre-determined anatomical landmark or an anomalous structure in the target anatomy.

[0019] In Example 12, the subject matter of Example 11 optionally includes the endoscope that can be a colonoscope including the imaging device to obtain images or video streams of colon segments during a colonoscopy procedure, wherein the AOI includes a colon landmark or anomaly in one of the colon segments.

[0020] In Example 13, the subject matter of any one or more of Examples 9-12 optionally includes the controller circuit that can be furtherDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 configured to generate the procedure video report further using contextual information about one or more workflow events during the procedure.

[0021] In Example 14, the subject matter of any one or more of Examples 9-13 optionally includes the controller circuit that can be further configured to generate the procedure video report further using procedure metadata including information about one or more of the patient, physician or clinical staff, procedural parameters, or a relevant clinical trial.

[0022] In Example 15, the subject matter of any one or more of Examples 1-14 optionally includes a user interface device configured to: provide a notification to a user about the triggered start of recording and the triggered stop of recording of the images or video streams; and receive a user input to modify the triggered start or the triggered stop of recording of the images or video streams.

[0023] In Example 16, the subject matter of any one or more of Examples 1-15 optionally includes the controller circuit that can be configured to: generate or receive at least one trained machine-learning (ML) model each trained using a training dataset comprising ambient temperature measurements and body temperature measurements obtained during endoscopy procedures on a patient population; and apply the sensed ambient temperature and the sensed body temperature of the patient during the procedure to the trained at least one ML model to determine an image or video quality parameter value.

[0024] Example 17 is a method for endoscopic video recording and curation during an endoscopy procedure in a patient using an endoscope that includes an imaging device and a temperature sensor. The method comprises steps of: sensing temperature, including an ambient temperature when the endoscope is outside the patient, and a body temperature when the endoscope is inside the patient using the temperature sensor of the endoscope; triggering a recorder device, via a controller circuit, to start recording images or video streams of a target anatomy of the patient during an endoscopy procedure based on a first transition from the sensed ambient temperature to the sensed body temperature; triggering the recorder device, via the controller circuit, to stop recording the images or video streams based on a second transition from the sensed body temperature to the sensed ambient temperature; and generating aDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 procedure video report using the triggered recording of the images or video streams.

[0025] In Example 18, the subject matter of Example 17 optionally includes triggering the recorder device to start recording the images or video streams that can occur when an increase or a rate of increase in the sensed temperature exceeds a first threshold for a first specific period, wherein triggering the recorder device to stop recording the images or video streams occurs when a decrease or a rate of decrease in the sensed temperature exceeds a second threshold for a second specific period.

[0026] In Example 19, the subject matter of any one or more of Examples 17-18 optionally includes tracking procedure time based on respective times of the triggered start of recording and the triggered stop of recording; and predicting an outcome of the endoscopy procedure based at least in part on the tracked procedure time.

[0027] In Example 20, the subject matter of any one or more of Examples 17-19 optionally includes, based at least in part on a variation of the sensed body temperature, adjusting an imaging or video recording mode, including at least one image or video quality parameter, to record the images or video streams.

[0028] In Example 21, the subject matter of Example 20 optionally includes the variation of the sensed body temperature that can include an intrabody temperature variation between the target anatomy and a non-target anatomy inside the body, wherein adjusting the imaging or video recording mode occurs when the sensed intra-body temperature variation indicates an increased temperature above a threshold to increase the image or video quality.

[0029] In Example 22, the subject matter of any one or more of Examples 20-21 optionally includes the variation of the sensed body temperature that can include a temperature variation in relation to laser treatment of the target anatomy, wherein adjusting the imaging or video recording mode occurs when the temperature variation in relation to the laser treatment of the target anatomy indicates an increased temperature above a threshold to increase the image or video quality.

[0030] In Example 23, the subject matter of any one or more of Examples 17-22 optionally includes, via the controller circuit: selecting orDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 adapting a portion, less than an entirety, of the recorded images or video streams; and generating the procedure video report using the selected or adapted portion of the recorded images or video streams.

[0031] In Example 24, the subject matter of Example 23 optionally includes analyzing the recorded images or video streams and identifying therefrom an area of interest (AO I) in the target anatomy during the procedure, and selecting or adapting the portion, less than the entirety, of the recorded images or video streams that is spatially related to the identified AOI.

[0032] In Example 25, the subject matter of any one or more of Examples 23-24 optionally includes generating the procedure video report further using at least one of: contextual information about one or more workflow events during the procedure; or procedure metadata including information about one or more of the patient, physician or clinical staff, procedural parameters, or a relevant clinical trial.

[0033] In Example 26, the subject matter of any one or more of Examples 17-25 optionally includes providing a notification on a user interface about the triggered start of recording and the triggered stop of recording of the images or video streams; and receiving a user input from the user interface to modify the triggered start or the triggered stop of recording of the images or video streams.

[0034] In Example 27, the subject matter of any one or more of Examples 17-26 optionally includes generating or receiving at least one trained machine-learning (ML) model each trained using a training dataset comprising ambient temperature measurements and body temperature measurements obtained during endoscopy procedures on a patient population; and applying the sensed ambient temperature and the sensed body temperature of the patient during the procedure to the trained at least one ML model to determine an image or video quality parameter value.

[0035] This summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. Other aspects of the disclosure will be apparent to persons skilled in the art upon reading and understanding the following detailed description and viewing the drawings thatDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 form a part thereof, each of which are not to be taken in a limiting sense. The scope of the present disclosure is defined by the appended claims and their legal equivalents.BRIEF DESCRIPTION OF THE DRAWING

[0036] FIGS. 1-2 are schematic diagrams illustrating an example of an endoscope system for use in endoscopy procedures, procedures, such as a colonoscopy procedure.

[0037] FIG. 3 illustrates an example of an endoscope system with temperature-based endoscopic video recording and automated video curation capabilities.

[0038] FIG. 4 illustrates an example process of generating a procedure report of a colonoscopy procedure.

[0039] FIG. 5 illustrates an example process of selecting and editing images or video segments and using the same to create a procedure report.

[0040] FIGS. 6A-6B illustrate examples of training a machine learning (ML) model, and generating a procedure report comprising video documentation with text or audio overlay using the trained ML model.

[0041] FIG. 7 is a flowchart illustrating an example method of temperature-based endoscopic video recording and automated video curation in an endoscopy procedure.

[0042] FIG. 8 is a block diagram illustrating an example machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform.DETAILED DESCRIPTION

[0043] This document describes systems, devices, and methods for temperature-based endoscopic video recording and automated video curation in an endoscopy procedure. An exemplary endoscope system includes an endoscope having an imaging device to produce images or video streams of a target anatomy, and a temperature sensor to sense temperature during the procedure, including an ambient temperature when the endoscope is outside the patient and a body temperature when the endoscope is inside the patient. A controller circuit of the system can trigger a recorder device to start videoDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 recording based on a first transition from the ambient temperature to the body temperature, and to stop video recording based on a second transition from the body temperature to the ambient temperature. The controller circuit can adjust a video recording mode in accordance with intra-body temperature variation or treatment-related temperature variation. The controller circuit may generate a procedure video report using a selected or adapted portion of the recorded images or video streams, and contextual information and procedure metadata. The procedure report can be used in diagnosis and / or treatment planning, clinical trial recruitment, physician training, patient education, or for guiding clinical research.

[0044] The systems, devices, and methods described herein may be used in various endoscopy procedures to improve reliability and efficiency of endoscopic image and video recording during a procedure. Compared to conventional manual recording, automatic sensor-triggered video recording, such as based on a temperature sensor that senses transitions between an ambient room temperature and a body temperature of the patient during the procedure, can reduce human burden and procedure time (e.g., by avoiding or reducing button clicks or other types of manual inputs during the procedure), and prevent or reduce human errors (e.g., missed video recording by mistake, or clinical distraction of manually triggering start and stop of procedure), thereby allowing the surgeon and staff to focus on the patient care thus improving the procedure outcome. The sensor-triggered start and stop of video recording also facilitates automatic tracking of procedure time, which can be an predictor or indicator of procedure outcome. Another aspect of the present document relates to automated video curation, where a selected or adapted images or video segments, along with the contextual information and procedure metadata, can be integrated into a comprehensive procedure video report. In an example, lengthy videos can be curated and transformed into a concise (e.g., 60-second) report incorporating selected images or video segments with contextual information of key procedure workflow events (manually identified and / or automatically detected), including operator (e.g., endoscopist) observations, commentary, and actions during the procedure, patient vital signs and medical events occurred, device settings and usage of consumables, and machine-generated insights. The automated video curation can help reduce downstream video storage expense and processingDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 costs. In certain examples, the curated procedure report or an aspect thereof can be created using artificial intelligence (Al) or machine-learning (ML) techniques. The automatically curated procedure video report enables more effective and efficient procedure documentation, fast retrieval of critical information for diagnosis and subsequent treatment planning, archival for longterm storage with low footprint, patient education, and physician training. It may also be used to optimize individualized patient treatment, assist in clinical trial recruitment, or serve as a guidance for clinical research.

[0045] In accordance with various embodiments, the endoscope system as described herein may be designed for efficient communication with medical institutions, archiving, and enhancing patient records. For example, the automated video documentation system proposed herein (or a portion thereof) may be integrated into existing endoscopy equipment, such as via an edgecomputing device mounted on the endoscopy tower. The edge-computing device may be connected to an electronic health record (EHR) system, or a hospital integration system. The edge-computing device may capture live video feed through a video port from the endoscope’s video processor (CV), such as a serial digital interface (SDI) connection for transmitting digital video and audio signals using coaxial or fiber optic cables). The recorded procedure video may be processed in real-time or post-procedure, depending on the setup and requirements. The summarized video can then be converted into proper format (e.g., Digital Imaging and Communications in Medicine, or DICOM) and send to the EHR or other hospital integration system.

[0046] Although the discussion in this document focuses endoscope systems and methods of using the same in endoscopy procedures, this is meant only by way of example and not limitation. It is within the contemplation of the inventors, and within the scope of this document, that the systems, devices, and methods pertaining to automated video documentation as discussed herein may also be used in other medical examination and diagnostic procedures involving imaging capabilities including, for example, fluoroscopy, ultrasound imaging, magnetic resonance imaging and radiography procedures, among others. For example, during an endobronchial ultrasound (EBUS) procedure, ultrasound image analysis of the lungs and nearby lymph nodes can be used for landmark orDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 nodule detection and characterization, and one or more procedure reports can be generated using the automated video documentation as described herein.

[0047] FIG. 1 is a schematic diagram of an endoscope system 10 for use in an endoscopy procedure, such as colonoscopy. The system 10 can include an imaging and control system 12 and an endoscope 14. The system 10 is an illustrative example of an endoscope system suitable for use with the systems, devices, and methods described herein, such as a colonoscopy system for use in image-guided colonoscopy with automated video documentation as described in this document.

[0048] The endoscope 14 can be insertable into an anatomical region for imaging or to provide passage of or attachment to (e.g., via tethering) one or more sampling devices for biopsies or therapeutic devices for treatment of a disease state associated with the anatomical region. The endoscope 14 can interface with and connect to imaging and control system 12. The endoscope 14 may be a colonoscope, though other types of endoscopes can be used with the features and teachings of the present disclosure. The imaging and control system 12 can include a control unit 16, an output unit 18, an input unit 20, a light source unit 22, a fluid source 24, and a suction pump 26.

[0049] The imaging and control system 12 can include various ports for coupling with the endoscope system 10. For example, the control unit 16 can include a data input / output port for receiving data from and communicating data to the endoscope 14. The light source unit 22 can include an output port for transmitting light to the endoscope 14, such as via a fiber optic link. The fluid source 24 can include a port for transmitting fluid to the endoscope 14. The fluid source 24 can include, for example, a pump and a tank of fluid or can be connected to an external tank, vessel, or storage unit. The suction pump 26 can include a port to draw a vacuum from the endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and the input unit 20 can be used by an operator of the endoscope system 10 (e.g., an endoscopist) to control functions of the endoscope system 10 and view the output of the endoscope 14. The control unit 16 can also generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical output, acoustic output, fluid output, and the like forDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 treating the anatomical region with, for example, cauterizing, cutting, freezing, and the like.

[0050] The fluid source 24 can be in communication with control unit 16 and can include one or more sources of air, saline, or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels, or the like) and connectors (barb fittings, fluid seals, valves, or the like). The fluid source 24 can be utilized as an activation energy for a biasing device or a pressure-applying device of the present disclosure. The imaging and control system 12 can also include the drive unit 46, which can include a motorized drive for advancing a distal section of endoscope 14.

[0051] The endoscope 14 can include an insertion section 28, a functional section 30, and a handle section 32, which can be coupled to a cable section 34 and a coupler section 36. The insertion section 28 can extend distally from the handle section 32, and the cable section 34 can extend proximally from the handle section 32. The insertion section 28 can be elongated and can include a bending section and a distal end to which the functional section 30 can be attached. In some examples, the functional section 30 can be an portion at least partially integrated into the distal end of the insertion section 28. The bending section can be controllable (e.g., by a control knob 38 on the handle section 32) to maneuver the distal end through tortuous anatomical passageways (e.g., stomach, duodenum, kidney, ureter, etc.). The insertion section 28 can also include one or more working channels (e.g., an internal lumen) that can be elongated and can support the insertion of one or more therapeutic tools of the functional section 30. The working channel can extend between the handle section 32 and the functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires, can also be provided by the insertion section 28 (e.g., via suction or irrigation passageways or the like).

[0052] A coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the input unit 20, the light source unit 22, the fluid source 24, and the suction pump 26.

[0053] The handle section 32 can include the knob 38 and the port 40A. The knob 38 can be connected to a pull wire or other actuation mechanisms that can extend through the insertion section 28. The port 40 A, as well as other ports,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 such as a port 40B (FIG. 2), can be configured to couple various electrical cables, guide wires, auxiliary scopes, tissue collection devices, fluid tubes, and the like to the handle section 32, such as for coupling with the insertion section 28.

[0054] According to examples, the imaging and control system 12 can be provided on a mobile platform (e.g., a cart 41) with shelves for housing the light source unit 22, the suction pump 26, an image processing unit 42 (FIG. 2), etc. Alternatively, several components of the imaging and the control system 12 (shown in FIGS. 1 and 2) can be provided directly on the endoscope 14 to make the endoscope “self-contained.”

[0055] The functional section 30 can include components for treating and diagnosing anatomy of a patient. The functional section 30 can include an imaging device, an illumination device, and an elevator. The functional section 30 can further include optically enhanced biological matter and tissue collection and retrieval devices. For example, the functional section 30 can include one or more electrodes conductively connected to the handle section 32 and functionally connected to the imaging and control system 12 to analyze biological matter in contact with the electrodes based on comparative biological data stored in the imaging and control system 12. In other examples, the functional section 30 can directly incorporate tissue collectors.

[0056] In some examples, the endoscope 14 can be robotically controlled, such as by a robot arm attached thereto. The robot arm can automatically, or semi-automatically (e.g., with certain user manual control or commands), via an actuator, position and navigate the endoscope 14 (e.g., the functional section 30 and / or the insertion section 28) in the target anatomy or position a device at a desired location with desired posture to facilitate an operation of an anatomical target. In accordance with various examples discussed in this document, a controller can generate a control signal to the actuator of the robot arm to facilitate anomaly inspection and diagnosis under the target or recommended imaging mode in a robotically assisted endoscopy procedure.

[0057] FIG. 2 is a schematic diagram of the endoscope system 10 of FIG.1 including the imaging and control system 12 and the endoscope 14. FIG. 2 schematically illustrates components of the imaging and the control system 12Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 coupled to the endoscope 14, which in the illustrated example includes a colonoscope. The imaging and control system 12 can include the control unit 16, which can include or be coupled to an image processing unit 42, a treatment generator 44, and a drive unit 46, as well as the light source unit 22, the input unit 20, and the output unit 18. The control unit 16 can include, or can be in communication with, an endoscope, a surgical instrument 48, and an endoscope system, which can include a device configured to engage tissue and collect and store a portion of that tissue and through which imaging equipment (e.g., a camera) can view target tissue via inclusion of optically enhanced materials and components. The control unit 16 can be configured to activate a camera to view target tissue distal of the endoscope system. Likewise, the control unit 16 can be configured to activate the light source unit 22 to shine light on the surgical instrument 48, which can include select components configured to reflect light in a particular manner, such as enhanced tissue cutters with reflective particles.

[0058] The coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the image processing unit 42 and the treatment generator 44. In examples, the port 40A can be used to insert another surgical instrument 48 or device, such as a daughter scope or auxiliary scope, into the endoscope 14. Such instruments and devices can be independently connected to the control unit 16 via the cable 47. In examples, the port 40B can be used to connect coupler section 36 to various inputs and outputs, such as video, air, light, and electric.

[0059] The image processing unit 42 and light source unit 22 can each interface with the endoscope 14 (e.g., at the functional section 30) by wired or wireless electrical connections. The imaging and control system 12 can accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 can include the light source unit 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). The imaging and control system 12 can connect (e.g., via an endoscope connector) to the endoscope 14 for signal transmission (e.g., light output fromDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 light source, video signals from imaging system in the distal end, diagnostic and sensor signals from a diagnostic device, and the like).

[0060] The treatment generator 44 can generate a treatment plan, which can be used by the control unit 16 to control the operation of the endoscope 14, or to provide with the operating physician a guidance for maneuvering the endoscope 14, during an endoscopy procedure. In an example, the treatment generator 44 can generate an endoscope navigation plan, including estimated values for one or more cannulation or navigation parameters (e.g., an angle, a force, etc.) for maneuvering the steerable elongate instrument, using patient information including an image of the target anatomy. The endoscope navigation plan can help guide the operating physician to cannulate and navigate the endoscope in the patient anatomy. The endoscope navigation plan may additionally or alternatively be used to robotically adjust the position, angle, force, and / or navigation of the endoscope or other instrument.

[0061] FIG. 3 illustrates an example of an endoscope system 300 with temperature-based endoscopic video recording and automated video curation capabilities. The endoscope system 300 may be used in an endoscopy procedure to automatically record image or video data, perform substantially real-time scene analysis and identify critical anatomical and pathologic findings, and create a procedure report including selected or adapted images or video segments, contextual information about one or more workflow events during the procedure (e.g., an event log, surgeon’s observation, clinical notes, and action log, etc.), and procedure metadata (e.g., information about one or more of the patient, the physician and relevant clinical staff, procedural parameters such as date, time, facility, equipment, devices, and tools used, procedure type, environment, among other elements involved in carrying out a specific medical procedure, or the clinical trial in which the present patient participates.) By way of example and not limitation, the endoscope system 300 may be used in a colonoscopy procedure. The endoscope system 300 may be implemented as a part of the control unit 16 in FIG. 1.

[0062] The endoscope system 300 may include one or more of an endoscope 310, a contextual information and procedure metadata (CPM) collector 350, a controller circuit 320, a user interface 330, and a storage device 340. The endoscope 310 can be an example of the endoscope 14 as describedDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 above and shown in FIGS. 1-2. The endoscope 310 may include, among other things, an imaging system 312 and a lighting system 314. The imaging system 312 can include at least one imaging sensor or device (e.g., a camera) configured to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure. The imaging sensor or device may be located at a distal portion or a distal end of the endoscope 310. The imaging system 312 may be controllably adjusted to operate on different settings including zoom settings, contrast settings, exposure levels, or viewing angles toward and around the target anatomy. The lighting system 314 may include one or more light sources to produce illumination on the target anatomy via one or more lighting lenses. The lighting system 314 may be controllably adjusted to provide different lighting or illumination conditions. The imaging system 312 and the lighting system 314 together can define an imaging mode for capturing endoscopic images or video streams of the target anatomy.

[0063] The endoscope 310 may include a temperature sensor 316.Examples of the temperature sensor 316 can include a thermocouple, a thermistor, a resistance temperature detector, or a semiconductor-based sensor incorporated in integrated circuits (IC). The temperature sensor 316 can be disposed at a distal portion of the endoscope 310 (e.g., at or proximal to the functional section 30 of endoscope 14 of FIG. 2), such that when portions of the endoscope 310 (e.g., the functional section and at least a portion of the insertion section 28 of endoscope 14) are inserted into the patient body, the temperature sensor 316 is also inside the patient body and sense the body temperature throughout various phases of the procedure (e.g., endoscope insertion phase, observation and / or treatment of the anatomical target (e.g., tissue or an object such as calculi to be observed, sampled, or treated), and endoscope withdrawal phase). Alternatively, in examples where a laser delivery device (e.g., a laser fiber) is passed inside the insertion section of the endoscope to deliver laser pulses to the anatomical target, the temperature sensor 316 may be positioned at a distal portion of the laser delivery device to directly sense patient body temperature throughout various phases of the procedure.

[0064] In addition to sensing the body temperature (Tb) when the endoscope is inside the patient, the temperature sensor 316 can also sense an ambient temperature (Ta, such as temperature of the procedure room) when theDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 endoscope is outside the patient. In some examples, in various phases of the endoscopy procedure while the sensor is inside the patient body, the temperature sensor 316 can sense an intra-body temperature variation, such as a difference between a non-target anatomy and the target anatomy inside the body.Additionally or alternatively, the temperature sensor 316 can sense a temperature variation in relation to treatment of the target anatomy, such as a thermal response (e.g., temperature rise) to laser treatment of the target anatomy. One or more of such temperature changes can be detected by the temperature sensor 316 and, when determined to satisfy a specific condition, may trigger start or stop of recording of image or video streams, or a change or imaging or video recording mode, as will be discussed further below.

[0065] The controller circuit 320 may include circuit sets comprising one or more other circuits or sub-circuits that may, alone or in combination, perform the functions, methods, or techniques described herein. In an example, the controller circuit 320 and the circuits sets therein may be implemented as a part of a microprocessor circuit, which may be a dedicated processor such as a digital signal processor, application specific integrated circuit (ASIC), microprocessor, or other type of processor for processing information including physical activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that may receive and execute a set of instructions of performing the functions, methods, or techniques described herein. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the other components of the circuit set member when the device is operating. In anDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0066] The controller circuit 320 can automatically record images or video streams during the procedure, and create a video report of the endoscopy procedure. The video report may comprise selected or adapted portions of the recorded endoscopic images or video streams, and at least a portion of the contextual information about one or more workflow events during the procedure and procedure metadata collected by the CPM collector 350. The contextual information may include an event log containing records of endoscope setup information, operating modes of one or more devices used in the procedure, or patient responses or medical events occurred during the procedure. The contextual information may additionally or alternatively include an operator (e.g., the endoscopist) observation or action log, such as related to recognition of critical anatomical structures or clinically significant events during the procedure. Examples of the operator observations or actions may include verbal command or commentary during the procedure (e.g., commentary about the findings during the procedure, commentary about patient past and current indications and procedures received, patient demographic and medical information, recognition of landmarks, characterization of anatomical structures, diagnosis of anomalies), manipulation of the endoscope system or one or more devices being used in the procedure (e.g., press of an endo button such as for waterjet or insufflation, actuation of a foot switch, selection of a zooming mode such as for closer mucosal observation, changing of an imaging mode or lighting mode, such as high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), etc., operation of tools fortissue section, sampling, or treatment, among others), or treatment actions taken during the procedure (e.g., polypectomy, clipping, or clotting during the procedure). The procedure metadata may include patient information, clinical staff information, procedure information such as date, time,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 facility, equipment and tools used, and procedure type, etc., or clinical trial information relevant to the present procedure.

[0067] The controller circuit 320 may include an image / video recorder and analyzer 321, an area of interest (AOI) detector 322, a device tracking circuit 323, a procedure summarization circuit 324, and a real-time alert and recommendation circuit 329. The image / video recorder and analyzer 321 can automatically record images or video streams, such as based at least in part on the sensed temperature by the temperature sensor 316. In one example, while the imaging system 312 continuously acquires images or videos (e.g., from preprocedure preparation to post withdrawal of endoscope from the patient body), the image / video recorder and analyzer 321 may start recording the images or video streams (and saved in a storage device) based on a first transition from an ambient temperature (Ta, such as the temperature of the procedure room) to the patient body temperature (Tb), and to stops recording the images or video streams based on a second transition from the body temperature Tb to the ambient temperature Ta. Because the ambient temperature Tais generally lower than the body temperature Tb, to detect the temperature transitions between Taand Tb, the image / video recorder and analyzer 321 may automatically start video recording when an increase (ATa-b) or a rate of increase (ATa-b / At) in the sensed temperature exceeds a first threshold Tthi. The image / video recorder and analyzer 321 may trigger a stop of video recording when a decrease (ATb-a) or a rate of increase (ATb At) in the sensed temperature exceeds a second threshold Tth2. In some examples, the threshold crossing as described above may be coupled with a duration to detect a “sustained” temperature change. For example, the video recording may be started when ATa-b (or ATa.b / At exceeds the first threshold Tthi and sustains for a specified duration tsi . Likewise, the video recording may be stopped when ATb-a(or ATb At) exceeds the second threshold Tth2and sustains for a specified duration tS2. The sustaining periods tsi and tS2 can have predetermined or programmable values. Such sustained temperature change can help reduce false triggers of video recordings (also known as video debris) caused by intermittent thermal disturbance due to, for example, suction, flush, tool use, or environmental or other confounding factors.

[0068] One or more of the thresholds Tthi or Tth2 may be predetermined based on prior knowledge of a difference between the ambient temperature TaDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 and body temperature Tb. Tthi and T*2 can be set to different values, or the same value (Tth). In an example, the ambient temperature Tais approximately 24-26 degrees Celsius, the body temperature Tbis approximately 36-37 degrees Celsius. For a temperature difference of approximately 10 degrees, the thresholds Tthi and Tth2 can be set to approximately 5-6 degrees, such that video recording can be triggered to start when the sensed body temperature Tb exceeds 31-32 degrees (and optionally sustains for a period of approximately 2-5 seconds), and triggered to stop when the sensed body temperature Tb drops below 31-32 degrees (and optionally sustains for a period of approximately 2-5 seconds).

[0069] Referring to FIG. 4, the diagram 400 illustrates, by way of example, events timing, including temperature sensing, video acquisition, and video recording, during an endoscopy procedure. Time instant ti marks the time when the procedure preparation starts but the endoscope is not yet inserted into the patient body. At ti, the temperature sensor 316 is turned on to sense ambient temperature Ta, and the imaging system 312 begins to acquire video signals. The temperature sensing and the video data acquisition can go continuously throughout the procedure. Because the sensed temperature (the ambient temperature Ta) is below the threshold Tth, the image / video recorder and analyzer 321 is not “awake”, such that no video recording is made. At t2, the endoscope is inserted into the patient body. The temperature sensor 316 senses a temperature rise from the ambient temperature Tato the body temperature Tb. When the temperature rise exceeds the threshold Tth, the image / video recorder and analyzer 321 is activated to start recording the video data. The video recording continues until at t3, when the endoscope is withdrawn from the patient body, at which the temperature sensor 316 senses a temperature drop from the body temperature Tb to the ambient temperature Ta. When the temperature drop exceeds the threshold Tth, the image / video recorder and analyzer 321 is activated to stop recording the video. As such, the image / video recorder and analyzer 321 record video only during the time when the endoscope is inside the patient body, during which the sensed temperature remains above the threshold Tth.

[0070] Referring back to FIG. 3, in addition to the temperature-triggered start and stop of video recording, in some examples, the image / video recorderDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 and analyzer 321 may automatically adjust an imaging or video recording mode based on the sensed temperature. The imaging or video recording mode may include one or more image or video quality parameters, such as a frame rate, an image resolution, a bitrate, or shutter speed and aperture of the imaging device, among others. As described above, the temperature sensor 316 can sense an intra-body temperature variation, such as a difference between a non-target anatomy and the target anatomy inside the body. The image / video recorder and analyzer 321 may adjust the imaging or video recording mode in accordance with the sensed intra-body temperature variation. For example, as many of the anatomical targets to be treated are internal tissue or organs that generally have a higher temperature than non-target anatomies (e.g., endoscope entry points or soft tissue along the endoscope insertion or withdrawal pathways away from the target anatomy), when the temperature sensor 316 senses an increase in intra-body temperature above a preset value, it is speculated that the endoscope tip (and the temperature sensor 316 therein) has approached or reached the target anatomy, then the image / video recorder and analyzer 321 may adjust the imaging or video recording mode, such as by increasing the video frame rate and / or resolution, thereby increasing the image or video quality of the target anatomy. In some examples, a correspondence between intra-body temperatures and corresponding image or video quality parameter values can be created, such as using an artificial intelligence (Al) or machine-learning (ML) method. The image / video recorder and analyzer 321 may adjust the imaging or video recording mode by changing one or more image or video quality parameters based on such established correspondence and the actual sensed temperature by the temperature sensor 316.

[0071] In another example, the temperature sensor 316 can sense a temperature variation in relation to treatment of the target anatomy, such as a thermal response (e.g., temperature rise) to treatment of the target anatomy. As similarly described above, the image / video recorder and analyzer 321 may adjust the imaging or video recording mode in accordance with the treatment-related temperature variation. For example, in an endoscopic laser treatment of tissue or calculi structures, when the temperature sensor 316 senses an increase in temperature above a preset value, it is speculated that laser treatment has been delivered and cause a thermal response of the target anatomy. The image / videoDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 recorder and analyzer 321 may accordingly adjust the imaging or video recording mode by increasing, for example, the video frame rate and / or resolution, thereby increasing the image or video quality of the target anatomy. In some examples, a correspondence between treatment-related temperatures and corresponding image or video quality parameter values can be created, such as using an Al or ML method. The image / video recorder and analyzer 321 may adjust one or more image or video quality parameters using such established correspondence and the actual sensed temperature by the temperature sensor 316.

[0072] In addition to temperature, other information may be used to trigger the start and stop, or to change the mode, of video recording. Examples of such input information may include a user input (e.g., verbal or action-based input) or an automated awareness of the beginning and end of a procedure, contextual data such as procedure type or other pre-registered instruction such as collected by the contextual information and procedure metadata collector 350, or an automated detection of anatomy, presence of a particular tool or scope, or an activity of using a particular tool (e.g., snaring) during the procedure, using artificial intelligence (Al) or machine learning (ML) based techniques.

[0073] The image / video recorder and analyzer 321 can analyze the recorded images or video streams, and generate endoscopic image or video features. Examples of the image or video features include statistical features of pixel values or morphological features, such as comers, edges, blobs, curvatures, speeded up robust features (SURF), or scale-invariant feature transform (SIFT) features, among others. In some examples, the image / video recorder and analyzer 321 may pre-process the images or video streams, such as filtering, resizing, orienting, or color or grayscale correction, and the endoscopic features may be extracted from the pre-processed images or video streams. In some examples, the image / video recorder and analyzer 321 may post-process the image features to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges. In various examples, based on the images captured, the image / video recorder and analyzer 321 may determine the location of image sensor within or with respect to the target anatomy in real time during the detected temperature change.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0074] The AOI detector 322 may detect an AOI in the target anatomy based at least in part on the endoscopic image or video streams and identify one or more characteristics of the AOI. The AOI may include one or more predetermined anatomical landmarks in the target anatomy. Systemic endoscopic image documentation of such anatomical landmarks serves the purposes of showing crucial anatomic structures at these landmark regions, documenting the extent of the examination, and reflecting the quality of cleansing and mucosal visualization. In an example of upper GI endoscopy, the pre-determined landmarks may include, for example, some or all of proximal esophagus, distal esophagus, Z-line and diaphragm indentation, cardia and fundus on retroflexed view, body (including lesser curvature), body on retroflexed view, Angulus on partial retroflexion, antrum, duodenal bulb, and second part of the duodenum (including the ampulla). In an example of lower GI (colon, in particular) endoscopy, the pre-determined landmarks may include, for example, some or all of lower part of the rectum in retroflexed view, lower part of the rectum, middle part of the sigmoid, descending colon just distal to the splenic flexure, transverse colon just proximal to the splenic flexure, transverse colon just distal to the hepatic flexure, ascending colon just proximal to the hepatic flexure, cecum and ileocecal valve, and cecum and appendiceal orifice. In an example of colonoscopy procedure, the AOI detector 322 may identify one or more predetermined colon landmarks from images or video streams of one or more colon segments, which can be captured during withdrawal of the colonoscope.

[0075] The AOI to be identified from the endoscopic images or video streams may additionally or alternatively include an anomalous structure in the target anatomy. This may include one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the target anatomy. In an example of colonoscopy, the anomalous structure may include pathological tissue segments, such as mucosal abnormalities (polyps, inflammatory bowel diseases, Meckel’s diverticulum, lipoma, bleeding, vascularized mucosa etc.), or obstructed mucosa (e.g., segments with bad bowel preparation, distended colon etc.).

[0076] Various techniques may be used to detect and characterize an AOI. In an example, the AOI detector 322 can detect an AOI using a templateDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 matching technique involving a comparison of the endoscopic image features (e.g., features characterizing shapes or contours of a structure) to one or more pre-generated templates of known anomalous structure. In another example, the AOI detector 322 can use an Al-driven technique to detect an AOI, where the endoscopic images or video streams (or features extracted therefrom) may be fed into a trained ML model that automatically outputs an AOI identification. The ML model may be trained to establish a correspondence between an endoscopic image or video stream (or features extracted therefrom) and one or more recognizable types or characterizations of landmarks or anomalies. Examples of ML models for recognizing an anatomical landmark include Deep Belief Network, ResNet, DenseNet, Autoencoders, capsule networks, generative adversarial networks, Siamese networks, Convolutional Neural Networks (CNN), deep reinforcement learning, support vector machine (SVM), Bayesian models, decision trees, k-means clustering, among other ML models. Examples of ML models for recognizing anomaly from endoscopic images or video streams include Convolutional Neural Networks, bi-directional LSTM, Recurrent Neural Networks, Conditional Random Fields, Dictionary Learning, or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering), among other ML techniques. The trained ML model for automatically recognizing an AOI may be a part of the trained ML modes 342 stored in the storage device 340. Examples of training an ML model and using the same to detect an AOI are discussed below with respect to FIGS. 6A-6B.

[0077] The device tracking circuit 323 may track in substantially real time location and orientation of the endoscope or a portion thereof in a particular segment of the anatomy during an endoscopy procedure. In an example of colonoscopy, the device tracking circuit 323 can continuously monitor and record the location and orientation of the colonoscope, and align the position and orientation information with a standardized colon location template. In an example, the device tracking circuit 323 can track the location and orientation of an imaging device (e.g., a camera) at a distal portion of the endoscope during the procedure. With such real-time tracking information, the AOI detector 322 may associate a detected AOI (e.g., an anatomical landmark or an anomaly) with the segment of the anatomy. In an example, the device tracking can be based on anDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 electromagnetic property of the imaging device, or the distal portion of the endoscope detected during the endoscopy procedure. The device tracking circuit 323 may register a tracked location of the imaging device to a pre-generated template of the target anatomy during the endoscopy procedure, and the location of the imaging device can be determined based on a degree of template matching. In another example, the device tracking can be based on one or more anatomical landmarks identified from the images or video streams (or features extracted therefrom) such as obtained during the insertion phase of endoscopy. Once the endoscope location and orientation are determined, the device tracking circuit 323 may register the substantially real-time endoscope location and orientation to a pre-generated template of the anatomy. Information of the endoscope location and orientation may be presented to the user on the user interface 330.

[0078] The procedure summarization circuit 324 may generate a video report of the procedure. The video report may include selected or adapted images or video segments, contextual information about one or more workflow events related to the selected or adapted images or video segments, and procedure metadata. Real-time location and orientation of the endoscope or the imaging device associated therewith, such as produced by the device tracking circuit 323, may also be used for creating the video documentation. The procedure summarization circuit 324 can comprise subcircuits or functional modules, including for example on or more of an image / video selector 325, an image / video editor 326, a context synthesizer 327, and a procedure report compiler 328. The image / video selector 325 can select or adapt a portion, less than an entirety, of the images or video streams based on one for more of the identification of the AOI by the AOI detector 322, the contextual information about one or more workflow events, or the relevant procedure metadata collected by the CPM collector 350. In an example, images or video segments spatially related to the identified AOI (e.g., images or video segments containing at least a portion of the AOI) can be selected. Additionally or alternatively, the image / video selector 325 can select or adapt images or video segments that are temporally related to one or more workflow events during the procedure. During the procedure, images and video recordings and the workflow events can both be timestamped. The image / video selector 325 can recognize one or more workflowDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 events from the CPM collector 350, such as an event log or an operator observation or action log, and use such recognized workflow events to “flag” clinically important video segments. For example, images or video segments that are time-synchronized with the one or more workflow events can be selected.

[0079] In some examples, the contextual information includes an operator observation or action log, and the image / video selector 325 can include a natural language processing (NLP) script generator to produce structured operator observation or action script, and to recognize one or more workflow events from the structured observation or action script. The NLP may involve one or more of computational linguistic models, machine learning or deep learning models, among other Al-based algorithms to recognize crucial observations or actions satisfying a specified condition. The image / video selector 325 can select images or video segments temporally related to (e.g., time-synchronized with) the operator’s verbal cues indicative of crucial observations or actions.

[0080] In some examples, the image / video selector 325 may identify the one or more workflow events using a trained machine-learning (ML) model trained to establish a correspondence between contextual information and characteristics of one or more workflow events. In some examples, the ML model may be further trained to prioritize the recognized workflow events such as based on the procedural context and actions taken by the operator. The trained ML model may be a part of the trained ML modes 342 stored in the storage device 340. Examples of training an ML model and using the same to identify workflow events are discussed below with respect to FIGS. 6A-6B.

[0081] The image / video editor 326 can edit the selected or adapted video segments, including one or more operations of image / video trimming, cropping, stitching, or compression. For example, time-cropping can be applied to each of the selected video segments to trim unnecessary footage, focusing on the core content. The trimmed segments can be stitched together into a continuous stream with smooth transitions between scenes, while maintaining logical flow and visual consistency.

[0082] In an example, the image / video editor 326 can dynamically adjust video playback speed of one or more selected video segments. The dynamic speed adjustment can help fit the selected video segments (along with theDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 associated or time-synchronized contextual information) into a compact video documentation of a specified duration (e.g., approximately 60 seconds) without losing important details. By way of non-limiting example, the video playback speed can vary within a specific range between 0.5x and 4x of a “base” speed (e.g., the video recording speed), depending on an informational density of the video segment and / or its clinical importance. Events with higher informational density or of higher clinical significance (e.g., an identified anatomical landmark or a suspicious anomaly with complex structural details, or associated with operator commentary or actions indicative of higher significance) can trigger a slower playback speed, and vice versa. This allows more time to be allocated to complex or significant portions of the procedure, while less critical portions can be displayed briefly. In some examples, the image / video editor 326 can use an Al-based algorithm, such as a trained ML model, to automatically determine a proper video playback speed for each identified video segment based on the informational density and its clinical importance.

[0083] In some examples, the image / video editor 326 may edit the recorded images / video or adjust video playback speed using a temperature profile that contains temperature data continuously and concurrently measured with the image or video recording during the procedure. As discussed above, the anatomical target of interest (e.g., treatment target) generally has a higher temperature than non-target anatomies. The image / video editor 326 may, based on the temperature profile, perform a lower rate of data reduction (e.g., image trimming or cropping), less data compression, and / or a slower video playback speed for images or video segments temporally corresponding to high temperature measurements above a threshold (which indicates the target anatomy). This allows for a higher quality image / video of the target anatomy with least distortion or information loss, while at the same time saves system sources (e.g., memory spaces and bandwidth).

[0084] The procedure report compiler 328 can generate a procedure report using the selected or adapted images or video segments, at least a portion of the contextual information about the procedure workflow events relevant to the selected or adapted images or video segments, and procedure metadata. In some examples, the context synthesizer 327 may synthesize multiple contextual sources including, for example, an event log (e.g., Al-generated logs detailingDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 characteristics of the recognized AOI or detected workflow events), operator observation or action log (e.g., tool usage, technique application), ambient verbal commands or commentary (or a transcription via NLP), endoscopy report entries (including, for example, critical insights from the user-generated endoscopy report, diagnostic conclusions, and recommendations), among others. The synthesized contextual information may be presented as a text or audio overlay on the selected images or video segments. In an example where the contextual information or a portion thereof takes the form of text, the context synthesizer 327 may perform text-to-speech synthesis, such as using a large language model (LLM) with appropriate grounding and prompt engineering, to generate timestamped narrative to be overlaid upon the selected images or video segments. In some examples, visual cues or markers may be superimposed on the video segments included in the procedure report to signify the AOI (e.g., identified landmarks or anomalous areas), thereby guiding the user’s focus on those areas of interest.

[0085] In an example, the procedure report compiler 328 may feed the selected image or video segments, the synthesized contextual information related to (e.g., time-synchronized with) the selected images or video segments, and the procedure metadata to a trained machine-learning (ML) model to generate the procedure report. The trained ML model may be a part of the trained ML modes 342 stored in the storage device 340. Examples of training an ML model and using the same to generate the procedure report are discussed below with respect to FIGS. 6A-6B.

[0086] The system 300 may include a procedure time tracker 360, which according to an example as illustrated, may be included in the controller circuit 320. The procedure time tracker 360 can track the endoscopy procedure time, such as time of active use of the endoscope inside the patient body. In an example, the tracked procedure time can be based on the triggered start of recording time (t2 in FIG. 4) and the triggered stop of recording time (E in FIG.4). The tracked endoscopy procedure time may be provided to the procedure report compiler 328 and to be included in the resulting procedure video report. In some examples, the controller circuit 320 may establish a correlation between the tracked procedure time and an outcome of the endoscopy procedure, or a correlation between the tracked procedure time with procedure metadata (e.g.,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 doctor, scope type, procedure type, outcomes, and / or equipment features such as light mode or video enhancements). Based on one or more of such established correlations, the controller circuit 320 may predict a procedure outcome (e.g., chance of success). The correlation(s) may also be output for training and educational purposes.

[0087] The procedure report compiler 328 may refine the procedure report before being presented to a user or archived for future use. The refinement can include, for example, “fine-tuning” the images or video segments included therein (e.g., with additional video trimming and stitching), modifying the text and audio overlay (e.g., with further editing or annotation), or data compression to reduce the file size of the report for efficient storage and transmission without significantly impacting video quality. The procedure report may be formatted to comply with a standard, such as a Digital Imaging and Communications in Medicine (DICOM) format, to allow or facilitate interoperability.

[0088] The procedure report may be provided to a user, such as displayed on the user interface 330. Additionally or alternatively, the procedure report may be stored in the storage device 340. The storage device 340 can be local to the endoscope system 300, or alternatively a remote storage device, such as an Electronic Health Records (EHR) system, a Picture Archiving and Communication System (PACS), or other integrated patient record systems. In response to a user command or a trigger event, the procedure report may be transferred to the remote storage device for easy access and review by healthcare professionals. The procedure report may assist in subsequent patient diagnosis, treatment planning, or for physician training or patient education purposes. In some examples, the remote storage device can be a part of a cloud comprising one or more storage and computing devices (e.g., servers) that provides secure access to cloud-based services including, for example, data storage, computing services, and provisioning of customer services, among others. In some examples, at least some of the data processing and computation by the controller circuit 320 as described above may be performed in a cloud. For example, images or video streams or features extracted therefrom may be streamed to the cloud, get processed therein, and the computation results (e.g., the personalized pathology-specific imaging modality) may be relayed back to local endoscope system.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0089] In some examples, the procedure report compiler 328 may generate one or more themed video summaries each corresponding to a particular theme. Each themed video summary comprises themed images or video segments and contextual information related thereto (e.g., time-synchronized therewith). The themed images or video segments can be selected using the image / video selector 325. The contextual information can be a subset of the time-synchronized, synthesized contextual information produced by the context synthesizer 327. Examples of the themed video summaries may include a pre-procedure preparation themed video summary (e.g., bowel preparation theme for a colonoscopy procedure), a procedure phases themed video summary (e.g., an insertion phase and a withdrawal phase of a colonoscopy procedure, withdrawal speed at various colon segments), an anatomical landmark themed documentation (e.g., one or more colon segments or specific structures therein), or an anomaly detection themed documentation (e.g., polyps detected during the colonoscopy procedure). For example, a pre-procedure preparation themed video summary may include selected images or video segments of various colon segments or areas indicative of quality of bowel preparation, as well as operator observations, commentaries, or actions related to patient bowel preparation. In another example, an anomaly themed video summary may include selected images or video segments of identified anomalous structures (such as automatically detected by the AOI detector 322), as well as operator observations, commentaries, and actions related to anomaly recognition. The procedure report compiler 328 may rank multiple themed video summaries in a specific order for prioritized presentation to the user, or for prioritized storage or transmission. In an example, the order may be based on clinical significance such as determined by computer-aided diagnostic or other anomaly detection and characterisation techniques. For example, a video summary of a cancerous colon polyp can be ranked higher than a video summary of a benign colon polyp. In some examples, anomalous structures (e.g., colon polyps) can be characterized or classified based on their appearance under narrow-band imaging (NBI). This classification helps distinguish between different types of polyps (e.g., hyperplastic polyp, adenomatous polyps, deep submucosal invasive cancer), which is crucial for determining the appropriate treatment and management. In some examples, multiple themed video summaries may be indexed andDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 selectable by the themes. A user may select, via the user interface 330, a themed video summary for presentation, or for storage or transmission. In some examples, a multimedia report template can be created for an individual user, which may include one or more user-preferred themed video summaries. For example, users (e.g., endoscopists) of different levels of experience or focusing on different clinical issues may identify their preferred themed video summaries to be created, and a preferred order to be presented.

[0090] The real-time alert and recommendation circuit 329 can generate a notification about the triggered start of recording and the triggered stop of recording of the images or video streams. The notification may be display on a display of the user interface 330, or presented in other audio or visual formats. In some examples, the user interface 330 may receive a user input to modify the triggered start or the triggered stop of recording of the images or video streams.

[0091] In some examples, the real-time alert and recommendation circuit 329 can generate an alert of the identified AOI (e.g., an anatomical landmark or an anomaly) or other events detected during the procedure. The alert may be delivered through optical means on the diagnostic monitor, or via auditory means like a warning alarm. In an example, highlighting, flash alerts, audible or haptic feedback may be provided to the user. Images or video streams of AOI (e.g., anomalies) identified by the AOI detector 322 may be presented to the user on the user interface 330. In some examples, the real-time alert and recommendation circuit 329 can provide real-time recommendation to record images or video recordings related to the identified AOI and to include the same in the procedure report. The user may review the images or video recordings of the identified AOIs, provide adequate contextual information such as commentary on the AOI (e.g., anomaly characterization and diagnosis), and either take the recommendation to make video documentation as presented, or reject the recommendation and ignore the identified anomaly. In some examples, the endoscope system 300 may integrate an iterative feedback mechanism for continuous enhancement in recognizing AOI and workflow events of interest. This can involve collecting feedback from endoscopists on recommendations, such as their explicit feedback (e.g., satisfaction levels with the recommendations, a “like” button etc.). Alternatively, the endoscope system 300 can run in a shadow mode to monitor actions of experienced endoscopists andDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 correct or refine the recommendation based on user interactions. For instance, if a suggestion is ignored or times out, the system 300 can instantly recalibrate its subsequent recommendations. Advanced algorithms like Q-leaming or Markov decision processes might be employed, adapting the system to the specific preferences of each institution. Metrics like recommendation acceptance rate can be tracked to further optimize the efficacy of refining recommendations.

[0092] FIG. 5 illustrates an example process of generating a procedure report of a colonoscopy procedure, such as using the endoscope system 300. The colonoscopy procedure, as illustrated, includes colonoscope insertion and subsequent colonoscope withdrawal phases. Recording of the endoscopic images or video streams, as described above with respect to FIG. 3, can be automatically trigged by temperature sensed by the temperature sensor 316. A first transition from an ambient temperature to a body temperature may indicate an insertion of the endoscope into the patient body, and triggers start of image or video recording. A second transition from the body temperature to the ambient temperature may indicate a withdrawal of the endoscope from the patient body, and triggers stop of image or video recording. During the insertion phase, endoscopic images or video streams can be obtained respectively for each of a plurality of colon segments, including the rectosigmoid segment, sigmoid segment, descending segment, transverse segment, ascending segment, and cecum, such as using the imaging system 312. Various AOIs, including predetermined anatomical landmarks and anomalies, may be identified automatically using the AOI detector 322 during the insertion phase or alternatively the withdrawal phase. In an example, at the end of the insertion phase when the distal tip of the colonoscope is determined to be reaching the cecum (such as determined by the device tracking circuit 323), AOI detection process can be be manually or automatically initiated. As described above with respect to FIG. 3, the AOI detector 322 may analyze the scans (i.e., images or video streams) of the endoscopic feed using template-matching or AI / ML based algorithm, and identify unusual or suspicious regions indicative of pathological abnormalities, such as polyps or cancerous tissue. Images or video segments that are spatially related to the identified AOIs can be selected, along with the associated timestamps.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0093] Contextual information about one or more workflow events, such as an event log or an operator observation or action log, may be collected during the procedure. As illustrated in FIG. 5, the operator observation or action log can include tracked operator actions such as “scope in” 521, “reaching caecum” 522, “polyp detected” 523, “polyp removed” 524, and “scope out” 525. The operator observation or action log can further include operator’s voice commands or commentary on findings during the procedure, such as voice commentaries 531, 532, 533, and 535 as shown in FIG. 5. In some examples, the voice commands or commentary can be processed using a NPL script generator, as described above with respect to FIG. 3. Each of the workflow events, such as the tracked operator actions and / or operator voice commands and commentary, can be timestamped, and synchronized with the images or video segments which are also timestamped during the procedure. In the example illustrated in FIG. 5, image or video segment 512 is temporally related to user action 522 and voice commentary 532, images or video segments 513 and 514 are temporally related to user actions 523 and 524 and voice commentary 533, image or video segment 515 is temporally related to user action 525 and voice commentary 535. Images or video segments that are temporally related to the workflow events can be selected, along with the associated timestamps. In one example, the temporally related images or video segments can be those time-synchronized with the workflow events. In another example, the temporally related images or video segments can be those suggested or implied by the operator’s voice commands or commentary. For example, if the commentary is made at time T, and the commentary refers to a landmark encountered or an event occurred at a prior time (T-5 seconds) or a prior scene, then the images or video segments taken at that time can be selected.

[0094] As illustrated in FIG. 5, the contextual information may further include user commands to freeze frames and / or intra-procedure reporting, such as commands or reports 541 through 547. Similar to operator actions and voice commentary, these user commands and reports can also be timestamped and used for selecting images or video segments to be included in the video summarization 550.

[0095] Various contextual information as described above may be combined with the selected images or video segments to create a videoDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 summarization 550 of the colonoscopy procedure. Various sources of contextual information, including user actions, voice commentary, and user commands to freeze frames, may be synthesized into text or audio overlay on the selected images or video segments, such as an audio voiceover 551 that can be integrated into the video summarization 550 to create a multimedia procedure report 560, which can further be presented to the user or archived in a storage system such as a PACS 570.

[0096] FIGS. 6A-6B illustrate examples of training a machine learning (ML) model, and generate a procedure report comprising video documentation with text or audio overlay using the trained ML model. In an example, the same ML model may also be trained to accomplish multiple subtasks and output various aspects of the procedure report, such as AOI detection, workflow events recognition, image or video segments selection, or dynamically determined video segment playback speed. In some examples, multiple ML models may be trained separately to determine respective different aspects of the procedure report. The multiple, individually trained ML models may have different model structures. Examples of the such individually trained ML models may include one or more of an ML model trained to determine a imaging or video recording mode (e.g., to predict values of one or more image or video quality parameters) based on input including sensed body temperatures, an ML model trained to recognize an AOI (e.g., an anomaly) from the image or video streams, an ML model can be trained to recognize one or more workflow events, an ML mode trained to select or adapt video segments to be included in the procedure report, an ML model trained to determine a video playback speed of a video segment, or an ML model trained to produce a procedure report such as based on the inference results from other trained ML models.

[0097] By way of example, FIG. 6A illustrates an ML model training phase during which an ML model 630 may be trained to generate a procedure report, or an aspect thereof as described above. The model training includes constructing a training dataset that comprises a plurality of endoscopic images or video streams 610 of a particular anatomical target (e.g., one or more colon segments), and training contextual information and procedure metadata (CPM) 620 associated with or relevant to the endoscopic images or video streams 610. The endoscopic images or video streams 610 can be recorded from the sameDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 type of procedures (e.g., colonoscopy) performed on a plurality of patients. The associated contextual information includes information about workflow events (e.g., event log, and operator observation, commentary, and action log) recorded during respective procedures. The associated procedure metadata collected from past procedures performed on the plurality of patients, including information about patients, physicians and relevant clinical staff, procedural parameters such as date, time, facility, equipment, devices, and tools used, procedure type, environment, among other elements involved in carrying out a specific medical procedure, or the clinical trial in which the patients participate. Endoscope location information for each of the plurality of endoscopic images or video streams may also be included in the training dataset.

[0098] The ML model 630 may have a neural network structure comprising an input layer, one or more hidden layers, and an output layer. The plurality of endoscopic images or video streams 610 and the associated training CPM 620 may be fed into the input layer of the ML model 630, which propagates the input data through one or more hidden layers to the output layer that outputs a recommended pathology-specific imaging modality. The ML model 630 can perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The ML model 630 explores the study and construction of algorithms (e.g., ML algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building the ML model 630 from training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0099] The ML model 630 may be trained using supervised learning or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model can implement the same relationships when given inputs to generate the corresponding outputs.Unsupervised learning is the training of an ML algorithm using information that is neither classified nor labelled and allowing the algorithm to act on that information without guidance. Unsupervised learning is useful in exploratory analysis because it can automatically identify structure in data.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0100] Common tasks for supervised learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values. Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM). Examples of DNN include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), or a hybrid neural network comprising two or more neural network models of different types or different model configurations. Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0101] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machine-learning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0102] The training of the ML model 630 may be performed continuously or periodically, or in near real time as additional procedure data are made available. The training process involves algorithmically adjusting one or more ML model parameters (e.g., weights or bias at any particular layer of a neural network model), until the ML model being trained satisfies a specified training convergence criterion. By way of example and not limitation, the ML model 630 may be trained with weighted square loss (for explicit feedback) or with binary cross-entropy loss (for implicit feedback). Other training techniques, such as deep factorization machine, wide and deep learning, deep structuredDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 semantic models, or autoencoder based recommender systems, may be used. The trained ML model 630 can establish a correspondence between the endoscopic images or video streams 610 and an ideal or recommended endoscope maneuvering plan, including a guided endoscope navigation path and / or a recommended imaging mode or setting.

[0103] The trained ML model 630 can establish a correspondence between the training dataset (e.g., the endoscopic images or video streams 610 and the associated training CPM 620) and a procedure report, such as a video documentation with text or audio overlay. In various examples, different ML algorithms may be used to learn different aspects or components of the procedure report. In an example of anomaly detection and characterization, video stream of the target anatomy (e.g., colon segments) can be subjected to real-time analysis using a trained ML model. Leveraging Convolutional Neural Networks, bi-directional LSTM, Recurrent Neural Networks, and Conditional Random Fields, the trained ML model can effectively pinpoint an anomaly and accordingly select video segments spatially related to the detected anomaly. Other machine learning techniques such as decision trees and k-means clustering, may also be employed for thorough analysis. In some examples, the trained ML model can include a spatio-temporal network trained to not only detect anomalies but also to extract essential features from the video stream. These features can then be utilized to assist in video documentation and procedure recommendation.

[0104] FIG. 6B illustrates an inference phase, where live images or video feed 650 during an endoscopy procedure, and testing contextual information and procedure metadata (CPM) 660 associated with the present procedure, can be applied to the trained ML model 630 (obtained from the training phase as illustrated in FIG. 6A) to generate a procedure report 670. The procedure report 670 may be presented to a user (e.g., an endoscopist), or archived for future use (e.g., subsequent diagnosis, treatment planning, or for physician training or patient education). Depending on the architecture of the trained ML model 630, in some examples, other aspects of the procedure report, such as anomaly detection and characterization, workflow events recognition, image or video segment selection, or dynamic video playback speed, may also be determined during the inference phase, and provided to the user or archived as needed.Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1

[0105] FIG. 7 is a flowchart illustrating an example method 700 for temperature-based endoscopic video recording and automated video curation in an endoscopy procedure. The method 700 may be implemented in the endoscope system 300 to generate a procedure report comprising selected or adapted images or video segments and relevant contextual information and procedure metadata. Although the processes of the method 700 are drawn in one flow chart, they are not required to be performed in a particular order. In various examples, some of the processes can be performed in a different order than that illustrated herein.

[0106] At step 710, temperature is continuously monitored during an endoscopy procedure such as using the temperature sensor 316 associated with an endoscope. The monitored temperature includes an ambient temperature (Ta, such as temperature of the procedure room) when the endoscope is outside the patient (e.g., pre-procedure preparation, and post withdrawal of endoscope from the patient body at the conclusion of the procedure), and a body temperature (Tb) when the endoscope is inside the patient.

[0107] At step 720, a recorder device can be “waken up” to start recording the images or video streams captured by an imaging device in response to a first transition from the sensed ambient temperature Tato the sensed body temperature Tb. As described above with respect to FIGS. 3 and 4, the first transition may be detected when an increase (ATa-b) or a rate of increase (ATa-b / At) in the sensed temperature exceeds a first threshold Tthi. The threshold Tthi may be predetermined based on prior knowledge of a difference between the ambient temperature Taand body temperature Tb.

[0108] While the sensed temperature remains above the first threshold Tthi, the endoscope is indicated to be within the patient body, and the recording continues. At step 730, an imaging or video recording mode, including at least one image or video quality parameter, may be adjusted to record the images or video streams while the endoscope is indicated to be within the patient. The mode adjustment may be based at least in part on a variation of the sensed body temperature. One example of the body temperature variation is an intra-body temperature variation between the target anatomy and a non-target anatomy inside the body. Another example of the body temperature variation is related to treatment of the target anatomy, such as thermal response (e.g., temperature rise)Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 of the anatomical target to laser treatment. In an example, when the sensed intrabody temperature variation or the treatment-related temperature variation indicates an increased temperature above respective thresholds, the imaging or video recording mode can be automatically adjusted, such as by increasing the video frame rate and / or resolution, to increase the image or video quality of the target anatomy.

[0109] The image or video recording continues until at step 740 when the recorder device is triggered to stop recording the images or video streams in response to a second transition from the sensed body temperature Tb to the sensed ambient temperature Ta. As described above with respect to FIGS. 3 and 4, the second transition may be detected when a decrease (ATb-a) or a rate of increase (ATb-a / At) in the sensed temperature exceeds a second threshold Tth2. The second threshold T*2 may be predetermined based on prior knowledge of a difference between the ambient temperature Taand body temperature Tb. In an example, ambient temperature Tais generally lower than the body temperature Tb by approximately 10 degrees. The first and the second thresholds Tthi and Tth2 can be set to approximately 5-6 degrees, such that video recording can be triggered to start when the sensed body temperature Tb exceeds 31-32 degrees (and optionally sustains for a period of approximately 2-5 seconds), and triggered to stop when the sensed body temperature Tb drops below 31-32 degrees (and optionally sustains for a period of approximately 2-5 seconds).

[0110] At step 750, a procedure report can be generated using the triggered recording of the images or video streams. As described above with respect to FIGS. 3 and 5, the collected images or video streams may be analyzed to identify an area of interest (AOI) in the target anatomy, such as using the AOI detector 322. One example of the AOI includes one or more pre-determined anatomical landmarks in the target anatomy, such as landmarks of the upper GI, or landmarks of the lower GI duct. The AOI may additionally or alternatively include an anomalous structure in the target anatomy, such as one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the target anatomy. In some examples, the AOI may include a “blind spot” region of the target anatomy, such as hidden places inadvertently missed for lesion screening during colonoscopy. In colonoscopy for example,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 because colon has innumerable folds and two flexures, such anatomical characteristics may cause certain. The AOI (e.g., a pre-determined landmark or an anomaly) may be identified using techniques such as a template matching technique, or Al or ML based techniques, as described above with respect to FIG. 3.

[0111] A portion, less than an entirety, of the obtained images or video streams of the target anatomy may be selected or adapted from the receive images or video streams. In an example, the selection or adaptation can be based on a spatial relationship to the identified AOI, such that video segments containing the identified AOI can be selected. Additionally or alternatively, selection or adaptation can be based on a temporal relationship to (e.g., time-synchronized with) the recognized one or more workflow events. In some examples, the received contextual information includes an operator observation or action log. A natural language processing (NLP) script generator and generate structured operator observation or action script, and one or more workflow events can then be recognized using the structured observation or action script. The selected or adapted video segments can further be edited, such as by apply one or more of image or video trimming, cropping, stitching, or compression, before being used to construct the procedure report.

[0112] In some examples, the procedure report may further include contextual information about one or more workflow events during the procedure, and / or procedure metadata. The contextual information may include an event log containing records of setup information of the endoscope system, operating modes of one or more devices being used in the procedure, or the patient’s response or medical events occurred during the procedure. The contextual information may additionally or alternatively include an operator observation or action log identifying structures or events of clinical significance during the procedure. Examples of the operator observations or actions may include verbal command or commentary during the procedure, manipulation of the endoscope system or one or more devices being used in the procedure, or treatment actions taken during the procedure (e.g., polypectomy, or clipping or clotting of an anatomical structure during the procedure). In some examples, multiple sources of contextual information may be synthesized into a text or audio overlay on the selected images or video segments to produce the procedure report. Examples ofDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 the multiple contextual data sources can include an event log, operator observation or action log, ambient verbal commands or commentary, or endoscopy report entries, among others. In an example where the contextual information or a portion thereof takes the form of text, text-to-speech synthesis, such as using a large language model (LLM) with appropriate grounding and prompt engineering, may be used to generate timestamped narrative overlaid upon the selected images or video segments. In some examples, visual cues or markers may be superimposed on the video segments included in the procedure report. The procedure metadata may include patient information, clinical staff information, procedure information such as date, time, facility, equipment and tools used, and procedure type, etc., or clinical trial information relevant to the present procedure.

[0113] The procedure report may be refined before being released or archived. Examples of the refinement process include “fine-tuning” the video segments, further editing or annotation of the text and audio overlay, data compression of the report, among others. The procedure report may be formatted to comply with a standard to facilitate interoperability.

[0114] In an example, the procedure report can include one or more themed video summaries each comprising themed images or video segments and contextual information related thereto. Examples of the themed video summaries may include a pre-procedure preparation themed video summary, a procedure phases themed video summary, an anatomical landmark themed video summary, or an anomaly detection themed documentation. In an example, multiple themed video summaries may be ranked in a specific order (e.g., an order of clinical significance) for prioritized presentation to the user, or for prioritized storage or transmission. In another example, multiple themed video summaries may be indexed and selectable by the themes. A user may select a themed video summary for presentation, or for storage or transmission.

[0115] In some examples, one or more trained machine-learning (ML) models may be used to identify an AOI, to recognize one or more workflow events from the contextual information, to select or adapt a portion of the images or video segments, or to generate the procedure report. The one or more ML models may each be trained using respective training datasets comprising, for example, images or video streams of a target anatomy and contextualDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 information obtained during endoscopy procedures on a patient population. As described above with respect to FIGS. 6A-6B, during the inference phase, one or more of (i) the selected or adapted portion of the images or video streams or (ii) the related potion of the contextual information may be fed into the one or more trained ML models to perform actions such as identifying the AOI, recognizing one or more workflow events, or generating the procedure report.

[0116] The procedure report may be provided to a user (e.g., an endoscopist) or a process. More specifically, in one example, at step 762, the procedure report may be displayed to the user (e.g., an endoscopist). In another example, at step 764, an alert may be generated to notify or warn the user about the identified AOI, and receiving user’s feedback. The temperature-triggered start and strop of recording may be notified to the user, and in some examples, a user input may be received to modify the start and / or stop of the recording of the image or video data. In some examples, real-time recommendation to record images or video recordings related to the identified AOI and to include the same in the procedure report may also be provided to the user. In yet another example, at step 766, the procedure report may be archived in a storage device, such as an Electronic Health Records (EHR) system or a Picture Archiving and Communication System (PACS), among other integrated patient record systems. The procedure report may be used in subsequent diagnosis, treatment planning, or for physician training or patient education, among other uses.

[0117] FIG. 8 illustrates generally a block diagram of an example machine 800 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Portions of this description may apply to the computing framework of various portions of the endoscope system 300.

[0118] In alternative embodiments, the machine 800 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 800 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 800 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0119] Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership may be flexible over time and underlying hardware variability. Circuit sets include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively connected to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0120] Machine (e.g., computer system) 800 may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memoryDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 804 and a static memory 806, some or all of which may communicate with each other via an interlink (e.g., bus) 808. The machine 800 may further include a display unit 810 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In an example, the display unit 810, input device 812 and UI navigation device 814 may be a touch screen display. The machine 800 may additionally include a storage device (e.g., drive unit) 816, a signal generation device 818 (e.g., a speaker), a network interface device 820, and one or more sensors 821, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensors. The machine 800 may include an output controller 828, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0121] The storage device 816 may include a machine-readable medium 822 on which is stored one or more sets of data structures or instructions 824 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 824 may also reside, completely or at least partially, within the main memory 804, within static memory 806, or within the hardware processor 802 during execution thereof by the machine 800. In an example, one or any combination of the hardware processor 802, the main memory 804, the static memory 806, or the storage device 816 may constitute machine-readable media.

[0122] While the machine-readable medium 822 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 824.

[0123] The term “machine-readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 800 and that cause the machine 800 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Nonlimiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. In an example, a massed machine-readableDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 medium comprises a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0124] The instructions 824 may further be transmitted or received over a communication network 826 using a transmission medium via the network interface device 820 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as WiFi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or phonejacks) or one or more antennas to connect to the communication network 826. In an example, the network interface device 820 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 800, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.Additional Notes

[0125] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. TheDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0126] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0127] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed DescriptionDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 What is claimed is:

1. An endoscope system, comprising:an endoscope, including:an imaging device to produce images or video streams of a target anatomy of a patient during an endoscopy procedure; anda temperature sensor to sense temperature including an ambient temperature when the endoscope is outside the patient and a body temperature when the endoscope is inside the patient; anda controller circuit configured to monitor the sensed temperature, and to trigger a recorder device to start recording the images or video streams based on a first transition from the sensed ambient temperature to the sensed body temperature, and to stop recording the images or video streams based on a second transition from the sensed body temperature to the sensed ambient temperature.

2. The endoscope system of claim 1, wherein the temperature sensor is disposed at a distal tip of the endoscope.

3. The endoscope system of any of claims 1-2, wherein the controller circuit is configured to trigger the recorder device to start recording the images or video streams when an increase or a rate of increase in the sensed temperature exceeds a first threshold for a first specific period, and to stop recording the images or video streams when a decrease or a rate of decrease in the sensed temperature exceeds a second threshold for a second specific period.

4. The endoscope system of any of claims 1-3, wherein the controller circuit is further configured to:track procedure time based on respective times of the triggered start of recording and the triggered stop of recording; andpredict an outcome of the endoscopy procedure based at least in part on the tracked procedure time.

5. The endoscope system of any of claims 1-4, wherein the controller circuit is configured to, based at least in part on a variation of the sensed bodyDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 temperature, adjust an imaging or video recording mode, including at least one image or video quality parameter, to record the images or video streams.

6. The endoscope system of claim 5, wherein the controller circuit is configured to:establish, or receive, a correspondence between body temperatures and corresponding values of the at least one image or video quality parameter; and adjust the imaging or video recording mode using the sensed body temperature and the established or received correspondence.

7. The endoscope system of any of claims 5-6, wherein the variation of the sensed body temperature includes an intra-body temperature variation between the target anatomy and a non-target anatomy inside the body,wherein the controller circuit is configured to adjust an imaging or video recording mode to increase the image or video quality when the sensed intra-body temperature variation indicates an increased temperature above a threshold.

8. The endoscope system of any of claims 5-7, wherein the variation of the sensed body temperature includes a temperature variation in relation to laser treatment of the target anatomy,wherein the controller circuit is configured to adjust an imaging or video recording mode to increase the image or video quality when the temperature variation in relation to the laser treatment of the target anatomy indicates an increased temperature above a threshold.

9. The endoscope system of any of claims 1-8, wherein the controller circuit is further configured to:select or adapt a portion, less than an entirety, of the recorded images or video streams; andgenerate a procedure video report using the selected or adapted portion of the recorded images or video streams.

10. The endoscope system of claim 9, wherein the controller circuit is further configured to analyze the recorded images or video streams to identify an area ofDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 interest (AOI) in the target anatomy during the procedure, and to select or adapt the portion spatially related to the identified AOI.

11. The endoscope system of claim 10, wherein the AOI includes a predetermined anatomical landmark or an anomalous structure in the target anatomy.

12. The endoscope system of claim 11, wherein the endoscope is a colonoscope including the imaging device to obtain images or video streams of colon segments during a colonoscopy procedure,wherein the AOI includes a colon landmark or anomaly in one of the colon segments.

13. The endoscope system of claim 9, wherein the controller circuit is further configured to generate the procedure video report further using contextual information about one or more workflow events during the procedure.

14. The endoscope system of claim 9, wherein the controller circuit is further configured to generate the procedure video report further using procedure metadata including information about one or more of the patient, a physician or clinical staff, procedural parameters, or a relevant clinical trial.

15. The endoscope system of any of claims 1-14, comprising a user interface device configured to:provide a notification to a user about the triggered start of recording and the triggered stop of recording of the images or video streams; andreceive a user input to modify the triggered start or the triggered stop of recording of the images or video streams.

16. The endoscope system of any of claims 1-15, wherein the controller circuit is configured to:generate or receive at least one trained machine-learning (ML) model each trained using a training dataset comprising ambient temperatureDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 measurements and body temperature measurements obtained during endoscopy procedures on a patient population; andapply the sensed ambient temperature and the sensed body temperature of the patient during the procedure to the trained at least one ML model to determine an image or video quality parameter value.

17. A method for endoscopic video recording and curation during an endoscopy procedure in a patient using an endoscope that includes an imaging device and a temperature sensor, the method comprising:sensing temperature, including an ambient temperature when the endoscope is outside the patient, and a body temperature when the endoscope is inside the patient using the temperature sensor of the endoscope;triggering a recorder device, via a controller circuit, to start recording images or video streams of a target anatomy of the patient during an endoscopy procedure based on a first transition from the sensed ambient temperature to the sensed body temperature;triggering the recorder device, via the controller circuit, to stop recording the images or video streams based on a second transition from the sensed body temperature to the sensed ambient temperature; andgenerating a procedure video report using the triggered recording of the images or video streams.

18. The method of claim 17,wherein triggering the recorder device to start recording the images or video streams occurs when an increase or a rate of increase in the sensed temperature exceeds a first threshold for a first specific period,wherein triggering the recorder device to stop recording the images or video streams occurs when a decrease or a rate of decrease in the sensed temperature exceeds a second threshold for a second specific period.

19. The method of any of claims 17-18, further comprising:tracking procedure time based on respective times of the triggered start of recording and the triggered stop of recording; andDocket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 predicting an outcome of the endoscopy procedure based at least in part on the tracked procedure time.

20. The method of any of claims 17-19, further comprising, based at least in part on a variation of the sensed body temperature, adjusting an imaging or video recording mode, including at least one image or video quality parameter, to record the images or video streams.

21. The method of claim 20, wherein the variation of the sensed body temperature includes an intra-body temperature variation between the target anatomy and a non-target anatomy inside the body,wherein adjusting the imaging or video recording mode occurs when the sensed intra-body temperature variation indicates an increased temperature above a threshold to increase the image or video quality.

22. The method of claim 20, wherein the variation of the sensed body temperature includes a temperature variation in relation to laser treatment of the target anatomy,wherein adjusting the imaging or video recording mode occurs when the temperature variation in relation to the laser treatment of the target anatomy indicates an increased temperature above a threshold to increase the image or video quality.

23. The method of any of claims 17-22, further comprising, via the controller circuit:selecting or adapting a portion, less than an entirety, of the recorded images or video streams; andgenerating the procedure video report using the selected or adapted portion of the recorded images or video streams.

24. The method of claim 23, further comprising analyzing the recorded images or video streams and identifying therefrom an area of interest (AOI) in the target anatomy during the procedure, and selecting or adapting the portion,Docket No.: 5409.988WO1Client Ref No.: GAP25056-URMS-WO1 less than the entirety, of the recorded images or video streams that is spatially related to the identified AOI.

25. The method of any of claims 23-24, wherein generating the procedure video report further includes using at least one of:contextual information about one or more workflow events during the procedure; orprocedure metadata including information about one or more of the patient, a physician or clinical staff, procedural parameters, or a relevant clinical trial.

26. The method of any of claims 17-25, further comprising:providing a notification on a user interface about the triggered start of recording and the triggered stop of recording of the images or video streams; and receiving a user input from the user interface to modify the triggered start or the triggered stop of recording of the images or video streams.

27. The method of any of claims 17-26, further comprising:generating or receiving at least one trained machine-learning (ML) model each trained using a training dataset comprising ambient temperature measurements and body temperature measurements obtained during endoscopy procedures on a patient population; andapplying the sensed ambient temperature and the sensed body temperature of the patient during the procedure to the trained at least one ML model to determine an image or video quality parameter value.