Virtual gastrointestinal tract of a person for examination

Abstract

A system includes at least one processor and at least one memory having instructions stored thereon. The instructions, when executed by the processor(s), cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, where the virtual GIT of the person is created based on images captured by a capsule endoscope; displaying a view within the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the portion of the virtual GIT, where the user commands include commands used during a live examination; and navigating the view within the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

Description

PATENT APPLICATIONDocket: A0013107W001VIRTUAL GASTROINTESTINAL TRACT OF A PERSON FOR EXAMINATIONCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Patent Application No.63 / 735,436, filed December 18, 2024, which is incorporated herein by reference in its entirety.FIELD

[0002] This disclosure relates to a gastrointestinal tract of a person, and more particularly, to a virtual gastrointestinal tract (GIT) of a person that can be examination by a healthcare provider.BACKGROUND

[0003] Capsule endoscopy (CE) allows examining the entire GIT endoscopically. There are capsule endoscopy systems and methods that are aimed at examining a specific portion of the GIT, such as the small bowel (SB) or the colon. CE is a non-invasive procedure which does not require the patient to be admitted to a hospital, and the patient can continue most daily activities while the capsule is in his body.

[0004] On a typical CE procedure, the patient is referred to a procedure by a physician. The patient then arrives at a medical facility (e.g., a clinic or a hospital), to perform the procedure. The capsule, which is about the size of a multi-vitamin, is swallowed by the patient under the supervision of a health professional (e.g., a nurse or a physician) at the medical facility and the patient is provided with a wearable device, e.g., a sensor belt and a recorder placed in a pouch and strap to be placed around the patient’s shoulder. The wearable device typically includes a storage device. The patient may be given guidance and / or instructions and then released to his daily activities.

[0005] The capsule captures images as it travels naturally through the GIT. Images and additional data (e.g., metadata) are then transmitted to the recorder that is worn by the patient. The capsule is typically disposable and passes naturally with a bowel movement. The procedure data (e.g., the captured images or a portion of them and additional metadata) is stored on the storage device of the wearable device.PATENT APPLICATIONDocket: A0013107W001

[0006] The wearable device is typically returned by the patient to the medical facility with the procedure data stored thereon. The procedure data is then downloaded to a computing device typically located at the medical facility, which has an engine software stored thereon. The received procedure data is then processed by the engine to a compiled study (or “study”). Typically, a study includes thousands of images (around 6,000 to 9,000). Typically, the number of images to be processed is of the order of tens of thousands and about 90,000 to 100,000 on average.

[0007] A reader (which may be the procedure supervising physician, a dedicated physician or the referring physician) may access the study via a reader application. The reader then reviews the study, evaluates the procedure and provides his input via the reader application. Since the reader needs to review thousands of images, the reading time of a study may usually take between half an hour to an hour on average and the reading task may be tiresome. A report is then generated by the reader application based on the compiled study and the reader’s input. On average, it would take an hour to generate a report. The report may include, for example, images of interest, e.g., images which are identified as including pathologies, selected by the reader; evaluation or diagnosis of the patient’s medical condition based on the procedure’s data (i.e., the study) and / or recommendations for follow up and / or treatment provided by the reader. The report may be then forwarded to the referring physician. The referring physician may decide on a required follow up or treatment based on the report.

[0008] There is continued interest in developing and improving capabilities of capsule endoscopy procedures, colonoscopy procedures, and other GIT evaluation and treatment procedures.SUMMARY

[0009] In accordance with aspects of the disclosure, a system includes at least one processor, and at least one memory having instructions stored thereon. The instructions, when executed by the at least one processor, cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, where the virtual GIT of the person is created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, where the user commands include at least commandsPATENT APPLICATIONDocket: A0013107W001 used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0010] In accordance with aspects of the disclosure, a method includes: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, where the virtual GIT of the person is created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, where the user commands include at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0011] In accordance with aspects of the disclosure, a non-transitory processor-readable medium has instructions stored thereon. The instructions, when executed by at least one processor of a system, cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, where the virtual GIT of the person is created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, where the user commands include at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0012] Aspects of the present disclosure are further described in the Examples shown at the end of the detailed description. The entirety of such Examples are hereby incorporated by reference into this Summary.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0014] A detailed description of embodiments of the disclosure will be made with reference to the accompanying drawings, wherein like numerals designate corresponding parts in the figures:

[0015] FIG. 1 is a diagram of a gastrointestinal tract (GIT);PATENT APPLICATIONDocket: A0013107W001

[0016] FIG. 2 is a block diagram of an exemplary system for analyzing medical images captured in vivo via a Capsule Endoscopy (CE) procedure, in accordance with aspects of the disclosure;

[0017] FIG. 3 is a block diagram of an exemplary computing system which may be used with the systems of the disclosure;

[0018] FIG. 4 is a diagram of example components for interfacing with a virtual GIT of a person, in accordance with aspects of the disclosure;

[0019] FIG. 5 is a diagram of separate images which may be processed by image stitching, in accordance with aspects of the disclosure;

[0020] FIG. 6 is a diagram of an example of a result of processing the images of FIG. 5 using image stitching, in accordance with aspects of the disclosure;

[0021] FIG. 7 is a diagram of examples of data which may be used to create a virtual GIT, in accordance with aspects of the disclosure;

[0022] FIG. 8 is a diagram of another example of image stitching, in accordance with aspects of the present disclosure;

[0023] FIG. 9 is a flow diagram of an example operation for interfacing with a virtual GIT, in accordance with aspects of the disclosure.DETAILED DESCRIPTION

[0024] The present disclosure relate to providing a virtual gastrointestinal tract (GIT) of a person that allows a user (e.g., a physician) to build his or her understanding of a case by reviewing the virtual GIT of the person. As used herein, the term “virtual GIT” means and includes a virtual version of an entire GIT or a virtual version of a portion of a GIT, such as a small bowel or a colon. According to aspects of the present disclosure, the virtual GIT of a patient is provided rather than a “study.” As used herein, a “study” is a display of images (e.g., captured by the CE imaging device 212) that were selected, e.g., automatically, as images that may be of interest. Generally, a relatively small number of images from the captured images is displayed for the user’s review per case in a study. By “relatively small number” it is meant on the order of hundreds or on the order of thousands at most or at least on average. However, reviewing a study may not be a healthcare provider’s typical method of examining a person’s GIT. Rather, a healthcare provider (e.g., physician, clinician, technology specialist, etc.) may typically review a person’s GIT by livePATENT APPLICATIONDocket: A0013107W001 examination during a colonoscopy or endoscopy procedure, which may take about twenty minutes. Therefore, a healthcare provider may not be efficient or proficient in reviewing a study, which may take one hour or more. Aspects of the present disclosure allow a healthcare provider to examine a virtual GIT of a person in the same manner or essentially the same manner that the healthcare provider would perform a live examination. In the description below, any reference to a “virtual GIT” shall be understood to mean a virtual version of an entire GIT or to mean a virtual version of a portion of a GIT. An aspect of the present disclosure is that the virtual GIT may provide full visual tissue coverage along a person’s GIT or along the GIT portion of interest. In aspects of the present disclosure, the minimum amount of images that provides such a full visual tissue coverage may be used in generating the virtual GIT. Such and other aspects will be described in more detail later herein.

[0025] In the following detailed description, specific details are set forth in order to provide a thorough understanding of the disclosure. However, it will be understood by those skilled in the art that the disclosure may be practiced without these specific details. In other instances, well- known methods, procedures, and components have not been described in detail so as not to obscure the present disclosure. Some features or elements described with respect to one system may be combined with features or elements described with respect to other systems. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0026] Although the disclosure is not limited in this regard, discussions utilizing terms such as, for example, “processing,” “computing,” “calculating,” “determining,” “establishing,” “analyzing,” “checking,” or the like, may refer to operation(s) and / or process(es) of a computer, a computing platform, a computing system, or other electronic computing device, that manipulates and / or transforms data represented as physical (e.g., electronic) quantities within the computer’s registers and / or memories into other data similarly represented as physical quantities within the computer’s registers and / or memories or other information non-transitory storage medium that may store instructions to perform operations and / or processes. Although the disclosure is not limited in this regard, the terms “plurality” and “a plurality” as used herein may include, for example, “multiple” or “two or more”. The terms “plurality” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like. The term “set” when used herein may include one or more items. Unless explicitly stated, the methods described herein are not constrained to a particular order or sequence.PATENT APPLICATIONDocket: A0013107W001Additionally, some of the described methods or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.

[0027] A type of CE procedure may be determined based on, inter alia, the portion of the GIT that is of interest and is to be imaged (e.g., the colon or the small bowel (“SB”)), or based on the specific use (e.g., for checking the status of a GI disease, such as Crohn’s disease, or for colon cancer screening).

[0028] The terms screen(s), view(s) and display(s) may be used herein interchangeably and may be understood according to the specific context.

[0029] The term “adjacent” as referred to herein with respect to images (e.g., images that are adjacent to other image(s)), may relate to spatial and / or temporal characteristics unless specifically indicated otherwise. For example, images that are adjacent to other image(s) may be images of GIT portions that are estimated to be located near GIT portions captured in the other image(s) along the GIT and / or images that were captured near the capture time of another image, within a certain threshold, e.g., within one or two centimeters, or within one, five, or ten seconds.

[0030] The terms “GIT” and “a portion of the GIT” may each refer to or include the other, according to their context. Thus, the term “a portion of the GIT” may also refer to the entire GIT and the term “GIT” may also refer only to a portion of the GIT.

[0031] The terms “image” and “frame” may each refer to or include the other and may be used interchangeably in the present disclosure to refer to a single capture by an imaging device. For convenience, the term “image” may be used more frequently in the present disclosure, but it will be understood that references to an image shall apply to a frame as well.

[0032] Referring to FIG. 1 , an illustration of the GIT 100 is shown. The GIT 100 is an organ system within humans and other animals. The GIT 100 generally includes a mouth 102 for taking in sustenance, salivary glands 104 for producing saliva, an esophagus 106 through which food passes aided by contractions, a stomach 108 to secret enzymes and stomach acid to aid in digesting food, a liver 110, a gall bladder 112, a pancreas 114, a small intestine / small bowel 116 (“SB”) for the absorption of nutrients, and a colon 40 (e.g., large intestine) for storing water and waste material as feces prior to defecation. The colon 40 generally includes an appendix 42, a rectum 48, and an anus 43. Food taken in through the mouth is digested by the GIT to take in nutrients and the remaining waste is expelled as feces through the anus 43.PATENT APPLICATIONDocket: A0013107W001

[0033] Studies of different portions of the GIT 100 (e.g., colon 40, esophagus 106, and / or stomach 108) may be presented via a suitable user interface. As used herein, the term “study” refers to and includes at least a set of images selected from the images captured by a CE imaging device (e.g., 212, FIG. 2) during a single CE procedure performed with respect to a specific patient and at a specific time, and can optionally include information other than images as well. The type of procedure performed may determine which portion of the GIT 100 is the portion of interest. Examples of types of procedures performed include, without limitation, a small bowel procedure, a colon procedure, a small bowel and colon procedure, a procedure aimed to specifically exhibit or check the small bowel, a procedure aimed to specifically exhibit or check the colon, a procedure aimed to specifically exhibit or check the colon and the small bowel, or a procedure to exhibit or check the entire GIT: esophagus, stomach, SB and colon.

[0034] FIG. 2 shows a block diagram of a system for analyzing medical images captured in vivo via a CE procedure. The system generally includes a capsule system 210 configured to capture images of the GIT and a computing system 300 (e.g., local system and / or cloud system) configured to process the captured images.

[0035] The capsule system 210 may include a swallowable CE imaging device 212 (e.g., a capsule) configured to capture images of the GIT as the CE imaging device 212 travels through the GIT. The images may be stored on the CE imaging device 212 and / or transmitted to a receiving device 214 typically including an antenna. In some capsule systems 210, the receiving device 214 may be located on the patient who swallowed the CE imaging device 212 and may, for example, take the form of a belt worn by the patient or a patch secured to the patient.

[0036] The capsule system 210 may be communicatively coupled with the computing system 300 and can communicate captured images to the computing system 300. The computing system 300 may process the received images using image processing technologies, machine learning technologies, and / or signal processing technologies, among other technologies. The computing system 300 can include local computing devices that are local to the patient and / or the patient’s treatment facility, a cloud computing platform that is provided by cloud services, or a combination of local computing devices and a cloud computing platform.

[0037] In the case where the computing system 300 includes a cloud computing platform, the images captured by the capsule system 210 may be transmitted online to the cloud computing platform. In various embodiments, the images can be transmitted via the receiving device 214PATENT APPLICATIONDocket: A0013107W001 worn or carried by the patient. In various embodiments, the images can be transmitted via the patient’s smartphone or via any other device connected to the Internet and which may be coupled with the CE imaging device 212 or the receiving device 214.

[0038] FIG. 3 shows a high-level block diagram of an exemplary computing system 300 that may be used with image analyzing systems of the present disclosure. Computing system 300 may include a processor or controller 305 that may be or include, for example, one or more central processing unit processor(s) (CPU), one or more Graphics Processing Unit(s) (GPU or GPGPU), a chip or any suitable computing or computational device, an operating system 215, a memory 320, a storage 330, input devices 335 and output devices 340. Modules or equipment for collecting or receiving (e.g., a receiver worn on a patient) or displaying or selecting for display (e.g., a workstation) medical images collected by the CE imaging device 212 (FIG. 2) may be or include, or may be executed by, the computing system 300 shown in FIG. 3. A communication component 322 of the computing system 300 may allow communications with remote or external devices, e.g., via the Internet or another network, via radio, or via a suitable network protocol such as File Transfer Protocol (FTP), etc.

[0039] The computing system 300 includes an operating system 315 that may be or may include any code segment designed and / or configured to perform tasks involving coordination, scheduling, arbitration, supervising, controlling or otherwise managing operation of computing system 300, for example, scheduling execution of programs. Memory 320 may be or may include, for example, a Random Access Memory (RAM), a read-only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units. Memory 320 may be or may include a plurality of possibly different memory units. Memory 320 may store for example, instructions to carry out a method (e.g., executable code 325), and / or data such as user responses, interruptions, etc.

[0040] Executable code 325 may be any executable code, e.g., an application, a program, a process, task or script. Executable code 325 may be executed by controller 305 possibly under control of operating system 315. For example, execution of executable code 325 may cause the display or selection for display of medical images as described herein. In some systems, more than one computing system 300 or components of computing system 300 may be used for multiplePATENT APPLICATIONDocket: A0013107W001 functions described herein. For the various modules and functions described herein, one or more computing systems 300 or components of computing system 300 may be used. Devices that include components similar or different to those included in the computing system 300 may be used, and may be connected to a network and used as a system. One or more processor(s) 305 may be configured to carry out methods of the present disclosure by for example executing software or code. Storage 330 may be or may include, for example, a hard disk drive, a floppy disk drive, a Compact Disk (CD) drive, a CD-Recordable (CD-R) drive, a universal serial bus (USB) device or other suitable removable and / or fixed storage unit. Data such as instructions, code, medical images, image streams, etc. may be stored in storage 330 and may be loaded from storage 330 into memory 320 where it may be processed by controller 305. In some embodiments, some of the components shown in FIG. 3 may be omitted.

[0041] Input devices 335 may include for example a mouse, a keyboard, a touch screen or pad or any suitable input device. It will be recognized that any suitable number of input devices may be operatively coupled to computing system 300. Output devices 340 may include one or more monitors, screens, displays, speakers and / or any other suitable output devices. It will be recognized that any suitable number of output devices may be operatively coupled to computing system 300 as shown by block 340. Any applicable input / output (I / O) devices may be operatively coupled to computing system 300, for example, a wired or wireless network interface card (NIC), a modem, printer or facsimile machine, a universal serial bus (USB) device or external hard drive may be included in input devices 335 and / or output devices 340.

[0042] Multiple computer systems 300 including some or all of the components shown in FIG. 3 may be used with the described systems and methods. For example, a CE imaging device 212, a receiver, a cloud-based system, and / or a workstation or portable computing device for displaying images may include some or all of the components of the computer system of FIG. 3. A cloud platform (e.g., a remote server) including components such as computing system 300 of FIG. 3 may receive procedure data such as images and metadata, processes and generate a study, and may also display the generated study for the doctor’s review (e.g., on a web browser executed on a workstation or portable computer). An “on-premise” option, may use a workstation or local server of a medical facility to store, process and display images.

[0043] Other aspects of the computing system 300 and the capsule system (210, FIG. 2) are described in co-pending International Application No. PCT / US2020 / 033341, entitled “SystemsPATENT APPLICATIONDocket: A0013107W001 and Methods For Capsule Endoscopy Procedure,” which is hereby incorporated by reference in its entirety. Generally, the disclosed technology may be utilized by the systems and methods of the co-pending applications and / or by any other capsule endoscopy systems or methods, such as PillCam™ capsule endoscopy systems (e.g., SB 3, colon 2, or Crohn’s PillCam™ systems).

[0044] As mentioned above, according to some aspects of the present disclosure, a virtual gastrointestinal tract (GIT) of a person is provided that allows a user (e.g., a healthcare provider) to build his or her understanding of the health of the person by examining the virtual GIT of the person. Aspects of the present disclosure allow a healthcare provider to examine a virtual GIT of a person in essentially the same manner that the healthcare provider would perform a live examination, which will be described below in connection with FIG. 4. In contrast to a study, in which a healthcare provider may review hundreds or thousands of images, a virtual GIT may include substantially all of, or a significant portion of, the images captured by a capsule endoscope, e.g., tens of thousands or hundreds or thousands of images, which allows a healthcare provider to consider more information than would be considered in a study. As mentioned above, an aspect of the present disclosure is that the virtual GIT may provide full visual tissue coverage along a person’s GIT or along the GIT portion of interest. In aspects of the present disclosure, the minimum amount of images that provides such a full visual tissue coverage may be used in generating the virtual GIT. Examples of building a virtual GIT based on images captured by a capsule endoscope will be described below in connection with FIGS. 5-8.

[0045] FIG. 4 is a diagram of an example user interface for a user (e.g., healthcare provider) to examine a virtual GIT of a person. The diagram includes a user computer 410, an optional virtual reality device(s) 420, and a computing system 430 that provides the virtual GIT 432 of the person. The user computer 410 and / or the virtual reality device(s) 420 may be used to communicate with the computing system 430 to access the virtual GIT 432. Communication between the user computer 410, the virtual reality device(s) 420, and the computing system 430 are shown by dashed communications. In various embodiments, the communications may be local communications for implementations where the components 410-430 are all provided at the same location (e.g., same building or facility). In various embodiments, the communications may involve communications across a network or across the Internet where the computing system 430 may be remote from the user computer 410 and / or the virtual reality device(s) 420.PATENT APPLICATIONDocket: A0013107W001

[0046] In various embodiments, a user (e.g., healthcare provider) may access the virtual GIT 432 using just the user computer 410. For example, the user computer 410 may access the virtual GIT 432 (e.g., locally and / or remotely over a network) and display a view of the virtual GIT 432 on a display screen of the user computer 410. A user may use input devices of the user computer 410, such as keyboard, mouse, touchscreen, and / or other input devices, to control navigation of a view within the virtual GIT 432. The user computer 410 may display a user interface screen that includes such controls for navigating a view within the virtual GIT 432. The user interface screen may be provided by a local application executing on the user computer 410 or may be an interface displayed in a web browser. In various embodiments, the user interface controls may include controls that simulate a physician’s controls in a live examination. In various embodiments, the user interface controls may include controls for advancing, retracting, turning, rotating, and / or panning a view within the virtual GIT 432. Such and other controls are contemplated to be within the scope of the present disclosure.

[0047] In various embodiments, a user may access the virtual GIT 432 using just the virtual reality device(s) 420 (e.g., locally and / or remotely over a network). The virtual reality device(s) 420 may include a goggle or other visual device (e.g., glasses) to display a view of the virtual GIT 432 and may include a handset or other gesture input device to control navigation of a view within the virtual GIT 432. In various embodiments, the virtual reality handset may detect hand motions similar to a healthcare provider’s hand motions during a live examination, and the view within the virtual GIT 432 may be controlled based on such detected motions. Other user motions may be implemented and detected for controlling navigation of a view within the virtual GIT 432. Such and other controls are contemplated to be within the scope of the present disclosure.

[0048] In FIG. 4, although the user computer 410 is illustrated as a desktop workstation, the user computer 410 is not limited thereto. In various embodiments, the user computer 410 may be a laptop, a tablet, a smartphone, or other device. The virtual reality device(s) 420 is illustrated as including a headset and hand-held devices but is not limited thereto. In various embodiments, the virtual reality device(s) may include other devices such as, without limitation, virtual reality glasses, gesture-sensing gloves, camera and motion / gesture detection instructions, other virtual reality visualization devices, and / or other motion / gesture sensors, and / or motion or gesture detection instructions. In various embodiments, portions or all of the user computer 410 andPATENT APPLICATIONDocket: A0013107W001 portions or all of the virtual reality device(s) 420 may cooperate to provide access to the virtual GIT 432 and provide control for navigating a view within the virtual GIT 432.

[0049] As mentioned above, in various embodiments, a healthcare provider may conduct the examination of the virtual GIT 432 in substantially the same manner as performing a live examination, which typically takes on the order of fifteen to thirty minutes. For example, a healthcare provider may insert a virtual scope (not shown) into a first end 434 of the virtual GIT 432, navigate the virtual scope through the virtual GIT 432 to another end or another portion 436 of the virtual GIT 432, and withdraw the virtual scope through the virtual GIT 432 and back out the first end 434. In this manner, for example, a healthcare provider may make observations of a virtual colon in a manner similar to colonoscopy - from anus to cecum and back - and which eliminates the need for a healthcare provider to review a huge amount of data or discrete images. It is contemplated that a healthcare provider may take approximately the same amount of time (e.g., fifteen to thirty minutes) to perform an examination of the virtual GIT 432 when using hand motions similar to those performed for a live examination, e.g., by using a virtual reality handset. In contrast to a study, which includes a small number of static images that are preselected, e.g., automatically by algorithm(s), the visual flow of a virtual scope through the virtual GIT 432 is dynamic, similar to a live examination of a GIT, and a healthcare provide can navigate a view within the virtual GIT to determine health condition and to seek out any potential concerns.

[0050] The components and operations described above in connection with FIG. 4 are merely examples. Variations are contemplated to be within the scope of the present disclosure.

[0051] FIGS. 5-7 will now be described in connection with generating a virtual GIT based on images and information obtained by a capsule endoscope.

[0052] FIG. 5 and FIG. 6 are diagrams of an example of an image stitching technique for combining images whose contents overlap. In FIG. 5, four images 510, 520, 530, and 540 of portions of a GIT are shown. Image 510 partially overlaps with image 520, image 520 partially overlaps with image 530, and image 530 partially overlaps with image 540. Imaging stitching is a technique that combines images which have overlapping contents. An example of an implementation of image stitching is provided by Mann, Steve, “Compositing Multiple Pictures of the Same Scene.” Proceedings of the 46th Annual Imaging Science & Technology Conference, (May 9-14, 1993), which is hereby incorporated by reference herein in its entirety. Persons skilled in the art will understand how to implement various image stitching techniques. Using imagePATENT APPLICATIONDocket: A0013107W001 stitching techniques, the four images 510-540 can be combined to provide the result shown in FIG. 6, which may form a portion of the virtual GIT 432 of FIG. 4.

[0053] Referring to FIG. 7, in aspects of the present disclosure, a computing system 750 may construct a virtual GIT 752 using imaging stitching techniques on images captured by a capsule endoscope, which may acquire tens or hundreds of thousands of images for a procedure. By using imaging stitching techniques, proper overlap of images is determined without need for data on the orientation, movement, or location of the capsule endoscope when the images were captured. Even in situations when a capsule endoscope moves backwards and captures portions of a person’s GIT that were already imaged, image stitching techniques can correctly operate in such situations without need for data on a capsule endoscope’s orientation, movement, or location. Accordingly, in various embodiments, the virtual GIT 752 can be constructed using just acquired images 710 and image stitching techniques. An example of stitching images in an axial direction of a GIT is shown in FIG. 8. In the example of FIG. 8, images 802-808 capture a certain depth of a GIT in the axial direction 820 of the GIT. Thus, the images 802-808 can be stitched in the axial direction 820 of the GIT to form a virtual GIT section 852. The virtual GIT 752 of FIG. 7 can include portions stitched together using the example shown in FIG. 5 and can include portions stitched together using the example shown in FIG. 8, among other possible image stitches. All such image stitches are contemplated to be within the scope of the present disclosure.

[0054] With continuing reference to FIG. 7, in embodiments where data on a capsule endoscope’s orientation 720, movement 730, and / or location 740 is available, such data 720-740 may be used to enhance a virtual GIT and provide a more accurate virtual GIT. For example, orientation data 720 may be used to determine where the capsule endoscope was pointing when it captured an image. In various embodiments, orientation data 720 may indicate roll, yaw, and / or pitch, which may be implemented via a gyroscopic device. Movement data 730 may be used to determine, e.g., acceleration of the capsule endoscope, which can indicate GIT distance between captured images. In various embodiments, movement / acceleration data 730 may be provided by a micromachined micro-electromechanical systems (MEMS) accelerometer. In various embodiments, location data 740 may indicate a “localized location” of where the capsule endoscope is located. Such a localized location may be provided, for example, using the antenna array technology described in U.S. Patent No. 6,904,308, by Given Imaging Ltd., which is hereby incorporated by reference herein in its entirety. Location data 740 may be used to determine thePATENT APPLICATIONDocket: A0013107W001 exact location where an image was captured. The orientation data 720, movement data 730, and location data 740 may be used by the computing device 750 to improve the accuracy of the virtual GIT 752. Other embodiments of orientation, movement, and / or location data, and other embodiments of sensor and devices for providing such data, are contemplated to be within the scope of the present disclosure.

[0055] In accordance with aspects of the present disclosure, whether a virtual GIT is constructed using captured images and image stitching alone, or constructed using also orientation, movement, and / or location data, a virtual GIT may be provided that includes a minimum specified frames per unit distance, e.g. a minimum frames per centimeter (cm). In various embodiments, the virtual GIT may have a minimum of 10 frames per centimeter, or another number of minimum frames per cm. In various embodiments, a virtual GIT may be provided that includes a maximum specified frames per unit distance, e.g. a maximum frames per centimeter (cm). In various embodiments, the virtual GIT may have a maximum of 1,000 frames per centimeter, or another number of maximum frames per cm. A typical length of an adult colon is about 150cm. Thus, a virtual GIT needs at least 1,500 frames to provide a minimum of 10 frames per cm, and needs at least 150,000 frames to provide a minimum of 1,000 frames per cm. Requiring a minimum frames per cm provides a degree of quality guarantee for the fidelity of the virtual GIT, while requiring a maximum frames per cm keeps the data / storage size of the virtual GIT to a reasonable size. In various embodiments, where storage size is not an issue, providing even higher frames per cm will provide greater fidelity for the virtual GIT, without increasing a healthcare provider’s examination time (e.g., about fifteen to thirty minutes).

[0056] In accordance with aspects of the present disclosure, when movement data and distance data are not available, portions of the virtual GIT may be assigned a distance by uniformly distributing the frames of the virtual GIT across the length of a typical GIT or across a segment of a typical GIT. For example, as mentioned above, a typical length of an adult colon is 150 cm. After building a virtual GIT, the frames of the virtual GIT can be uniformly spread out across the GIT length or across the GIT segment length of interest (e.g., across 150 cm for an adult colon), and locations may be assigned to the images based on the uniform spread. An assigned location may be, for example, expressed as a distance from a starting point or may be expressed as the name of a GIT segment (e.g., ascending colon, transverse colon, descending colon, etc). Demographic and statistical information may be used to access the approximate GIT length or GIT portion lengthPATENT APPLICATIONDocket: A0013107W001 of a person. For example, children two years or less have an average colon length of 52 cm, children 4-6 years old have an average colon length of 73 cm, children 9-11 years old have an average colon length of 95 cm. Such and other statics may be used to estimate the GIT length or GIT segment length of a person.

[0057] In aspects of the present disclosure, statistical information of GIT length or GIT segment length may be collected for various demographics, such as for gender, age, weight, height, nationality, and / or ethnicity, among other demographic traits. For a particular person, the statistical GIT length or GIT segment length may be accessed, and the images forming a virtual GIT may be uniformly spread out in the virtual GIT or in the virtual GIT segment(s) to assign positions to the frames in a virtual GIT.

[0058] In aspects of the present disclosure, processing techniques may be used to assign a colon segment to each image frame. Such techniques are described in U.S. Patent No. 11,934,491, to Given Imaging Ltd., which is hereby incorporated by reference in its entirety. In aspects of the present disclosure, processing techniques may be used to detect transitions between certain GIT segments. Such techniques are described in International Patent Publication No. WO / 2021 / 220272, to Given Imaging Ltd., which is hereby incorporated by reference herein in its entirety. Such information may be used together with statistical information to provide a localized location for each image of the virtual GIT and to keep or discard various frames to achieve a minimum or maximum frames per cm metric. After obtaining a localized location for each image, images may be kept or discarded to achieve minimum and / or maximum frames per cm requirements.

[0059] FIG. 7 is merely an example, and variations are contemplated to be within the scope of the present disclosure.

[0060] FIG. 9 is a flow chart of an example of an operation for navigating a virtual GIT. In various embodiments, certain operations may be performed by the user computer 410 or virtual reality device(s) 420 of FIG. 4. In various embodiments, certain operations may be performed by the computing system 430 of FIG. 4.

[0061] At block 910, the operation involves accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, where the virtual GIT of the person is created based on images captured by a capsule endoscope.

[0062] At block 920, the operation involves displaying a view within the at least the portionPATENT APPLICATIONDocket: A0013107W001 of the virtual GIT.

[0063] At block 930, the operation involves receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, where the user commands include at least commands used during a live examination. In various embodiments, the machine interface includes at least one virtual reality handset, and the user commands correspond to gestures of the virtual reality handset. In various embodiments, the gestures of the virtual reality handset are the same as hand gestures of a healthcare provider performing a live examination of a GIT

[0064] At block 940, the operation involves navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0065] The operation of FIG. 9 is merely an example. Variations are contemplated to be within the scope of the present disclosure. For example, in various embodiments, the operation may include blocks not shown in FIG. 9. Such and other variations are within the scope of the present disclosure.

[0066] Further examples of the present disclosure are provided below. In the examples below, any “means” may be implemented by at least one processor and processor-readable instructions, unless the context indicates otherwise.

[0067] Example 1. A system comprising: at least one processor; and at least one memory having instructions stored thereon which, when executed by the at least one processor, cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.PATENT APPLICATIONDocket: A0013107W001

[0068] Example 2. The system of Example 1, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.

[0069] Example 3. The system of any one of the preceding Examples, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

[0070] Example 4. The system of any one of the preceding Examples, wherein the instructions, when executed by the at least one processor, further cause the system to at least perform: accessing demographic information for the person; accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; and uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.

[0071] Example 5. The system of any one of the preceding Examples, wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

[0072] Example 6. The system of any one of the preceding Examples, wherein the machine interface includes at least one virtual reality handset, wherein the user commands correspond to gestures of the virtual reality handset.

[0073] Example 7. The system of Example 6, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

[0074] Example 8. A method comprising: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0075] Example 9. The method of Example 8, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.PATENT APPLICATIONDocket: A0013107W001

[0076] Example 10. The method of Example 8 or Example 9, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

[0077] Example 11. The method of any one of Examples 8-10, further comprising: accessing demographic information for the person; accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; and uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.

[0078] Example 12. The method of any one of Examples 8-11 , wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

[0079] Example 13. The method of any one Examples 8-12, wherein the machine interface includes at least one virtual reality handset, wherein the user commands correspond to gestures of the virtual reality handset.

[0080] Example 14. The method of Example 13, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

[0081] Example 15. A non- transitory processor- readable medium having stored thereon instructions which, when executed by at least one processor of a system, cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0082] Example 16. The non-transitory processor-readable medium of Example 15, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.PATENT APPLICATIONDocket: A0013107W001

[0083] Example 17. The non-transitory processor-readable medium of Example 15 or Example 16, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

[0084] Example 18. The non-transitory processor-readable medium of any one of Examples 15-17, wherein the instructions, when executed by the at least one processor, further cause the system to at least perform: accessing demographic information for the person; accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; and uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.

[0085] Example 19. The non-transitory processor-readable medium of any one of Examples 15-18, wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

[0086] Example 20. The non-transitory processor-readable medium of any one of Examples 15-19, wherein the machine interface includes at least one virtual reality handset,

[0087] wherein the user commands correspond to gestures of the virtual reality handset.

[0088] Example 21. The non-transitory processor-readable medium of Example 20, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

[0089] Example 22. The non-transitory processor-readable medium of Example 15, wherein the navigating the view within the at least the portion of the virtual GIT comprises navigating the view in the virtual GIT from an anus to a cecum and back from the cecum to the anus, wherein the navigating the view allows a healthcare provider to observe tissue images in the same manner as a live colonoscopy procedure.

[0090] Example 23. A system comprising: means for accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; means for displaying a view within the at least the portion of the virtual GIT;PATENT APPLICATIONDocket: A0013107W001 means for receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and means for navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

[0091] Example 24. The system of Example 22, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.

[0092] Example 25. The system of any one of the preceding Examples, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

[0093] Example 26. The system of any one of Examples 23-25, further comprising: means for accessing demographic information for the person; means for accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; and means for uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.

[0094] Example 27. The system of any one of Examples 23-26, wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

[0095] Example 28. The system of any one of Examples 23-27, wherein the machine interface includes at least one virtual reality handset, wherein the user commands correspond to gestures of the virtual reality handset.

[0096] Example 29. The system of Example 28, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

[0097] The embodiments disclosed herein are examples of the disclosure and may be embodied in various forms. For instance, although certain embodiments herein are described as separate embodiments, each of the embodiments herein may be combined with one or more of the other embodiments herein. Specific structural and functional details disclosed herein are not to be interpreted as limiting, but as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailedPATENT APPLICATIONDocket: A0013107W001 structure. Like reference numerals may refer to similar or identical elements throughout the description of the figures.

[0098] The phrases “in an embodiment,” “in embodiments,” “in various embodiments,” “in some embodiments,” or “in other embodiments” may each refer to one or more of the same or different embodiments in accordance with the present disclosure. A phrase in the form “A or B” means “(A), (B), or (A and B).” A phrase in the form “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C) ”

[0099] The systems, devices, and / or servers described herein may utilize one or more processors to receive various information and transform the received information to generate an output. The processors may include any type of computing device, computational circuit, or any type of controller or processing circuit capable of executing a series of instructions that are stored in a memory. The processor may include multiple processors and / or multicore central processing units (CPUs) and may include any type of device, such as a microprocessor, graphics processing unit (GPU), digital signal processor, microcontroller, programmable logic device (PLD), field programmable gate array (FPGA), or the like. The processor may also include a memory to store data and / or instructions that, when executed by the one or more processors, causes the one or more processors to perform one or more methods and / or algorithms.

[0100] Any of the herein described methods, programs, algorithms or codes may be converted to, or expressed in, a programming language or computer program. The terms “programming language” and “computer program,” as used herein, each include any language used to specify instructions to a computer, and include (but is not limited to) the following languages and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, metalanguages which themselves specify programs, and all first, second, third, fourth, fifth, or further generation computer languages. Also included are database and other data schemas, and any other meta- languages. No distinction is made between languages which are interpreted, compiled, or use both compiled and interpreted approaches. No distinction is made between compiled and source versions of a program. Thus, reference to a program, where the programming language could exist in more than one state (such as source, compiled, object, or linked) is a reference to any and all such states. Reference to a program may encompass the actual instructions and / or the intent of those instructions.PATENT APPLICATIONDocket: A0013107W001

[0101] It should be understood that the foregoing description is only illustrative of the present disclosure. Various alternatives and modifications can be devised by those skilled in the art without departing from the disclosure. Accordingly, the present disclosure is intended to embrace all such alternatives, modifications and variances. The embodiments described with reference to the attached drawing figures are presented only to demonstrate certain examples of the disclosure. Other elements, steps, methods, and techniques that are insubstantially different from those described above and / or in the appended claims are also intended to be within the scope of the disclosure.

Claims

PATENT APPLICATIONDocket: A0013107W001WHAT IS CLAIMED IS:

1. A system comprising: at least one processor; and at least one memory having instructions stored thereon which, when executed by the at least one processor, cause the system to at least perform: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

2. The system of claim 1, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.

3. The system of any one of the preceding claims, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

4. The system of any one of the preceding claims, wherein the instructions, when executed by the at least one processor, further cause the system to at least perform: accessing demographic information for the person; accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; and uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.PATENT APPLICATIONDocket: A0013107W0015. The system of any one of the preceding claims, wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

6. The system of any one of the preceding claims, wherein the machine interface includes at least one virtual reality handset, wherein the user commands correspond to gestures of the virtual reality handset.

7. The system of claim 6, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

8. A method comprising: accessing at least a portion of a virtual gastrointestinal tract (GIT) of a person, the virtual GIT of the person created based on images captured by a capsule endoscope; displaying a view within the at least the portion of the virtual GIT; receiving user commands from a machine interface for navigating the view within the at least the portion of the virtual GIT, the user commands comprising at least commands used during a live examination; and navigating the view within the at least the portion of the virtual GIT in accordance with the user commands and displaying the view during the navigating.

9. The method of claim 8, wherein the at least the portion of the virtual GIT is created based on at least the captured images and image stitching techniques.

10. The method of claim 8 or claim 9, wherein the at least the portion of the virtual GIT is created further based on at least one of: orientation data of the capsule endoscope, movement data of the capsule endoscope, or location data of the capsule endoscope.

11. The method of any one of claims 8-10, further comprising: accessing demographic information for the person; accessing, based on the demographic information, statistical information regarding a length of the at least the portion of the virtual GIT; andPATENT APPLICATIONDocket: A0013107W001 uniformly distributing the captured images across the length of the at least the portion of the virtual GIT.

12. The method of any one of claims 8-11, wherein the at least the portion of the virtual GIT satisfies a minimum frames per centimeter.

13. The method of any one claims 8-12, wherein the machine interface includes at least one virtual reality handset, wherein the user commands correspond to gestures of the virtual reality handset.

14. The method of claim 13, wherein the gestures of the virtual reality handset are same as hand gestures of a healthcare provider performing a live examination of a GIT.

15. A non-transitory processor-readable medium having stored thereon instructions which, when executed by at least one processor of a system, cause the system to at least perform a method as in any one of claims 8-14.

Images